Hi there!
Here's a great list of Osaka based Onsen (bottom of page) maintained by the Japanese National Tourism Organisation. Personally I really recommend Kinosaki Onsen, it is 2.5hrs away each way, but can be reached easily with a JR Pass and is a wonderful town.
Rather than go for a full hotspring, how about a spa bath, or Sento? You will find lots of these in Osaka and Kyoto (ask your hotel for the nearest one to you when you arrive). For Osaka, as you'll be with a child, how about [Spa World]? You'll find western and eastern baths, along with a amusement pool. Although the western ones may be a bit cheesy, the Japanese styled baths are good, and there will be something to do for your youngster. For Kyoto, I recommend the bath at Funaoka onsen.
In general, children will have to be accompanied by adults in Onsen as they can be extremely hot.
Hope this helps!