Hi there!
Sounds exiting! Here are some thoughts on your trip:
Travel from Haneda at 11.30pm.
At that time of night, the monorail service will not be operational. I recommend you look into these [late night buses]. Or expect to book a group taxi (or 2 or 3!) for the trip to your hotel.
Long distance travel
The good news is that while you will certainly be busy, the vast majority of your travel is covered by the JRPass. The only parts where an extra ticket will be needed are for Hakone and Mt Fuji. For Hakone the JR Pass will get you most of the way to Odawara station. From here you'll need to transfer to the non-JR Odakyu operated lines for the trip up to and around Hakone. I recommend the Hakone Free Pass which can be purchased on arrival into Odawara station. Where do you think you'll aim to go to see Mt Fuji? Hakone does have good views of the mountain, but if you let me know where you're thinking of for Mt Fuji (Gotemba? Kawaguchiko) I'll let you know if you have to pay an extra charge here. Finally, you can make great use of the JR Pass within Tokyo on the 8th-10th) - here's a map of JR lines in Tokyo for reference.
For a first trip, I recommend you visit Hiroshima and Miyajima island with the group. Nagasaki is an excellent destination, but will be a long journey from Osaka, especially with children. Hiroshima is arguably more interesting, and you have the wonderful island of Miyajima nearby (the Miyajima ferry is free for holders of the JRPass). I will assume Hiroshima for all the following calculations.
Your travel with your husband
Here is the cost breakdown of your trips with your husband. I think for this a 7 day pass is highly recommended and you'll make excellent savings.
-
Tokyo > Kyoto: ¥13,420 (164mins, 513km)
-
Kyoto > Tokyo: ¥13,420 (161mins, 513km)
-
Tokyo > Odawara: ¥3,840 (35mins, 83km)
-
Odawara > Osaka: ¥12,070 (190mins, 472km)
-
Osaka > Kyoto: ¥540 (29mins, 42km)
-
Kyoto > Hiroshima: ¥10,990 (112mins, 380km)
-
Hiroshima > Kansai airport: ¥11,830 (166mins, 402km)
JR ticket cost: ¥66,110 for 7 legs
Group travel
Here's the cost breakdown for the group.
-
Tokyo > Odawara: ¥3,840 (35mins, 83km)
-
Odawara > Osaka: ¥12,070 (190mins, 472km)
-
Osaka > Kyoto: ¥540 (29mins, 42km)
-
Kyoto > Hiroshima: ¥10,990 (112mins, 380km)
-
Hiroshima > Kansai airport: ¥11,830 (166mins, 402km)
JR ticket cost: ¥39,270 for 5 legs
Again, as you can see the group will also make great savings with the 7 day pass. Note I have not included possible savings on travel across Tokyo, but you can budget an extra ¥1000-2000 savings depending on how much they travel while you are in Kyoto.
Children
Looking at your group, the child aged 9 will qualify for the Child JR Pass (50% discount) and the child aged 5 will qualify for 100% free travel on JR provided they are with an adult. Unfortunately, the child aged 12 will not qualify for a child fare however and will need to use an adult JR Pass.
Accommodation
Accommodation wise, it's always hard to recommend, however be sure to check the good value Japanese discount hotel chains of Toyoko-Inn and [Dormy-Inn] along with normal recommendations on tripadvisor.
Recommendation
Overall, I can easily recommend the 7 day pass to both groups of traveller on your trip. You and your husband will make the most savings, but all will enjoy some savings and retain the flexibility to change plans if needed at any point (eg, for the second group to day trip to Nikko while you and your husband are in Kyoto, or travelling all together to Nara while you are in Osaka/Kyoto).
One last thing - as you'll be leaving early in the morning from Kansai airport, you'll need to be at the airport by 6am-6:30am. The [very earliest trains from Osaka] do get you in on time, but you may want to think about staying at a [hotel near Kansai Airport] on the last night.
Hope this helps!