Prior to our trip, I loosely planned out what we wanted to see and do each day and booked the Tower of London for day 4. We did not luck out with the weather the majority of our trip, but the day we went to the Tower of London and the Burough Market was the most beautiful (albeit cold) day we had. This was lucky given that is all mostly outdoors. (B wasn't too into having his photo taken this day either).
I purchased the audio guide for each of us which said to plan about 2 hours at the Tower of London and the grounds. B spent nearly 3 and I had to pull him away because I knew he was starving. He soaked up all the history and learned so much (I did too, despite having been there before, I didn't remember all the history).
From there we walked a mile down the river to Burough Market where we got burgers and sat on the ground people watching).
And of course hit up the famous chocolate covered strawberries after quite a long line.