Lightly grease a 9 x 13 baking dish and preheat oven to 350 F.
Coarsely chop the bread into 1 inch cubes, or rip into pieces of about that size. Place in the baking dish.
In a mixing bowl, whisk together milk, heavy cream, both sugars, eggs, vanilla extract, cinnamon, and nutmeg.
Pour mixture over bread. If you'd like to cook it now, let it stand for 15 minutes more before baking. Otherwise, cover and refrigerate until you're ready to bake.
Bake until golden and the top is just beginning to crisp up, about 45 minutes. Can be cooked longer if you'd like a more crisp texture.