Preheat the oven to 350°. Using a knife, peel the oranges; be sure to remove all of the bitter white pith. Working over a bowl, cut in between the membranes to release the sections into the bowl; pour off and reserve the juice for another use.
In a 10-inch cast-iron skillet, melt the butter with the brown sugar and cardamom until moistened. Remove from the heat and let stand for 10 minutes.
Arrange the orange sections in the skillet in 2 concentric circles with all of the oranges facing the same direction. Pack the oranges tightly together and place 1 orange section in the center to fill the empty space.
In a small bowl, mix the flour with the cardamom, baking powder and salt. In a large bowl, using an electric mixer, beat the butter with the sugar until fluffy. Add the eggs 1 at a time, beating well after each addition. Stir in half of the dry ingredients, then stir in the milk, vanilla and orange zest. Stir in the remaining dry ingredients until just combined.
Spoon the batter over the orange sections. Bake for about 40 minutes, or until the cake springs back when pressed. Let the cake cool for at least 15 minutes, then turn it out onto a platter. Serve with whipped cream.
The cake can be kept at room temperature for up to 1 day. Serve at room temperature or rewarm in a 325° oven for 10 minutes.