Preheat the oven to 350°. Butter two 9-inch cake pans and line the bottoms with parchment paper. Butter the paper.
In a bowl, combine the flour with the baking powder and salt. In another bowl, using an electric mixer, beat the granulated sugar with the almond paste until the paste turns into fine crumbs. Add the butter and beat until light and fluffy. Add the eggs and yolks, 1 at a time, beating well after each addition. Stir in half of the dry ingredients and then stir in the milk. Stir in the remaining dry ingredients until the batter is smooth.
Pour the batter into the prepared cake pans. Bake for about 40 minutes, or until a tester inserted in the center of each cake comes out clean. Transfer the cakes to a wire rack and cool them completely in their pans.
In a bowl, using an electric mixer, beat the butter with 1 1/2 cups of the confectioners' sugar and the lemon zest until smooth. Beat in the milk. Stir in the remaining 2 cups of confectioners' sugar and beat until smooth.
Invert the cakes onto a work surface; peel off and discard the parchment paper. Set 1 cake on a plate and spread the jam on top. Cover with the other cake and spread the frosting all over the top and side of the cake. Sprinkle the almonds on top of the cake and refrigerate for 30 minutes to firm up the frosting.
The cake can be refrigerated for up to 2 days. Bring to room temperature before serving.