Preheat the oven to 375 degrees F. Combine the butter and brown sugar in a 10" ovenproof skillet and stir until the mixture looks like wet sand. Add the vanilla and bourbon and cook over medium-high heat until it starts to bubble. Remove from the heat and arrange the peaches in concentric circles over the caramel and set the skillet aside.
Beat the butter and the remaining 1 cup sugar in a large bowl with a mixer on medium-high speed until light and fluffy, about 3 minutes. Add the eggs one at a time, beating after each addition, then beat in the vanilla and bourbon.
Whisk the flour, ground pecans, baking powder, baking soda and salt in a bowl.
Reduce the mixer speed to low; add the flour mixture in 3 batches, alternating with the buttermilk, beginning and ending with the flour, and mix until just incorporated. Pour the batter over the peaches and spread evenly.
Transfer the skillet to the oven and bake until the cake is golden and a toothpick inserted into the center comes out clean, about 50 minutes. (Cover loosely with foil if the cake is browning too quickly.) Let sit 15 minutes, then invert onto a platter and let cool completely.
Serve with sweetened peach slices and vanilla ice cream (optional)