Preheat oven to 375 degrees.
In a medium bowl, whisk flour, cocoa powder, baking soda, and salt together.
In the bowl of a stand mixer or in a large bowl with a hand mixer or whisk, cream butter and sugar together until light and fluffy, about 3 minutes. Add egg, milk, and vanilla, and beat until well combined.
Slowly add flour mixture and mix until just combined, taking care not to over-mix.
Drop about a tablespoon of dough onto an ungreased cookie sheet, about 2 inches apart. Bake until cookies for 9 minutes before removing from the oven.
Place a marshmallow, cut-side down, in the center of each cookie, pressing down slightly. Put the cookie sheet back into the oven and continue baking for just one more minute, until marshmallows begin to melt. Remove from oven and transfer cookies to a wire rack to cool completely before frosting.
Starting in the center of the marshmallow and continuing outward, spread about 1 tablespoon of frosting over the marshmallow, until it's completely covered. Top with sprinkles if desired and let frosting harden before serving.