Pour chocolate chips into a medium glass or heat resistant bowl. (We're going to pour warm milk into it in a minute.)
Combine the milk, cocoa powder, sugar, and vanilla in a medium saucepan over medium heat. Whisk constantly to dissolve the cocoa powder and let the mix come to a simmer.
Remove from heat and pour over the chocolate. Let rest for 5 minutes, then whisk to melt the chocolate. Whisk in the yogurt to evenly mix, then let cool complete.
Fill the popsicle molds about a third of the way, then sprinkle in 1 to 2 tsp mini marshmallows. Fill the molds another third, then repeat with the marshmallows. Fill one last time, then freeze for about 30 minutes. Poke in skewers, then freeze until solid.