Line an 8x8 inch or 9x9 inch baking pan with parchment paper. Set aside.
In a medium bowl, stir pumpkin puree, cinnamon, ginger, nutmeg, cloves, salt, and vanilla extract together. Set aside.
Melt butter, brown sugar, and whole milk together in a medium saucepan over medium heat.
Stir together and bring to a boil. Boil for 3 minutes, stirring occasionally.
Remove saucepan from heat. Stir in pumpkin puree mixture. Then add in powdered sugar in and stir until smooth. If powdered sugar is clumpy in the batter, place saucepan back over medium heat and stir until melted and smooth.
Spread fudge into prepared baking pan, and smooth the top. Let it sit at room temperature for one hour, then store in the refrigerator for another hour, or overnight, until set.
Remove from the refrigerator and slice into squares. Store leftovers in the refrigerator for up to 2 weeks.