In a pot, over medium to medium-high heat, pour in sugars, then butter, then pour the milk over everything. Don’t stir, but rather just gently nudge it to make sure all the sugar is covered by the milk.
Pretty quickly, you’ll start to notice it boiling around the edges. Let it get going, then let it boil for about 2 minutes. Do not stir.
Remove from heat and stir in the peanut butter and vanilla extract. Stir pretty vigorously for 1 to 2 minutes until smooth and thickened a bit. If you prefer, you can also use a stand mixer.
Pour into a parchment lined 9 x 9 pan and smooth out the top with a rubber spatula.
Let set at room temperature for 2 hours.
Slice and serve at room temperature or refrigerated. My old fashioned peanut butter fudge is pretty soft at room temperature, so if you prefer a slightly firmer bite, refrigerate it first.