You are technically correct in that your metabolic rate only varies by a few percentage points. But where you are way off is how much of a difference "a few percentage points" can make.
If you used to burn 2500 calories at rest and then that drops 3% to 2425, that is 75 calories a day. Which amounts to around a pound of excess weight every 5-6 weeks. Or 10 lbs a year. After a quick 5 years you look up and you are suddenly 50 lbs over weight just because your BMR dropped 3% and you kept eating the exact same number of calories.
If you used to burn 2500 calories at rest and then that drops 3% to 2425, that is 75 calories a day. Which amounts to around a pound of excess weight every 5-6 weeks. Or 10 lbs a year. After a quick 5 years you look up and you are suddenly 50 lbs over weight just because your BMR dropped 3% and you kept eating the exact same number of calories.