While scale weight does not in fact tell the whole story, being in the "healthy" BMI range for your height is optimal to prevent future problems, regardless of risk factor, having excess fat still means higher risk for diseases, injuries, etc, no matter what. I'm about the same size as OP, and I'd consider myself fat, even though I ran a half marathon, rock climb, lift 4x a week, etc etc. My bf% could be lower, and I could lose about 20 lbs to be somewhere in the higher end of the healthy range for my height. (which is coincidentally what I'm working on now)
Your weight very well could be higher due to large amounts of muscle, which would certainly discount the BMI scale, however, the vast majority of people DON'T have that sort of muscle mass. Honestly, bf% is much more effective in telling you more about your body composition and health than just BMI/weight (like you and OP said, it's definitely not a tell-all), but that doesn't mean that one should completely ignore BMI ranges and the number on the scale. I agree that a lot more goes into play, but that doesn't mean these factors aren't still important in the grand scheme of things is all.