Yes, this seems to me to be very good and valid point. We identify a subset of real-world experiences all of which are well-described by mathematics and then step back and marvel at the "universality" of mathematics. I'm reminded of this.
Nevertheless, there remains the question of exactly why we have access to a "language" (to take George's point below) which we can (perhaps) abstract from (a subset of) the world, play with in our minds, and then accurately re-apply to (a subset of) the world. We take this for granted, but, as I am trying to establish, this fact is stunning and (perhaps) non-obvious.