I think your last sentence contains a simple but key point Wessen. Here's my take on it:

We're told to call our choice of envelope "x", and presumably nothing else.

And to consider that the other envelope has either twice as much or half as much.

So far so good. But then comes the false step. If we now go back and use the very same original designation "x" in expressing these two possibilities as x and 2x we're caught up in a self-contradiction, since there are now at least two possibilities compared to one at the beginning. And if we add that beginning one to the present two we get three which we inevitably calculate as x, 2x, 1/2x. (And of course if we then continue and use the term x to designate our choice between these three, we're definitely sliding down a long road of confusion.)

Put it another way. It's as if in one envelope instead of cash we have a piece of paper with "x" written on it, and in the other just a piece with "2x" written on it. And if we then we go ahead and call our actual choice of envelope "x" as well, the logical clash (or collapse) appears to allow the possibilities to multiply.

This explanation needs no reference to probabilities or expected values.