By all means calculate the expected amount in envelope B as 5x/4 if you like. But since you're switching, you're giving up the amount in envelope A, which is x. So the expected gain on the switch must be 5x/4 - x = x/4

But what if it's A that contains 2x or x/2 and you switch it for B containing x? Then the expected gain is the result of reversing the subtraction in the above EV equation, namely x - 5x/4 = -x/4. Symmetry.

If you can't know which envelope is which, then you can't know either the advantage or disadvantage, the gain or loss, to switching. The expected gains, positive and negative, cancel. Symmetry.

If you know that A contains x, then sure, you switch. If you know A contains x/2 or 2x, then sure, you don't switch. Symmetry.