Isn't the problem in calling the amount in your first envelope x. Imagine that the quantities are 1 and 2 pounds in the envelope. In one scenario, x = 1 and by swapping you end up with 2 pounds, but in the other scenario x = 2 and by swapping you end up with one pound, x is variable and not a fixed value.

What is described here would be the situation if you had to decide between a guaranteed one pound OR picking one of two envelopes which contain either 2 pounds or 50 pence (either doubling or halving the one pound). In that situation if you looked at expected winnings you would go for the second option.