... but what it changes is unknown. This is why the issue is indeed the distribution.
An example might help. Say your monk (from the link) put a $10 bill in one envelope. He then put nine $5 bills and one $20 bill into a bucket. With his eyes closed, he picked one bill from the bucket and put it in the second envelope.
The statement "one envelope contains either half, or twice, what is the in other" is still a true statement. The chances that you pick the higher, or lower, envelope are indeed both 50%. But if you open your envelope and see $10, the chances are 90% that you have the larger envelope, not 50%.
The formula Exp(other) = (X/2)*Prob(higher) + (2X)*Prob(lower) is correct. What is wrong in the calculation is assuming Prob(higher) = Prob(lower) = 1/2 when you claim your envelope contains X. They are actually the relative probabilities that the pair of envelopes contained (X/2,X) and (X,2X), which you have no way of knowing.