Consider the following picture:
The tighter fit is the one for which the ratio of peg area to hole area is greater.
It is the friction force between the peg and the hole which causes it to offer resistance and to become a relatively tight fit.
The corners of the square will elastically deflect and create bearing stresses. Depending on the elasticity of these corners and that of the stiffer circular hole some friction force will make the fit a tight one.
However, in the case of the circular peg the bearing surfaces are determined by what is in effect a stiffer surface due to the non-pointed shape of the peg. Here the surface of the hole is flat offering greater stiffness too. This means less elastic deflection and greater forces. Consequently it is the round peg which is able to cause the tightest situation.
You could have just as easily said more surface area touching equals more friction which equals a tighter fit.
The illustration depicts this very clearly.
If we assume perfectly round and perfectly square elements, there are four points of contact in each case. Same tightness. The part of the peg that doesn't contact the hole is irrelevant.