It is the friction force between the peg and the hole which causes it to offer resistance and to become a relatively tight fit.

The corners of the square will elastically deflect and create bearing stresses. Depending on the elasticity of these corners and that of the stiffer circular hole some friction force will make the fit a tight one.

However, in the case of the circular peg the bearing surfaces are determined by what is in effect a stiffer surface due to the non-pointed shape of the peg. Here the surface of the hole is flat offering greater stiffness too. This means less elastic deflection and greater forces. Consequently it is the round peg which is able to cause the tightest situation.