Time is indeed a cultural construct. This is in part demonstrated by differing notions of time in different cultures.
Part of the strength of modern physics is that it has allowed useful mathematical calculation of some of what is entailed in passing through a wormhole. Part of the challenge is that this infrastructure provides restrictive overhead for properly understanding and moving toward a deeper knowledge. One obvious example for both of these comments is the apparent requirement of a physical process for creating a time travel machine, and for keeping the wormhole open.
The actual resolution of this matter may emerge from use of different mathematics and innovative experimental procedures. An attitude of open inquiry will be very important. Some time ago, as a graduate student in physics, I posited to a leading physicist at MIT that there might be two dimensions to time, or that time might behave differently than we think it does. He was outraged, and the conversation ended there, but now Kip Thorne and colleagues have done some of the work implied. So my hat is off to them. I think significant answers are just around the corner, and it will take considerable meticulous effort to get there.
Kip Thorne's work in the area has come a long way in very few years.