"If you were watching a film of the object's motion, say a ball gently rolling across a floor, you wouldn't be able to tell if the film was playing forwards or backwards. "

This is not quite true. If the observer had sufficiently accurate measuring instruments, they would notice the ball's energy changing due to friction as it rolls across the floor. This change of energy will tell the observer which direction the film is running. If the energy is increasing the film is running backwards. As there are no frictionless surfaces in the real world, we can always find the direction of time.

The error in thinking you are making is deciding which equations describe the "real" we live in and which describe Plato's cave - a world we do not inhabit.