The difficulty arises because of the fact that any travel in time also requires travel in space. He calls this the spatial problem. Here's his explanation:
Let’s do a quick thought experiment. Imagine you have a nifty time machine and decide to pop one month into the past .... In a typical story you’ll appear at precisely the same location, just a month earlier. But how on earth does your time machine ... get you to that unique physical place?Scharf's point is that because every spatial location is moving relative to other spatial locations in the universe, if you were to travel back in time, say, one week you would not arrive at the same room you're sitting in now. You'd probably find yourself floating in space somewhere far removed from the earth which has hurtled on through space in the intervening week.
On Earth’s surface we’re in constant motion. The planet’s spin has us racing around at about 1,600 kilometers an hour at the equator. The Earth is orbiting the Sun at an average of 110,000 kilometers an hour. The Sun is currently moving relative to the center of the Milky Way galaxy at about 828,000 kilometers an hour, and our Local Group of galaxies is plunging through the cosmos at a velocity of about 2.4 million kilometers an hour relative to the cosmic microwave background.
That radiation field offers a way to establish a universally agreed-upon measure of rest or motion.
But space is of course expanding, so on very large scales no physical object can be said to be truly at rest with respect to others – it may just be equally not at rest in all directions.
That’s gets us back to our time travel experiment. To go back 1 month, and to appear at the same place ... you must also move a significant amount of physical distance. And you must do this extremely accurately. This is the spatial problem.
Let’s take the Earth’s motion around the Sun. A month of orbit corresponds to moving in an arc of approximately 78 million kilometers. During that same period the entire solar system will have also moved approximately 600 million kilometers around our galaxy, and our entire Local Group of galaxies will have swept through about 1.7 billion kilometers of space relative to the cosmic microwave background. Not only do you need to traverse those kinds of distances, you need to get it correct to within a part in a trillion.
In other words: your time travel device has to be exceedingly good at figuring out where in the universe to place you, not just when....
On the one hand it’s scientifically interesting to think about how to actually deal with coordinates in a real, and very dynamic universe. Where you are at this instant is not a fixed point in any cosmic sense. Indeed, you follow a quite complex trajectory through the universe, and thanks to complicated gravitational and mechanical interactions and behaviors this trajectory is probably not fully predictable.
Earth’s spin varies, its orbit varies subtly over very long timescales, and even our intergalactic motion will evolve as other galaxies and mass concentrations get closer or further away over time.
True time travel, to be practical, must also somehow factor in the motion of all the spatial locations in the universe so that the time traveler doesn't wind up marooned in space.
Scharf concludes with a mention of one implication of this problem:
It’s also fun to consider that this could provide an answer to the question of why, if time travel is ever invented, we haven’t been visited by beings from the future.... Perhaps the reason is that no one has (ever) solved the spatial problem, and the cosmos is littered with time travelers adrift between the stars and galaxies.