That is a widely held view, but it has been shown to be wrong because of a few special cases where this does not hold.
Almost all fundamental processes on the quantum level are reversible, so they do not care about the direction of time. If time was going backwards, nobody would notice anything odd with these processes. But the entropy of a system increases, the process becomes irreversible. In principle i is still reversible, but the probability of that happening becomes extremely small due to statistics. So in these cases the entropy gradient gives you the direction of time.
However, there are a few fundamental processes which are not reversible, although the entropy does not change (Recent and only direct confirmation thus far:
http://physics.aps.org/articles/v5/129#c1). Although the asymmetry is very small, these cases point towards there being more to time than just the entropy gradient.
Anyway, in Relativity time is closely tied to space. Giving just a time is meaningless without also giving the location is space of any event. But as space is tied to our universe, time has also to be tied to our universe. So if there was a multiverse*, every universe must have its own time (times?) with no relation to ours. Thus if there was an event in our universe and one in another, it would be impossible to say which one was before or after the other, because those events are disconnected in time and space.
*not to be confused with the many worlds of the Many-Worlds-Interpretation of quantum mechanics, which do share time and space (at least in part)