Assuming that the horse is coming at walk, it is much safer for everyone if you wait until the horse is only, say, a dozen feet away from you before you move off the path, particularly when you're going in opposite directions. I agree that if it's coming up behind you at speed then you want to get out of the way! Also, no sudden movements if possible!

It's the people who're coming towards you and run to hide in the undergrowth as soon as they see you, even if you're not actually going to cross paths for several minutes more, who're (inadvertently) dangerous.