There is one common misconception about overrunning first base. Some people want to argue that a player has to turn to the right (foul territory) after crossing first base in order to safely return to the base without the chance of being tagged out. This is incorrect as there is nothing in the rules that state which direction the player must turn. The rule states that the runner cannot be tagged out after overrunning first base as long as he/she immediately returns to the base.
Of course there is an exception to this rule, in that the runner can make no attempt to go to second base or he can be tagged out. This comes down to the judgment of the umpire and not whether the player turned into foul territory or not after crossing first base.
The other exception to the rule has to do with immediately returning to the base. If a runner overruns first base and thinks that he is out or that there are 3 outs, and proceeds to walk towards the dugout or his defensive position, he can be called out on appeal after the base or the runner is tagged.
Although it's not stated in many of the rule books, the reason for overrunning first base is based on the fact that the momentum of the runner doesn't allow for him to stop on first base. So keep in mind that if a player runs to first base and then stops on the base, he is not then allowed to overrun the base and get back without the chance of being tagged out. Once he steps off the base in any direction in that scenerio, he is fair game.
Some weird things can happen on this play, one being the first baseman attempting to tag the runner as he returns to first (believing he made an attempt at second) and the runner attempting to not be tagged as he gets back to first base. Although it may make him appear to be guilty of of trying for second if he attempts to avoid the tag, he should be called safe even if tagged, if he didn't make an attempt toward second.
Finally, some leagues allow a batter-runner to overrun first base on a walk or hit by pitch while others do not. Even though it shouldn't come up, it's a good idea to know how your league rules regarding this situation. Little league for example allows this.