Brain Teaser for the day

There's a village full of polite people that has no reflective surfaces. Every villager sees every other villager every day. Some higher power identifies that there is at least one sinner among the people. Any person who has sinned is marked by the higher power, and must leave the village. No resident will give a sinner any indication that he is marked, but all of the sinners leave on the third day. How many sinners are there?

Answer (and a good story)