Demographic changes and marker properties affect detection of human population differentiation