Dermatologists say a lot of different factors can play a role here—and some are out of your control.