God will punish everyone that goes the wrong way. He will even punish people that never knew the right way. But these people knew the right way, and then they turned back and went the wrong way, so God will punish them more than those others.