But if the manslayer should at any time pass the boundary of the city of his refuge, whither he may have fled;