I go not yet. He does not say that he will not go, but he will not go "yet". He did not wish to go in the great multitude of pilgrims that were en route, as there were reasons why he should go quietly.