The rulers - The principal authorities of the city, in the absence of the special governor. The rest that did the work - i. e. “the laboring class that (afterward) actually built the wall.”