If it isn't bought back by the end of a full year, then ownership of the house in the walled town is permanently transferred to the one who bought it and their descendants. It won't be returned in the Jubilee.