He was born in Woolwich to Elizabeth and Edward Blackham Morris, the latter a bricklayer. In 1882 he married Hannah Short, a widow and housekeeper, at St Marylebone, London. They lived in Fulham, where James was a coal agent, and had five children. Hannah died in Fulham in 1917. An inquest on James’ sudden death in Chichester noted that he had lately been living with his son Albert Morris at 10 Pound Farm Road.