In 1807, Josiah Baily Morse married Abigail Sweetser and bought 25 acres of property near the home of her brother, Benjamin Sweetser.
Morse worked as a cobbler in a little building on the northern side of the property, which the women of the family would later convert into a dressmaking shop. Morse’s daughter, Mary Jordan Sweetser, a talented seamstress, and her husband Rufus Merrill inherited the house.
