By the end of the 18th century, the era of the great Narragansett plantations had ended. Lodowick Updike, grandson of the first Lodowick Updike to own Cocumscussoc, died in 1804 and divided his plantation lands among six sons.
Homestead Farm, the centerpiece of the holdings, was bequeathed to son Wilkins. His inheritance included Smith's Castle and 300 acres of land. Out of necessity in 1812, Wilkins Updike sold Cocumscussoc to Benjamin Congdon. Congdon and his heirs continued to sell off property until the house and remaining farm acreage were purchased in 1870 by the first in a series of successive short-term owners.
During the late 1800's the exterior of the Castle was substantially renovated in a fashionable, Victorian style. A veranda wrapped around the front and side of the house, the gable ends were clipped, and several new buildings were added.