The entire area was once an estate, called Berleith, part of a 1,000 acre track of land owned by by Henry Threlkeld and his wife Mary who purchased the property in about 1751 when they were married. (Mary was a daughter of Dr. Gustavus Brown of Maryland, and widow of Reverend Matthew Hopkins.) Henry Threlkeld died in 1781 and Mary died a few years later. Some have written that the house was built in 1716.
Col. John Cox
The manor house was probably on the site where Georgetown Visitation School is today. It was later the home to John Cox, who was Mayor of Georgetown from 1823 - 1845(It was then a city in Maryland)
It burned to the ground right after the end of the revolution and was rebuilt. The pecan trees in the convent garden around the property was a gift from Thomas Jefferson to Threlkeld's son, John, when he married Elizabeth Ridgely. John was also Mayor of Georgetown, in 1793. John Threlkeld gave most of the estate to his daughter Jane and her husband John Cox, who also served as mayor of Georgetown (1823 to 1845, a longer term than any other person) Cox built his own mansion on the property, called "The Cedars" on the site of what is now Duke Ellington School. Surrounding the estate was farmland and the Cox place was, essentially, the end of the city for some miles.
Cox appears to have been a southern sympathizer because the home was confiscated by the government during the Civil War and was used by the Home for Destitute Colored Women and Children. In 1866 it was restored Cox. Less than a decade later, the Cox mansion was converted into Misses Earles' Seminary, an academy for girls. In 1892, the Seminary, and what was left of the Cox mansion, was replaced by Western High School.