Legend states that a young woman was a student at Chowan College around the time of the Civil War. When her fiance went off to fight in the war, she pledged her eternal love for him. He was killed in battle, and the young girl died of a broken heart. Her ghost has been reported around the area of the Columns Building at Chowan College. People report of seeing a misty figure dressed in a brown dress wandering the halls of the Columns building. Leaves and twigs have supposedly been found in the halls in the morning by staff after the building has been locked during the night.

Chowan College used to have a Brown Lady Festival back during the 40's and 50's where a student was nominated to dress up as the Brown Lady, and the festival goers would wind up at the old Wise Family Cemetery - located across the ravine from the college campus. The spirit of the Brown Lady was supposed to imbue the new generations of students at Chowan College with pride and high ideals.