By David A. Carrino
The Cleveland Civil War Roundtable
Copyright © 2017, All Rights Reserved
Part 6 of a 6-part article about the daughters of Robert E. Lee
The Lee Family Deprived of Arlington
“The Memories of Those That to Us Rendered It Sacred”
After the Civil War, the Lee family had two significant encounters with Arlington. The first occurred in June 1873, when Mrs. Lee visited the place that had been her home for 53 years, from the day she was born until she was forced to leave it in the face of an imminent occupation by an enemy military force. Mrs. Lee had written to a friend in 1868 of her desire to visit Arlington and acknowledged, “The longing I have to revisit it is almost more than I can endure.” Other words that Mrs. Lee wrote prior to her visit make clear that the frail woman recognized both her need to visit Arlington and the dwindling time yet available for her to do so. “Life is waning away, and with the exception of my own immediate family, I am cut off from all I have ever known & loved in my youth & my dear old Arlington I cannot bear to think of that used as it is now & so little hope of my ever getting there again. I do not think I can die in peace until I have seen it once more.”
Three years after she had lost her husband to death and realizing that her already fragile health was declining even more, Mrs. Lee undertook a trip to Arlington. What she saw saddened her. Where once there had been scenic grounds and a lively home, instead her eyes fell upon row after row of graves and an empty, vandalized house. Mrs. Lee summarized her visit with the somber words, “I rode out to my dear old home but so changed it seemed but a dream of the past—I could not have realised it was Arlington but for the few old oaks they had spared & the trees planted by the Genl and myself which are raising their tall branches to the Heaven which seems to smile on the desecration around them.” The conversion of Arlington estate into a military cemetery was considered by many at that time to be a transformation of the land along the Potomac River into hallowed ground, but Mrs. Lee characterized this as “desecration” of her home. As sad as the visit was for Mrs. Lee, there was one benefit of seeing for herself what had become of Arlington. Mrs. Lee noted, “My visit produced one good effect. The change is so entire that I have not the yearning to go back there & shall be more content to resign all my right in it.” Five months after her visit, Mary Anna Randolph Custis Lee died and was laid to rest 150 miles away from her Arlington home.
The other post-Civil War encounter between the Lee family and Arlington, which involved an attempt to reclaim their former home, began in earnest in 1874, after some earlier post-war efforts by the Lee family to recover Arlington. Upon the death of Mrs. Lee in 1873, her eldest son, Custis, inherited Arlington. Custis’ younger brothers, Rooney and Rob, inherited two other properties of G.W.P. Custis, White House and Romancoke, respectively, and after the Civil War Rooney and Rob repaired each of them and lived in them. However, Custis’ inheritance had been seized during the Civil War, illegally as it turned out, and became a U.S. military cemetery.
Custis Lee 
Rooney Lee 
Robert Lee Jr.
The U.S. government’s basis for the seizure was failure to pay taxes on the property. In 1862, after passage of an act by Congress that imposed a tax on property in “Insurrectionary Districts,” the U.S. government demanded payment of property taxes for Arlington estate in the amount of $92.07 ($1,727 in 2017 dollars). Mrs. Lee sent a representative to make payment on her behalf, but the government mandated that payment must be made by the owner of the property, in this case Mrs. Lee, who at the time was living in Richmond and under no circumstances was about to enter Union territory after recently being caught behind enemy lines during McClellan’s Peninsula Campaign and escaping only because she was allowed to pass through the lines into Richmond. Since the U.S. government considered the taxes unpaid and the property in default, Arlington was seized, put up for auction, and purchased by the U.S. government. In 1864 Arlington began to be used for burial of Union dead, and by the end of the Civil War there were thousands of graves on the land that had been the home of the Lee family. In addition, on May 5, 1863, three days after Stonewall Jackson’s flank attack at Chancellorsville and five days before Jackson’s death, Freedman’s Village, a large camp for former slaves, was established on Arlington estate, and the camp was still occupied when Custis began his effort to recover Arlington.
In 1874 Custis petitioned Congress for compensation for Arlington, but Congress never acted on the matter. Three years later Custis took his case to court, this time seeking eviction of the U.S. government from the property. Several months passed before the case was heard, but after a six-day jury trial in U.S. Circuit Court, the jury found in favor of Custis on January 30, 1879. Regarding the requirement that the tax be paid in person, the presiding judge wrote, “The impolicy of such a provision of law is as obvious to me as is its unconstitutionality.” The U.S. government appealed the ruling, and in 1882 the case was heard by the Supreme Court, which ruled 5 to 4 in favor of Custis, to whom title to Arlington estate was returned.
By that time there were almost 20,000 graves in Arlington, and relocating that many graves would have required enormous effort and expense. Instead, the U.S. government sought to purchase Arlington from Custis, and an agreement was reached in March 1883. Custis was paid $150,000 (almost $3.5 million in 2017 dollars), and he shared this payment with his two surviving sisters, Mary and Mildred. While the monetary compensation for Arlington was certainly beneficial, it meant the irrevocable relinquishing of the Lee family’s home. It had to be a bittersweet victory for the Lee family, who were told, in essence, that their home had been taken from them illegally, but, for all practical purposes, their home could never be returned to them. For the living members of the Lee family, Arlington was gone, and all that they had of their former home were, in the words that Lee had written to his wife on Christmas day of 1861, “the remembrances of the spot, & the memories of those that to us rendered it sacred.”
“Nobody Will Hear or See”
Among the memories that made Arlington “sacred” to Lee is that it was the birthplace of all four of his daughters. Lee loved his wife and his sons, but he treasured his daughters. This is evident in the words of deep affection that Lee wrote in many of his letters to his daughters. With all due respect to mothers (and mothers are due an enormous amount of respect), there is no bond like the bond between a father and a daughter. A Lee biographer wrote, “Neither Lee nor his daughters were aware of how possessive he was, or of how much they acquiesced in that possessiveness.” This statement certainly applies to Annie, Agnes, and Mildred, although it may not apply as strongly to Mary, who chose to live so much of her life away from her family that she has been characterized as estranged from her family, and perhaps Mary’s estrangement from her family resulted from a sense of her father’s possessiveness. However, Mary eventually came to feel close to her family, although that happened only when Mary was near the end of her life. It was only during the last few years of her life, when Mary embraced her status as the last surviving child of Robert E. and Mrs. Lee, that Mary felt the closeness to her family that her younger sisters had felt throughout their lives. Nevertheless, Lee had great affection for all of his daughters. Unfortunately for them, Lee’s decision to side with the Confederacy had a disastrous effect on his daughters. As such, Lee, in light of the way events played out, placed his allegiance to Virginia above his devotion to his daughters, although he had no way of knowing beforehand what the consequences of his decision would be for his daughters.
This page records the date of birth of Mary Anna Randolph Custis and the date of her marriage to Robert E. Lee. The page also records the dates of birth of the seven Lee children and indicates that six of them were born at Arlington.
It is interesting to wonder how the Lee family’s future would have been changed had Lee decided differently in April 1861. If Lee had chosen to wear blue rather than gray, then when Union forces crossed the Potomac to occupy Arlington, that action would have not only denied Confederate forces a militarily strong vantage point over the U.S. capital, but also secured Arlington for the Lee family after the war. Had Lee chosen to remain loyal to the Union, then when his former assistant, Montgomery Meigs, selected a location for a Union cemetery, it would not have been the Lee family’s home, and more than 400,000 U.S. military and family members would now have their final resting places somewhere else. There would not have been a need for Lee’s son, Custis, to go to the Supreme Court in order to obtain his inheritance, and the house that G.W.P. Custis built would have been preserved for his descendants.
By no means were the Lee daughters the only Americans who suffered hardships and losses as a result of the Civil War, nor is it necessarily true that they suffered greater hardships or losses than anyone else. But the Lee daughters suffered a unique and highly publicized Civil War loss, Arlington, and the reason that they suffered this loss was also unique and highly publicized, specifically, the decision that their father made to serve the Confederacy. Moreover, while the loss of Arlington is frequently connected to Robert E. Lee, this loss is not as often associated with the Lee daughters. But this loss was as painful for them as it was for their father.
In one translation of a short story by Anton Chekhov, there is a passage which reads, “At the door of every contented, happy man somebody should stand with a little hammer, constantly tapping, to remind him that unhappy people exist, that however happy he may be, sooner or later life will show him its claws, some calamity will befall him – illness, poverty, loss – and nobody will hear or see.” When the Lee daughters were living the happy lives of their youth, no one was tapping with a hammer on the door of Arlington House to remind them that a calamity awaited them in the future. Nevertheless, life’s claws slashed Robert E. Lee, although he made himself a willing target of those claws due to the decision he made in the early morning hours of April 20, 1861. When life’s claws struck Lee, those claws also struck the members of his family, including his daughters.
Lee’s personal losses that resulted from his decision are well-known. The same cannot be said of Lee’s daughters, who, to paraphrase Anton Chekhov, were shown life’s claws, but nobody heard or saw. Not many people nowadays have heard about the sacrifices that Lee’s daughters were forced to make because of their father’s decision. One of the usual pieces of information that is given regarding Arlington is that it at one time was the home of Robert E. Lee. But it was also the home of Mary, Annie, Agnes, and Mildred Lee, and when Arlington was lost to them, they lost not only their home, but also the future which they would have had with the home of their contented youth as the physical and psychological epicenter of their adult lives.
Robert E. Lee’s story has been told many times, but his daughters are remembered mainly because they are the daughters of Robert E. Lee. However, Mary, Annie, Agnes, and Mildred are much more than just Lee’s daughters. History needs to see Mary, Annie, Agnes, and Mildred not simply as Lee’s daughters, but as the individuals they truly are.
Mary Custis Lee 
Anne Carter Lee 
Eleanor Agnes Lee 
Mildred Childe Lee
Most of the information for this article came from the following sources.
Coulling, Mary P., The Lee Girls. Winston-Salem, NC: John F. Blair, 1987.
deButts, Mary Custis Lee, ed., Growing up in the 1850s: The Journal of Agnes Lee. Chapel Hill: The University of North Carolina Press, 1984.
Dowdey, Clifford and Manarin, Louis H., eds., The Wartime Papers of R.E. Lee. New York: Bramhall House, 1961.
Thomas, Emory M., Robert E. Lee: A Biography. New York: W.W. Norton & Company, 1995.
Flood, Charles Bracelen, Lee: The Last Years. Boston: Houghton Mifflin Company, 1981.
The web site of Arlington House, The Robert E. Memorial (https://www.nps.gov/arho/index.htm).
Poole, Robert M., How Arlington National Cemetery Came to Be. Smithsonian Magazine, November 2009 (http://www.smithsonianmag.com/history/how-arlington-national-cemetery-came-to-be-145147007/).
Click on any of the book links on this page to purchase from Amazon. Part of the proceeds from any book purchased from Amazon through the CCWRT website is returned to the CCWRT to support its education and preservation programs.