Lee’s Daughters, Part 1

By David A. Carrino
The Cleveland Civil War Roundtable
Copyright © 2017, All Rights Reserved

Part 1 of a 6-part article about the daughters of Robert E. Lee


The Lee Family and Arlington

Prior to the Civil War, the four daughters of Robert E. and Mary Lee lived idyllic lives in a home with beautifully scenic surroundings, and they looked forward to a tranquil, secure future. All of that changed on April 20, 1861 when their father made a decision that drastically altered the lives of every member of his family. For the next four years Lee’s daughters, like daughters throughout the warring sections of the country, lived lives of sacrifice, hardship, and deep personal loss. When a country engages in war, many if not most people on the home front are adversely affected, particularly for a large-scale war like the Civil War. Such was the case for the women of the Lee family. During the Civil War all four of the men in the Lee family went into combat, but everyone in the Lee family, including the women, went to war. When the war ended, the Lee daughters were without a home and without a future, in one case literally. The lives that the Lee daughters lived after the Civil War in no way resembled their serene pre-war existence.

Although these four women are remembered primarily because their father was one of the most iconic figures of America’s greatest conflict, these women nevertheless deserve history’s attention, if only because of what they were forced to sacrifice due to their father’s ill-fated decision. To that end, this article tells the stories of Mary, Annie, Agnes, and Mildred Lee, the four daughters of Robert E. and Mary Lee, each of whom has a unique life story. Lee’s daughters’ entry into history was via their father, but the legacies of Mary, Annie, Agnes, and Mildred stand on their own. In fact, the legacies of Mary, Annie, Agnes, and Mildred deserve to stand on their own. The sections of this article which focus on each of the four daughters are not meant to be a detailed biography of each daughter, but are intended to describe the character of each of these women through their experiences, through some of their words, and through words about them from their contemporaries.

Arlington’s Founding Father

On April 22, 1861 Robert E. Lee left his family’s home at Arlington on his way to Richmond, Virginia. Two days earlier Lee had made his decision to join the Confederacy. Lee was traveling to Richmond to take up his duties in assisting the Confederacy in its effort to separate from the country that Lee had served for almost 32 years, the same country that Lee’s father had fought to create. The home that Lee was leaving had been built by the adopted son of the first president of the country that Lee was about to help break apart. That home contained many items which had been owned by that first president and which served as daily reminders of him and of the indispensable role that he had played in bringing into existence the country whose existence Lee would fight to end. The first president of the country that Lee sought to destroy was married to the great-grandmother of Lee’s wife, and that first president was eulogized by Lee’s father as “First in war, first in peace, and first in the hearts of his countrymen.” Despite these deep personal connections to the founding moments of the country that Lee had chosen to go to war against, and in spite of Lee’s lengthy and valorous military service to that country, Lee’s foremost objective at this time was that country’s dissolution. As Lee turned his back to Arlington and began his journey to Richmond, he was turning his back on the country that he had pledged to defend. Although he had no way of knowing it at the time, he was also turning his back on something else. It is not known if Lee took a moment to glance over his shoulder at the home that he was leaving, but if he did, it would be the last time that he ever looked upon his beloved Arlington while it was still in his possession.

No discussion of any member of Robert E. Lee’s family is complete without a consideration of Arlington. It is clear from the writings of Lee, his wife, and their children that there was a very strong emotional attachment to Arlington, which was due not only to their shared experiences there, but also to the beauty of the place and the numerous items that had been owned by George and Martha Washington, which were not just memorabilia, but tangible connections to the creation of the United States. Lee and his wife were married at Arlington, and six of the seven Lee children were born at Arlington. The Lee children spent much of their childhood at Arlington, even during some of the times when Lee’s military assignments required him to live elsewhere. When the Lee daughters spent extended periods away from Arlington, such as at boarding school, they wrote of how intensely they missed being there. And when the Lee family lost Arlington forever, the Lee daughters wrote of how profoundly painful this loss was.

George Washington Parke Custis

The person who built Arlington House was George Washington Parke Custis. G.W.P. Custis, who was born on April 30, 1781, was the grandson of Martha Washington from her first marriage. G.W.P. Custis’ father, John Parke Custis, died six months after the birth of G.W.P. Custis. When the elder Custis was serving as an aide to General George Washington during the siege of Yorktown, he contracted a severe febrile disease and died in 1781 at the age of 26, shortly after the British surrender. After his father’s death, G.W.P. Custis and one of his sisters were adopted by George Washington and grew up at Mount Vernon. Despite his exposure for almost all of his young life to one of the most heroic and accomplished men in American history, G.W.P. Custis fit the definition of a slacker. In a letter written shortly after the end of his presidency, George Washington characterized his adopted son as having “an almost unconquerable disposition to indolence in everything that did not tend to his amusements.” G.W.P. Custis was expelled from one school and left another without graduating. As an adult he repeatedly began projects, soon lost interest, and moved on to something else. But in spite of his shortcomings, G.W.P. Custis was charming, generous, and well-liked.

A painting of the Washington family done by Edward Savage in the 1790s
A young G.W.P. Custis is standing to the left of George Washington. Standing to the left of Martha Washington is G.W.P. Custis’ sister, Eleanor Parke Custis, after whom the third of the Lee daughters was named.

One project that G.W.P. Custis saw through to completion was his grand house at Arlington. After the deaths of George and Martha Washington (in 1799 and 1802, respectively), G.W.P. Custis received a vast inheritance that made him financially secure for the rest of his life. Among his inheritance were four large tracts of land, and he decided to have a house built on one of them. He chose the land along the Potomac River for the location of his house, because this site was the most scenic. This land had been purchased by G.W.P. Custis’ father, who named it Mount Washington. G.W.P. Custis renamed the land Arlington after the Custis homestead in eastern Virginia. Construction of the house, which occurred in stages, began in 1802 very soon after the death of Martha Washington. The north wing was completed first, and G.W.P. Custis initially lived in this portion of the final house. Almost immediately after construction of the north wing was finished, G.W.P. Custis began to accumulate many items that had been owned by George and Martha Washington, and he moved these items into his new home. The south wing was finished in 1804, but the remainder of the house, including the large center section and the portico, were not finished until 1818. G.W.P. Custis inherited two other Virginia plantations, both on the Pamunkey River. One, named White House, was on the south shore of the river, and the other, Romancoke, was on the north shore of the Pamunkey River and further downriver from White House.

Arlington House

Robert and Mary

In 1804 G.W.P. Custis married Mary Lee Fitzhugh. They had four children, but only one survived to adulthood, Mary Anna Randolph Custis. Because she was distantly related to the Lee family, she had known Robert E. Lee most of her life. Mary had another childhood connection to her future husband, which was through the man who was the benefactor of Robert E. Lee’s mother, Anne Carter Lee. Anne Carter Lee had need of a benefactor, because Robert E. Lee’s father, Revolutionary War hero Henry “Light Horse Harry” Lee, had made his family destitute as a result of some bad financial dealings. In the summer of 1813, when Robert was six years old, Henry Lee departed from the family for the West Indies to convalesce from severe injuries that he received the previous summer in a riot. Lee’s father died soon thereafter before ever rejoining his family, and his family was left in a dire financial situation. During Robert E. Lee’s youth, he and his mother spent time at the estate of Anne Carter Lee’s benefactor, who was the brother of Mary Anna Randolph Custis’ mother. These associations resulted in Robert E. Lee and Mary Anna Randolph Custis encountering each other at various times throughout their young lives.

Mary Anna Randolph Custis

In the summer following Lee’s graduation from the U.S. Military Academy, while he was awaiting his first assignment, Lee’s mother died at the age of 56. During that summer, Lee began a serious courtship of Mary Anna Randolph Custis. Lee’s first assignment was near Savannah, Georgia assisting in the construction of a fort to protect the harbor. Lee returned to Virginia the following summer, and at some point during the summer of 1830 Lee proposed to Mary Anna Randolph Custis. Her mother consented to the marriage without hesitation, but her father was not enthusiastic. G.W.P. Custis realized that the man who had proposed to his daughter was financially strapped and had little prospect for much improvement with the paltry salary he would receive as an army lieutenant. However, G.W.P. Custis regularly indulged his daughter, and it is likely that both he and his daughter knew that in time his consent would be given. Eventually G.W.P. Custis consented, and on June 30, 1831 Robert E. Lee and Mary Anna Randolph Custis were married at Arlington House.

The Lee Children

Lee and his wife went on to have seven children, three boys and four girls, all of whom lived to adulthood. The eldest child was a son, George Washington Custis Lee, who went by Custis. Another son, the third child in birth order, was William Henry Fitzhugh Lee, who was named after William Henry Fitzhugh, the benefactor of Robert E. Lee’s mother. (This son was sometimes called Fitzhugh, but he should not be confused with the Fitzhugh Lee who was the nephew of Robert E. Lee. Both Fitzhugh Lee (the nephew) and Fitzhugh Lee (the son) were cavalry officers in the Army of Northern Virginia, and both reached the rank of general. To avoid confusion, some authors refer to Lee’s son, Fitzhugh, by a childhood nickname, Rooney, and this is what is done herein.) The youngest Lee son, and second youngest child, was Robert E. Lee Jr., who was often referred to as Rob.

The eldest daughter, and second oldest child, was Mary Custis Lee. (Robert E. Lee’s wife is sometimes referred to as Mary Custis Lee, where Custis is her maiden name. The Lees’ eldest daughter, Mary, is also sometimes referred to as Mary Custis Lee, where Custis is her middle name. For brevity and to avoid confusion, Lee’s wife is referred to hereafter, if impersonally, as Mrs. Lee, and the Lees’ eldest daughter is referred to hereafter as Mary.) The Lees’ second and third daughters, who were their fourth and fifth children, were Anne Carter Lee, who was known as Annie, and Eleanor Agnes Lee, who was known as Agnes. The youngest of the Lee children was daughter Mildred Childe Lee, who went by Mildred.

Lee named one daughter after his mother, another daughter after his wife, one son after his father-in-law, another son after himself, and all of his other children after other relatives. But in spite of the fact that Lee’s father was a Revolutionary War hero, Lee named none of his sons after his father. This is thought to be because Lee did not feel particularly close to his father, who left his family when Lee was very young, never to return to them. As a result, Lee grew up without really knowing his father. In addition, bad financial dealings by Lee’s father led to the family becoming financially strapped. Moreover, Lee’s half-brother, Henry Lee IV, who carried on not only his father’s name, but his father’s irresponsible ways, caused the family to lose its home, Stratford Hall, which is where Robert E. Lee was born. Although he could not have known it at the time, the loss of Stratford Hall was a foreshadowing for Lee of the loss of Arlington.

Just before the wedding of Lee and Mrs. Lee, Lee received a new assignment at Fort Monroe in Virginia. Shortly after the wedding, the couple took up residence there, and it was at Fort Monroe that their first child, Custis, was born. All of the other Lee children were born at Arlington. In subsequent years, if Mrs. Lee was living away from Arlington at the location of her husband’s current assignment, she returned to her home for the birth of a child. Because Mrs. Lee’s parents lived at Arlington, she had their help in caring for her after the birth and in tending to her other children. After his work at Fort Monroe, Robert E. Lee was assigned to the Chief Engineer’s Office in Washington, which allowed his family to live at Arlington.

In succeeding years Lee received other assignments, which included St. Louis and New York City. Lee’s assistant for the St. Louis project, which involved improvements to the harbor, was Montgomery C. Meigs, who was instrumental in the Lee family losing possession of their Arlington home. Prior to Lee’s transfer to St. Louis, Lee was a member of a Corps of Engineers expedition that was sent to survey the border between Ohio and Michigan to resolve a border dispute. The border dispute resulted from Congress using an inaccurate map when it set the border between Ohio and Michigan. Based on this incorrect border, the city of Toledo was actually located in the state of Michigan. (This expedition and the border dispute are discussed in the May 2014 history brief, Robert E. Lee’s Invasion of Ohio, which is archived on the Cleveland Civil War Roundtable’s website.) During the time in their married life when Lee was stationed in various locations, Lee and Mrs. Lee were the parents of a growing family. A photograph of Lee with his young son, Rooney, shows a man who was described by one biographer as “incredibly handsome,” whose “eyes were dark brown, sharp, and engaging,” and whose “black hair waved and was thick and full.” A photograph of Mrs. Lee with her young son, Rob, shows an attractive woman whose robust appearance belied the ailments that would afflict her in the future and eventually make her an invalid.

The Mexican War

War broke out with Mexico in 1846, two months after the birth of the Lees’ last child, Mildred. Six months after Mildred’s birth, Lee was sent to join the war effort. This was Lee’s first combat experience, and he performed admirably. Years after the Mexican-American War, Lee’s commander, Winfield Scott, called Lee “the very best soldier I ever saw in the field.” While Lee was in Mexico, Mrs. Lee and the children lived at Arlington with Mrs. Lee’s parents. With the Lee children now numbering seven, the large expanse of Arlington House proved useful for accommodating the family. Mrs. Lee’s parents had a bedroom on the first floor in the north wing, in which they had initially lived before construction of Arlington House was completed. The first floor also contained a guest room, a dining room, and several other rooms. The bedroom that was used by Mrs. Lee (and her husband, when he stayed at Arlington) was on the second floor, and the bedroom for the three youngest daughters, Annie, Agnes, and Mildred, was across the hall in the southeast corner of the second floor. Next to the Lees bedroom was the bedroom for their sons, Custis, Rooney, and Rob. Mary had a bedroom in the front (northwest) of the second floor, although she often shared this room with a cousin of Mrs. Lee, Martha (Markie) Williams, who was eight years older than Mary and spent a good deal of time at Arlington.

Arlington House, first floor
Arlington House, second floor

When Lee returned to his family after two years in the Mexican-American War, he had been away for so long that he mistook one of the neighbor’s children for his youngest son, Rob. Lee’s first assignment after the war was to supervise renovations of the defenses protecting the harbor in Baltimore, which he did for three years. In 1852, Lee was appointed superintendent of the U.S. Military Academy at West Point, a position for which he protested that he was not qualified. However, Lee performed very well as West Point superintendent. One notable event during Lee’s tenure as West Point superintendent came when he, along with Secretary of War Jefferson Davis, had a hand in making James Whistler a painter rather than an army officer. (This life-changing event for James Whistler is discussed in the October 2013 history brief, The Major General Who Wasn’t, which is archived on the Cleveland Civil War Roundtable’s website.)

Two noteworthy events happened to the Lee family during Lee’s tenure as superintendent, one sad and one happy. On April 23, 1853, one day after her 65th birthday, the mother of Mrs. Lee died at Arlington, and on July 1, 1854 the Lees’ eldest son, Custis, graduated first in his class from West Point, one place higher than Lee, himself, had graduated exactly 25 years earlier. (The person who graduated ahead of Lee was Charles Mason, who is described in the October 2012 history brief, The Only Man to Beat Robert E. Lee in an Even Fight, which is archived on the Cleveland Civil War Roundtable’s website.) The motto of Custis Lee’s graduating class was, ironically, “When Our Country Calls,” but when that call came seven years later as the country was facing the most severe crisis in its history, both the person who graduated first in the class and the person who was the West Point superintendent at that time forsook that call.

The Secession Crisis

Robert E. Lee, circa 1850 (age 43)

In the spring of 1855 Lee received a new assignment, which necessitated that he not only leave West Point, but that he leave the Corps of Engineers. Lee was transferred to the cavalry and sent to Texas as part of an effort by the army to better protect white settlers on the frontier. While Lee was in Texas, his family lived at Arlington with Mrs. Lee’s widower father, G.W.P. Custis. Lee returned to Arlington in November 1857 after the death of his father-in-law, who died on October 10. Lee was the executor of his father-in-law’s will, and because the will of G.W.P. Custis was so complicated and his estate in such financial disarray, Lee remained at Arlington for more than two years to resolve issues with the estate.

Lee’s prolonged stay at Arlington proved fortuitous in one sense, because Lee was still at Arlington when John Brown carried out his infamous raid at Harpers Ferry, and Lee’s presence in Virginia made it possible for him to lead the troops who put down the insurrection. Because Lee had to remain in Virginia in the aftermath of the raid, it was not until early in 1860 that he left Arlington and his family to resume his duties in Texas. A year later, as the secession crisis was escalating, Lee was ordered to report to Washington to meet with Winfield Scott, his former commander in the Mexican-American War, who held Lee in extremely high regard. Lee arrived at Arlington on March 1, 1861, three days before the inauguration of the president whose election exacerbated the secession crisis, and Lee met with Scott a few weeks thereafter. Lee was offered command of U.S. forces in the looming war, but a couple of days after his meeting with Scott, Lee made the decision that led to him turning his back to Arlington and traveling to Richmond. Once Lee’s native state, Virginia, made the choice to join the Confederacy, Lee felt that he had to do likewise.

There is compelling evidence that Lee and the members of his family were not ardent supporters of secession. After Virginia voted to secede, Agnes Lee wrote in a letter to her sister, Mildred, who was away at school, “I cannot yet realize it, it seems so dreadful. But she [Virginia] had to take one side or the other & truly I hope she has chosen the right one.” Shortly after Lee made his decision to side with Virginia, Mrs. Lee said in a letter to Mildred that “the prospects before us are sad indeed, and as I think both parties are wrong in this fratricidal war there is nothing comforting even in the hope that God may prosper the right, for I see no right in the matter.” During the secession fever in Virginia prior to Virginia’s decision, Lee reputedly told someone, “I must say that I am one of those dull creatures that cannot see the good of secession.” But in spite of this, Lee chose to fight on the side of secession, because to Lee this seemed the better of two bad choices.

In no way was Lee’s decision an easy one for him. In the letter that Lee sent to Winfield Scott to tender his resignation, Lee mentioned “the struggle it has cost me to separate myself from a service to which I have devoted all the best years of my life & all the ability I possessed.” On the day that Lee made his decision, he wrote a letter to his sister, Anne Marshall, who remained loyal to the Union, if only through marriage, and who lived in Baltimore. In that letter Lee explained to his sister his reasons for his difficult decision. “With all my devotion to the Union and the feeling of loyalty and duty as an American citizen, I have not been able to make up my mind to raise my hand against my relatives, my children, my home.” But by making the decision not to raise his hand against his home, Lee’s home, Arlington, slipped from his hands and from his family’s hands.

The War Years

After the death of G.W.P. Custis, ownership of Arlington passed to Mrs. Lee through inheritance. At the outbreak of the Civil War, Mrs. Lee’s property lay on the northernmost boundary of the Confederate States of America across the Potomac River from the country which was intending to use force to prevent Mrs. Lee’s property, as well as a vast amount of other territory, from being permanently excised. The land which comprised Mrs. Lee’s property was militarily important, since it overlooked the capital of the nation that was preparing to go to war against the self-proclaimed country of which Mrs. Lee was now a citizen.

Within a short time after Lee’s departure for Richmond, Mrs. Lee was informed that the Union army was planning to cross the Potomac River and seize Arlington to prevent its occupation by Confederate forces that would have a strong vantage point over the U.S. capital. The person who provided this information to Mrs. Lee was Orton Williams. He was a cousin of Mrs. Lee and the younger brother of Markie Williams, who had spent so much time at Arlington that she shared a bedroom with the Lees’ eldest daughter, Mary. Like Markie, Orton had also spent time at Arlington during his youth. This was because Markie and Orton were orphaned at young ages, and Mrs. Lee’s father, G.W.P. Custis, perhaps recalling how George and Martha Washington had taken him in after his own father died, sought to give assistance to the children of his niece.

Orton had received a commission in the U.S. Army largely through the recommendation of Robert E. Lee, and at the outbreak of the Civil War, Orton was a staff officer in Winfield Scott’s office in Washington. As such, Orton, who later served as an officer in the Confederate army, was aware of much U.S. military information, including the impending seizure of Arlington. In early May he visited Arlington to inform Mrs. Lee of this. At the time, daughters Mary and Agnes were the only other family members with Mrs. Lee at Arlington, and they immediately began to make preparations to leave. This involved packing as much as they could for removal and placing the rest locked in the cellar. Among the items that were packed or stored were a number of things that had belonged to George and Martha Washington. In a letter to her sister, Annie, who was at White House at that time, Agnes wrote, “We have packed up a good many things…It is so so sad to leave home.”

Selina Gray

On May 15 Mrs. Lee, Mary, and Agnes left Arlington, and on May 24 Union troops crossed the Potomac River and occupied Arlington. Their commander, Irvin McDowell, issued orders to remain outside the house. The troops pitched tents on the lawn, and many of the trees were cut down for firewood. In a few months the cellar was broken into, and many of the items, including some of the Washingtons’ possessions, were stolen, probably by men who did not know the historical significance of the things that they were taking. During the Civil War Markie Williams, who remained loyal to the Union, but nevertheless also remained close with her cousins, played a role in preventing many items from being stolen from Arlington House. The person who was most instrumental in preventing items from being stolen was Selina Gray, who was an Arlington slave and was the personal maid of Mrs. Lee. When Selina saw Union soldiers looting Arlington House, she appealed to McDowell and in so doing saved many artifacts from being stolen, including items that had been possessions of George and Martha Washington.

Montgomery Meigs

When Union troops seized Arlington, Mrs. Lee’s property became one of the first Confederate territories to be returned to U.S. control. In the summer of 1862 Congress authorized the quartermaster general of the army to acquire land to be used as national cemeteries for military dead. The quartermaster general at that time was Montgomery Meigs, who had been Robert E. Lee’s assistant during the pre-war project at St. Louis. On June 15, 1864 Meigs sent a letter to Secretary of War Edwin Stanton in which Meigs recommended that military dead be interred at Arlington. Meigs saw this as an appropriate use of land that was picturesque and close to the nation’s capital, and which was also the home of the person who had turned his back on the U.S. to fight for the rebellion. Stanton agreed, and Meigs further ordered that the dead be buried close to Arlington House in order to make the house uninhabitable.

By the end of the Civil War, many Union dead had been interred at Arlington, and an extensive village for freed slaves had been built on the property. In 1877 the Lees’ eldest son, Custis, tried to recover Arlington through the courts, but the place that at one time was to be the home for future generations of the Lee family was lost to that family for all time. There is a letter that the Lees’ daughter, Annie, wrote when it became clear that there would be armed conflict between the two clashing sections of the U.S. In that letter, which Annie wrote to a friend in Georgia, Annie mentioned a letter that she had received from a Northern friend, in which the Northern friend evidently made a taunting remark about the impending war. Annie wrote to her friend in Georgia, “She asks me if we intend to make Virginia a graveyard, and I have replied ‘not for us, but for you.'” Ironically, one part of Virginia that was made into a graveyard for Northern military dead was the land that was the Lee family’s home.

War’s End

For much of the Civil War, Mrs. Lee and her daughters lived an itinerant life and stayed with different relatives or friends. In early 1864 Mrs. Lee and her daughters took up residence in a rented house on East Franklin Street in Richmond, the house that came to be known as the Richmond residence of the Lee family, and they were in that house when Richmond fell. On April 9, 1865, Palm Sunday, Lee surrendered at Appomattox Court House, although in his farewell message to his men, Lee did not admit that they were vanquished, but claimed that the Army of Northern Virginia was “compelled to yield to overwhelming numbers and resources.” Six days later Lee joined his wife and daughters in the rented house in Richmond. With Arlington taken from them, Lee and his family had no prospects for a home.

Later that spring the Lee family received an offer from a friend to live rent-free in a small cottage that was owned by that friend. The cottage, named Derwent, was in a wooded area along the James River 50 miles west of Richmond. Lee wanted his wife, whose health had been deteriorating for several years due to rheumatic disease, to avoid the hot Richmond summer, and he accepted the offer. Lee, Mrs. Lee, their son, Custis, and their daughters, Mary, Agnes, and Mildred, moved to Derwent in June of 1865. Sons Rooney and Rob were at White House, one of the properties that had been owned by G.W.P. Custis, which Rooney and Rob were restoring and returning to a working farm.

In August 1865 Lee was offered the presidency of Washington College, which he accepted, and the Lee family moved to Lexington, Virginia, and took up residence in the president’s house, which placed them further away from Arlington. Lee held his position as president of the college until his death on October 12, 1870. Lee’s son, Custis, was named to succeed his father as president of the college, which allowed the Lee family to continue to reside in the president’s house in Lexington. Their real home, Arlington, was no longer available to them.

Robert E. Lee once said that Arlington was the location “where my attachments are more strongly placed than at any other place in the world.” But those attachments were forever severed, because Arlington was transformed from the Lee home into a final resting place for those who served the country that claims as its symbolic father the man who was the adoptive father of the person who built the house that for many years was the Lee family’s home.

Arlington House during the Civil War, occupied by Union troops

Continue with Part 2 >>