Earl Gregg Swem Library is located on Landrum Drive (formerly Old Campus Drive), across new campus from Andrews Hall. The library is named for Earl Gregg Swem, College Librarian from 1920-1944.

Detailed discussions of plans for the library were held in 1963 and the groundbreaking ceremonies were held later that year on October 11, 1963 at Phi Beta Kappa Hall. The actual groundbreaking occurred a few weeks after the ceremonies. The cornerstone of the library was laid on October 22, 1964 and the building was scheduled for completion around December 1965. The building officially opened on January 4, 1966 although it was not fully complete. The official dedication ceremony for the library was held on Charter Day, February 12, 1966. The Tucker-Coleman Room of the library was dedicated on November 11, 1966.

At the time of its completion, the ground floor of Swem Library contained the Botetourt Gallery, an auditorium, the Institute of Early American History and Culture offices, a rare book room, an honors room, a museum, an audio/visual department, a film preview room, and a faculty lounge. The first floor contained a reserve room, an after-hours reading room, a reference department, and typing and meeting rooms. The second floor contained administration rooms, conference rooms, and stacks.

There was a fire in the Botetourt Theater in 1972 that destroyed a projection booth. A Micro Computer Lab opened in the library on February 13, 1984.

Construction officially began on an addition to the front of the library on March 3, 1986 to provide extra stack space, reading areas, administrative offices, and a 24-hour study room and snack area. The addition was dedicated on February 5, 1988. A seven-year renovation was officially completed on February 5, 2005, with rededication ceremonies during Charter Day weekend.