House Speaker Nancy Pelosi ordered Thursday that portraits of four of her predecessors be removed from the U.S. Capitol because they were part of the Confederate cause during the Civil War.

At a time when much of the nation is questioning certain monuments and memorials, it’s good that Pelosi is directing her gaze around the Capitol. But the issue of the speakers’ portraits is more nuanced than the controversies surrounding statues and other memorials to Confederate generals and secessionists who took up arms against the U.S. government.

And Pelosi ought to rethink what she’s doing.

The portraits of four speakers are in play: Robert Hunter, a Whig from Virginia; Democrats Howell Cobb of Georgia and James Orr of South Carolina, who were speakers of the House before the Civil War; and Democrat Charles Crisp of Georgia, who rose to that post after the war’s end. All four men served in the Confederacy, although Hunter, Cobb and Orr did so after their tenures as speaker; Crisp was elected to the House well after a general amnesty issued by President Andrew Johnson in 1868.

But here’s the problem: None of those men’s portraits are hanging in the Capitol because of their Confederate roles. The portraits are not intended to celebrate their personal acts of rebellion, but to place them within their historical context as leaders of the House of Representatives.

Context is key here. Much like memorials to George Washington and Thomas Jefferson, who were both slave owners, these portraits hang not because the subjects fought in a war against their own country, but because of their roles in governing that country.

That Washington, Jefferson and other early national leaders were part of, and profited from, slavery is important knowledge for framing how we view them today, and should not be overlooked or downplayed.

But the nation did not erect memorials to them, or place their images on currency, because of their embrace of slavery. They are feted for their contributions to the country. And if, say, it turns out the Jefferson Memorial was built to celebrate Jefferson’s ownership of slaves, I would say remove it.

Pelosi is removing depictions of people who preceded her in the speaker’s chair for reasons that have nothing to do with why their portraits are there in the first place. The proper corrective would be through informational plaques.

We need to look at what we commemorate and why much more closely than we have in the past, and work to remove from public places Confederate memorials that were erected as propaganda.

But we also must make certain we don’t overreach, and we must make certain not to cross the line between correcting representations of history and erasing them.

Scott Martelle, a veteran journalist and author of six history books, is a member of the Los Angeles Times editorial board.