CHARLOTTE, N.C. — Billy Graham didn't have a church to pastor. His congregation, for much of the last 50 years, was America's first families — the presidents, their wives and their children.

President George W. Bush says it was a walk with Billy Graham that led him to stop drinking. Former first lady Hillary Clinton says the Charlotte-born evangelist urged her to forgive her husband's sexual encounter with Monica Lewinsky. And Ronald Reagan's widow says she knows her husband will be waiting for her when she dies because "Billy says so."

A new book out also says he often crossed the line from pastor to political adviser, going on partisan errands and allowing himself to be used by presidents looking to piggy-back on his popularity.

History is starting to weigh in on Billy Graham, now 88, frail and living quietly in the mountains. The focus is on Graham's relationship with 11 presidents, from Truman to Bush II, several of whom the evangelist befriended before they reached the Oval Office.

Though Graham always insisted he was neutral, he was a behind-the-scenes consultant on Richard Nixon's three presidential campaigns — advising his GOP friend on which states to focus on and why he should name a Protestant running mate in his 1960 battle with Democrat John Kennedy, a Catholic.

The sting of Watergate caused Graham to lower his profile, but he sent election-eve signs that George H.W. Bush in 1988 and George W. Bush in 2000 had his blessing.

Graham was close to some Democrats, too.

In 1969, Graham spent the night at the White House on the last day of Democrat Lyndon Johnson's term and stayed again the next night, on Republican Nixon's first day.

Besides the new book — "The Preacher and the Presidents: Billy Graham in the White House" by Michael Duffy and Nancy Gibbs — other books are on the way. Duffy and Gibbs are editors at Time magazine, which is running a cover story on Graham.

Even the new Billy Graham Library in Charlotte is part of the spotlight on Graham's many pastoral — and political — visits to the White House.

Walking through the library "is like entering a time tunnel into presidential history," Gibson tells viewers.

Did Graham get too close to power by refusing to criticize the president, even during the Vietnam War and the Watergate scandal? Or did Graham provide just the solace these presidents wanted?

"Presidents need the comfort that faith gives," former President Bush, a Republican, says in the TV special. "Billy Graham was a great dispenser of comfort."

Johnson, demonized by protestors during the Vietnam War in the 1960s, recalled how Graham's visits picked him up: "When I was being called a crook and a thug and all, (he'd invite Graham to the White House) and we bragged on each other. I told him he was the greatest religious leader in the world and he said I was the greatest political leader."

And Reagan, after being shot by John Hinckley in 1981, called Graham from his hospital bed. Says wife Nancy: "He wanted to forgive Hinckley."

Reagan later helped Graham when he needed it, endorsing the evangelist's controversial decision to preach in the Soviet Union — one of several times presidents helped open doors for Graham crusades overseas.

But scholars and journalists predict history may fault Graham for straying over the line.

They say Graham's temptation wasn't money or women — the downfall of other evangelists — but his need to be close to the powerful.

Until Graham became a world-famous preacher in the early 1950s, evangelicals were considered outside the mainstream. To have one of their own welcome at 1600 Pennsylvania Avenue was a boost to the movement — and to Graham, who started his visits to the White House in 1950 in hopes of sparking a national religious revival that would include President Truman.

In succeeding decades, he kept coming, to pastor and to be at the center of things.

"No one else had this kind of access over so many years and I think that Graham knew he had to protect it," says "The Preacher and the Presidents" co-author Duffy, a former Observer news intern who is an assistant managing editor at Time. "He couldn't risk it. So he didn't challenge presidents, he didn't speak truth to power."

Randall Balmer, a professor of American religious history at Columbia University and author of the forthcoming "God in the White House," goes further.

"Such a lost opportunity," Balmer says. "To have the ear of these extraordinarily powerful people but, when all is said and done, to have very little influence on their personal behavior or public policy. Instead, he was kind of just falling all over these guys."

Graham repeatedly cast Nixon as religious and defended him against charges of dishonesty in 1968. "I can testify that he is a man of high moral principles."

But Duffy and Gibbs report that when Nixon established Sunday religious services in the White House in 1969 — Graham presided at the first one — the president ordered aides to use them for campaign fundraising.

Will all this attention to Graham's occasional stumbles affect his image?

No, says William Martin, who has updated his definitive biography of Graham — "A Prophet with Honor" — for a forthcoming edition.

Airing information that Graham played politics and was used by politicians "would have been more stunning in 1960 than 2007," says Martin. "We've had 50 years of the paring down of some pretty historic figures. No one expects public figures to be spotless."