WASHINGTON - House Republicans plan to huddle behind closed doors Thursday afternoon to work toward the election of a new House speaker after Majority Leader Steve Scalise narrowly won the party's nomination in a private ballot at the Capitol Wednesday.

During the closer-door session Thursday, Scalise will address the conference, sources told ABC News' Senior Congressional Correspondent Rachel Scott.

The House adjourned Wednesday night with no scheduled floor vote for a new speaker, further complicating Scalise's path to the gavel. On Wednesday, House Republicans narrowly elected Scalise -- pushing aside Rep. Jim Jordan, R-Ohio, a firebrand and chairman of the House Judiciary Committee. 
 
With all Democrats supporting Rep. Hakeem Jeffries, D-N.Y., Scalise can only afford to lose four votes on the floor (depending on absences) and still win the speakership.

By ABC News' count, at least 13 Republicans don't plan to back him -- the latest sign that Republicans are still deeply divided and fractured after the removal of Kevin McCarthy, R-Calif., as speaker. Scalise would need 217 votes on the House floor to be elected speaker.

McCarthy isn't doing much to help Scalise, he told reporters Thursday morning.

"It's not an easy task, you've got to listen to people ... but time is of the essence. There's not that much time left," McCarthy told Scott.

When asked if it's possible for Scalise to get the votes needed to be speaker, McCarthy said "it's possible."

"...It's a big hill though. He told a lot of people who would be at 150 [votes] and he wasn't there," McCarthy said.

The latest GOP holdouts come from across the party spectrum -- from deep-red rural districts to more moderate suburban regions -- and had a range of explanations for why they wouldn't back Scalise.

Rep. Marjorie Taylor Greene, R-Ga., suggested she won't back Scalise because he has cancer. The Louisiana congressman is currently undergoing aggressive treatment for blood cancer.

"We need a speaker who is able to put their full efforts into defeating the communist Democrats and save America," Greene wrote on X, formerly known as Twitter.

Rep. Nancy Mace, R-S.C., who, like Greene, still plans to vote for Jordan on the floor, says she can't vote for Scalise after learning that Scalise attended an event linked to a white supremacist group in 2002.

"Especially given what's happening in Israel right now, I just cannot support someone who's associated with anything that divisive, whether it's race or religion," Mace said to reporters.

In 2015, Scalise came under fire for attending and speaking at a workshop organized by an alleged white supremacist group in 2002. A Scalise spokesperson told ABC News at the time that he didn't remember attending the event, at which he reportedly gave a conventional stump speech, and he may not have been aware of its affiliation. A local blogger once said Scalise described himself as "David Duke without the baggage," but it's not clear that Scalise ever described himself that way publicly.

While Scalise has won over some of McCarthy's critics, such as Reps. Matt Gaetz, R-Fla., and Anna Paulina Luna, R-Fla., he may have alienated some of McCarthy's loyal allies, such as Rep. Carlos Gimenez, R-Fla., and Lloyd Smucker, R-Penn.

"He's going to have to give us a message or an understanding of how he's going to bridge that gap and make certain that he brings Congress together and not divide the Republican conference more," House Intelligence Committee Chairman Mike Turner, R-Ohio, said on CNN.

"Steve's going to have to talk to them all, see what their concerns are. But I'm supporting Steve," McCarthy said as he left the Capitol Wednesday.

Jordan, who was the first to officially launch a bid to become speaker, said he plans to vote for Scalise on the floor and is encouraging his colleagues to do the same, multiple sources told ABC News.