The Dow Jones industrial average was down 75 points at 13,018 shortly after 10:30 a.m. It was wavering between small gains and losses before 10 a.m., when the Conference Board's report came out.
The Standard & Poor's 500 was down nine points at 1,410 and the Nasdaq composite index fell 20 points to 2,970.
The Conference Board said its consumer confidence index fell to 65.1, down sharply from 71.5 in November. That's the second straight decline and the lowest level since August.
"An unpleasant surprise," is how Hugh Johnson, chairman and chief investment officer of Hugh Johnson Advisors in Albany, N.Y., described the numbers. "People are worried."
Consumers' outlook for the next six months deteriorated to its lowest level since 2011. Lynn Franco, the board's director of economic indicators, said that decline is a signal that consumers are worried about the "fiscal cliff."
Investors are still digesting this week's news that the holiday shopping season so far has been the weakest since 2008, another sign that the economy is still struggling.
In Washington, the budget deadline neared with no resolution in sight. Higher taxes and government spending cuts kick in next week if Republicans and Democrats can't reach a budget compromise by Dec. 31. The uncertainty has made investors pull money out of stocks.
Senate Majority Leader Harry Reid said Thursday that it appears the government will miss the deadline. President Barack Obama called Reid as well as Republican Senate Minority Leader Mitch McConnell, Republican House Speaker John Boehner and Democratic House Minority Leader Nancy Pelosi to talk about the negotiations.
The "fiscal cliff" talks gained an extra layer of urgency late Wednesday with the news that the government is going to hit its $16.4 trillion borrowing limit — its so-called debt ceiling — on Monday.
Treasury Secretary Tim Geithner told Congress that he would use accounting measures to try to save approximately $200 billion. But those are stopgap measures, which would keep the government from going over its borrowing limit for probably only a couple of months.
Chipmaker Marvell Technology Group dropped 4 percent after the company lost a patent case brought by Carnegie Mellon University. Marvell said it would fight the $1.2 billion ruling.
In Japan, the benchmark Nikkei 225 index rose to its highest close since March 2011. The country is preparing for the incoming, pro-business prime minister, Shinzo Abe. He has called for more public works spending to reinvigorate the economy, and measures to drag the country out of deflation, or steadily declining prices.