The Conference Board said Tuesday that its consumer confidence index dropped 8.1 points in January from December to a reading of 58.6, the lowest since November 2011.
The index has declined for three straight months since hitting a nearly five-year high of 73.1 in October 2012. It's still above the post-recession low of 40.9 reached in October 2011.
Conference Board economist Lynn Franco said the tax increase was the key reason confidence tumbled in January, making Americans less optimistic about the next six months.
For a worker earning $50,000 a year, take-home pay will shrink this year by about $1,000.
"It may take a while for confidence to rebound and consumers to recover from their initial paycheck shock," Franco said.
The index fell sharply in December as congressional Republicans and President Barack Obama moved closer to the fiscal cliff without reaching a resolution on sharp spending cuts and tax increases.
Congress and the White House ultimately struck a deal on Jan. 1 to prevent income taxes from rising on most Americans. But they delayed the spending cuts for only two months. And they allowed a temporary cut in Social Security taxes to expire.
The survey was conducted through Jan. 17, at which point most people began to realize their paychecks were lighter.
Consumers were less confident in January than December about current economic conditions, the survey showed. And their outlook for the job market also grew more pessimistic.
Most economists attributed the drop in confidence to the increase in payroll taxes.
Joshua Shapiro, chief U.S. economist at MFR Inc., also noted that sharp divisions in Washington over spending cuts and tax increases likely made consumers less optimistic about the economy, too.
"All the negative news about the dysfunction in Washington surrounding the fiscal cliff negotiations contributed to the December plunge, and ongoing shenanigans concerning the debt ceiling and fiscal sanity in general continued to weigh in January," Shapiro said.
Taxes are rising at a time when wages and salaries are barely growing. The combination is expected to hurt consumer spending and slow economic growth.
Many economists predict economic growth slowed in the October-December quarter to an annual rate of around 1 percent. That would be much weaker that the 3.1 percent rate in the July-September quarter. Most economists don't expect growth to pick up much in the first quarter of 2013.
The decline in confidence comes as the economy is signaling improvement elsewhere.
A recovery in housing market is looking more sustainable and is expected to strength this year.
A separate report Tuesday showed home prices accelerated this fall, pushed higher by rising sales and a tighter supply of available homes. The Standard & Poor's/Case-Shiller 20-city home price index rose 5.5 percent in November compared with the same month a year ago. That's the largest year-over-year gain in six years.
The U.S. auto industry and financial sector are also picking up.
Auto sales reached a five-year high of 14.5 million in 2012. Analysts expect sales will climb even higher this year, to 15.5 million.
Stocks are near their all-time highs. The Standard and Poor's 500 has more than doubled from its low in 2009.
Still, the job market remains sluggish. Employers have added an average of about 150,000 jobs a month for the past two years. That's enough for a gradual decline in the unemployment rate, which remains high at 7.8 percent.