"Californians have been gloomy about the economy and affordability for several years now, and it's altering their big-picture thinking," the survey's director said.