I am not talking about judicial activism in the sense of 'judges disregarding precedent' or 'judges disregarding the opinion of the legislature'. I have no problem with judicial activism in that vein. This is the judges' job.
Which brings us to the current problem. There is an established body of laws, regulations etc. that has accumulated over decades under an expansive view of the Constitution, started out by those same Holmeses and Braideises in the 1930's. It has gone so far now that had we abandoned the 1940's expansive interpretation of the Constitution now, everything would come crashing down in a pile of smoke and fail.
Clarence Thomas had once explained that up to 70% of the Federal regulations would be unconstitutional in a strict reading. This might be wrong if you evaluate them page by page - I am not an attorney - and perhaps the real number is more like 10%. What's certain is that this would alter the social fabric of our society in various ways, and the Court would be seen as politically responsible.
This preference by the court to keep the long-standing system, even if it is, on its face, in conflict with the Constitution is in itself a policy preference. With Scalia, we've often seen his policy views as a conservative further bias him (like taking the reverse of the position he took in Lopez during Raich, for example).