The issue with DU is that if you put a 30mm DU round, or a 120mm APFSDS round, through an armored vehicle some of it will get ablated on the way through and settle in fine dust inside (and outside in a fan from the exit hole). That dust can get blown around, or breathed by folks looking over the wrecks for salvage and breathing in Uranium is not good. As AJ said, heavy metal toxicity is a bigger problem than any radiation.
You can hold a chunk of it with no ill effects. Just don't grind on it or give it to a Marine (they put everything in their mouths).