Macworld explains how not to get bitten by malware
We recently mentioned the new OS X malware that's floating around the (nether side) of the net these days. Over at Macworld, Rob Griffiths has an extensive article discussing the ways you can tell if a piece of downloaded software is fishy. The tips range from the obvious (only download from trusted sources) to the arcane (diving into packages to examine the installer components). The overall strategy is to examine the software carefully and look for tell-tale signs that it's not legitimate.
In any case, it should give you a good set of strategies to use when evaluating a questionable download.