The first item of business here is to identify which contaminants are affecting the castings and then find the source of the contaminants. After identifying the contaminant source, it will be easier to provide a solution to this dilemma. You mentioned dirt, metal and dust as being sources of these specks in the coating, but are they all present on each reworked part? Although it may appear to be a speck or dirt, if it is not clearly visible below the coating, try to cut around one of the specks with a razor blade and look at the substrate side of the coating with either a jeweler's loupe or a 10X magnifying glass. Based on your observations under magnification, it may be possible to isolate the type of contaminant and help you establish a baseline for further investigation into the root cause of contamination. Quite possibly, you may be able to determine if the substrate is blemished or if the culprit is a metal chip, dirt or dust. This would be a good starting point.
You didn't mention if you were cleaning the parts prior to recoating them. The more involved your cleaning process is, the better off you will be. If you are only doing a solvent-wipe, are you using a clean white cloth for each casting? If these castings are run through a multiple stage pretreatment, is the cleaner stage set above 185°F? Some mold release agents may remain embedded within the pores and on the surface of the casting if the cleaner stage temperature is too low. The residual mold-release contaminants may then outgas during the cure cycle.