The new electron cyclotron resonance heating (ECRH) system at the ASDEX Upgrade tokamak allows for an adjustment of the polarization of the injected ECRH beam during plasma discharges. Three sniffer probes for millimeter wave stray radiation, with broad and polarization insensitive radiation characteristics, have been installed around the torus to monitor nonabsorbed radiation. The influence of varying ECRH-beam polarization on the detected stray radiation is studied. For perpendicular X2 heating the minimum detectable amount of wrong (O2-mode) polarization is found to be 5%. The system also allows full change of polarization from X2 to O2 mode, as it is useful for O2 heating above the X2-mode cutoff. These experiments show a high directivity of the stray radiation due to the toroidally inclined O2-mode injection.