In the radio-frequency (rf) power transmission system of an electron cyclotron heating and current drive (EC H&CD) system, the gyrotron power should couple with the fundamental mode of the corrugated waveguide (HE11 mode) because unwanted higher-order modes affect the beam radiation characteristics, which is a problem in the quasi-optical launcher design. To achieve high HE11 mode purity, a beam coupling method that measures the transmission mode in the waveguide was examined using a 170-GHz high-power gyrotron for the first time. In beam coupling, the offset and tilt angle of the input beam at the waveguide inlet were minimized by controlling the angles of the mirrors in the matching optical unit (MOU) to minimize unwanted LP11 modes in the waveguide. The rf field profile in free space after 1.3 m of the waveguide from the MOU was measured, and the transmission mode content was analyzed. According to the analyzed mode content, the HE11 mode content was optimized by remote adjustment of the mirror angles with a digital controller. The optimization procedure of beam coupling achieved 95% of HE11 mode purity at the entrance of transmission line, which is the first demonstration that meets the criteria of the ITER EC H&CD system.