In the system of electron cyclotron heating, highly overmoded, corrugated circular waveguides are used. To analyze propagating mode content in the waveguide, burn patterns of the thermal paper placed on the waveguide aperture are observed at several positions. Theoretical burn patterns are obtained by taking into account a nonlinear grayscale response of the thermal paper to the calculated power profiles. We have developed a new method of mode analysis by nonlinear optimization, which is based on an iterative error reduction of differences between observed and theoretical patterns. To examine the status of polarization, the transformation between hybrid modes and linearly polarized (LP) modes is derived. The method is applied to the 82.7-GHz transmission line connected with the gyrotron. The propagating wave is linear polarized and consists of [approximately]4% of the LP11 odd mode, [approximately]95% of the LP01 mode, and [approximately]1% of other modes. The calculated burn pattern is similar to the observed one, like a plateau. By using both center of power and weighted averages of the perpendicular wavenumber in these profiles, offset and tilting angles of an injecting electromagnetic beam to the waveguide entrance are inferred. These are verified to be consistent with the results by the coupling code of a Gaussian beam with hybrid modes.