Two-phase flow instability in parallel channels of a once-through steam generator (OTSG) is systematically analyzed by the modern frequency domain method. The mathematical expressions of heat transfer and flow for an OTSG are proposed, and the transfer function of the closed-loop system is deduced by the use of linearization and Laplace transfer. The OTSG's stability is judged according to the Nyquist Stability Criterion. The instability of OTSG in two cases (single-channel model and multichannel model) is researched, respectively, for a numerical example. The result shows the classical frequency domain method can be commonly used when the coupling effects in the system are negligible, and it is only a special case of the multivariable method. Furthermore, the stability sensitivity to the operating parameters is analyzed for the OTSG in this paper. The predicted results are in agreement with the experimental results given in the literature.