The helical coils in the Large Helical Device (LHD) are large-scale superconducting magnets for heliotron plasma experiments. The cooling system of the coils was upgraded in 2006 to improve the cryogenic stability of the coils using subcooled helium as the coolant. In the present study, characteristics of the upgraded cooling system have been investigated and the steady-state operating method in which the subcooled helium of 50 g/s at 3.2 K is supplied stably to the coils has been developed. The supplied helium is subcooled via a heat exchanger in a saturated helium bath. The bath pressure and the temperature are reduced by a series of two centrifugal cold compressors. Based on the measured characteristics of the subcooling system, the optimization of the operating method has been performed using an automatic control of the mass flow rate through the cold compressors by the heater. Consequently, the designed mass flow rate and temperature were obtained and stable long-term operations have been achieved. The improvement of the cryogenic stability was also confirmed and the maximum average current of three blocks of the coils has reached up to 11.833 kA.