A boiling water reactor (BWR) with high power density (BWR-HD) was designed through an optimization search that was constrained to a square lattice fuel array. It has a power level of 5000 MW(thermal), equivalent to a 26% uprated Advanced BWR (ABWR), the latest version of operating BWR. This results in economic benefits, estimated to be [approximately]20% capital and operations and maintenance costs and similar total fuel cycle cost per unit electricity. The stability of the ABWR and BWR-HD were assessed for the three modes of density wave oscillations: single-channel thermal hydraulics, coupled neutronic regional core oscillations, and coupled neutronic global core oscillations. The sensitivity to design parameters such as inlet subcooling, presence of water rods, and inlet orifice coefficient as well as to changes in reactor power, flow rate, and void coefficient were examined using the STAB frequency domain code. The BWR-HD's stability performance and sensitivity were concluded to be similar to those of the ABWR. The results of the frequency domain analysis indicate that the shorter core and smaller void coefficient lowered the oscillation decay ratio, while the cooler inlet temperature and higher void fraction increased the decay ratio. Also the S3K code was utilized to perform three-dimensional coupled stability analysis and to formulate an operation exclusion zone region for the BWR-HD design. It was found that a reduction in the allowable operational zone of the BWR-HD design is warranted, due to its decay ratio being higher than that of the ABWR for whole-core oscillations. However, the inlet orificing (pressure loss coefficient) of the assemblies can be increased to obtain the same stability performance as the ABWR. This strategy is deemed plausible since the pumping power needed for the BWR-HD, even with the increase in pressure losses at the inlet of assemblies, will still be less than that of the ABWR and will have negligible effects on the safety performance.