A method for finding a set of modular, poloidally closed stellarator coils confining a plasma in a given equilibrium configuration is described. The proposed technique performs a direct nonlinear optimization of the coil shapes with respect both to the desired structure of the magnetic field and to geometric constraints required by the fabrication process of the coils. This is in contrast to the method employed successfully for the design of the coil system of experiment W7-X, which divides the minimization of the field error and the adjustment of the geometric coil properties into consecutive steps. The viability of the new method is exemplified by two alternative coil designs for the plasma configuration of W7-X, offering more space inside the coils for installation of the divertor system or a blanket. The results are compared with the original coil configuration designed for W7-X.