A mathematical model for mass flow in a transmutation system has been established for a chain of two transuranic (TRU) radionuclides. The nonrecursive solutions for the fractions of the two TRU radionuclides in the transmuter core before and after the irradiation in the i'th cycle have been obtained by the similarity transformation. With the nonrecursive analytical solutions, the TRU reduction ratio has been formulated as a performance measure for the system. The stability of the system has been defined in terms of the moduli of the eigenvalues of the system. The conditions for a stable system and for a system to reach a quasi-steady state with fewer cycles have been shown in terms of the system parameters. A large value of the nondimensionalized destruction coefficient d is beneficial for effective waste reduction because (a) the system reaches a quasi-steady state faster; (b) the TRU mass in the waste can be reduced more effectively; and (c) the precursor effect becomes negligible, and each radionuclide can be approximately treated as a single radionuclide without precursors.