A magnetic thrust chamber concept in a laser fusion rocket is suitable for controlling the plasma flow, and it has an advantage in that thermalization with wall structures in a thrust chamber can be avoided. Rayleigh-Taylor instability would occur at the surface of expanding plasma, and it would lead to the degradation of thrust efficiency, which would result from diffusion of the plasma through an ambient magnetic field. A three-dimensional hybrid particle-in-cell code has been developed to analyze the plasma instability in the magnetic thrust chamber and to estimate the thrust efficiency. It is found that the instability would not have serious effects on the thrust efficiency; thrust efficiency in terms of momentum obtained here amounts to 65%. The effects of varying parameters on the thrust efficiency are also studied. The thrust efficiency seemed to reach its maximum value around c = 50 deg, where c is an angle subtended from the initial plasma position at the z axis to the solenoidal coil and its dependence on magnetic field energy produced by the coil is found to be weak for the cases studied here.