An autoregressive moving average model of neutron fluctuations with large measurement noise is developed from the Langevin stochastic equations with the noise equivalent source in the form of a vector Wiener process. The neutron field/detector interaction is explicitly treated, and delayed neutrons are included. The Kalman filter with nonzero covariance between input and output noise is applied in the derivations to reduce the state-space equations to the input-output form. Theoretical developments are verified using time series data from the prompt-neutron decay constant measurements at the zero-power reactor RB in Vinča. Model parameters are estimated by the maximum likelihood off-line algorithm and an adaptive pole estimation algorithm based on the recursive prediction error method with implemented regularization and stability control. The results show that subcriticality can be estimated from real data with high measurement noise using a shorter statistical sample than in standard methods based on the power spectral density or the Feynman variance-to-mean ratio method.