Obtaining sufficiently accurate geometric descriptions is a crucial prerequisite for reliable particle transport calculations. Conventional transport algorithms on Cartesian grids use a highly efficient sweep technique and numerous mature discretization methods despite their modeling deficiency for complex geometries. To achieve a more accurate geometric description, a cell-based nonmatching Cartesian grid algorithm is proposed on the basis of the multilevel octree architecture. Transport sweep is performed according to the tree branch relationship of nested mesh distributions. The angular flux transmission between discontinuous grids is handled by the flux spatial moment mapping technique, and multiple zero-order mapping schemes are developed, including finite element interpolation, distance interpolation, and exponential fitting methods for treating upwind flux distributions of different relative shapes. The first-order mapping schemes are modified and improved for linear and exponential short characteristic discretization methods. The mapping accuracy is evaluated for polynomial and exponential functions, and a new spatial shape factor is presented for measuring the degree of nonlinearity. The multilevel octree grid (MLTG) algorithm is tested for neutron transport benchmarks, and good agreement with Monte Carlo and standard SN results is achieved. The number of meshes in the VVER shielding model is reduced from 18 million to 2 million using 3-level octree grids with the same geometric description accuracy. Numerical verification of a one-group fixed-source problem shows that 4-level and 5-level MLTG results with proper spatial discretization schemes can achieve relative deviations of less than 3% and 5% for detector region flux, respectively.