The deterministic methods are efficient for solving the neutron transport equation (NTE), but suffer from discretization errors. The increasingly complex geometric models make spatial discretization errors the primary source of discretization errors. Considering that spatial discretization errors come from inaccurate geometric shape descriptions and low-accuracy numerical schemes, this paper develops a Discontinuous Galerkin Finite Element Method for the NTE on unstructured polygonal meshes to reduce spatial discretization errors. In this method, the physical modal basis is adopted to handle the polygonal mesh and to achieve high-order accuracy in a uniform and efficient way. The numerical results of various fixed-source and k-eigenvalue benchmarks demonstrate that the method developed in this paper can give accurate solutions on polygonal meshes with high convergence rates.