Long-standing problems of assigning uncertainties to scientific data became apparent in recent years when uncertainty information (“covariance files”) had to be added to applications-oriented large libraries of evaluated nuclear data such as ENDF and JEF. Questions arose about the best way to express uncertainties, the meaning of statistical and systematic errors, the origin of correlations and the construction of covariance matrices, the combination of uncertain data from different sources, the general usefulness of results that are strictly valid only for Gaussians or only for linear statistical models, and so forth. Conventional statistical theory is often unable to give unambiguous answers and tends to fail when statistics are poor, making prior information crucial. Modern probability theory, on the other hand, incorporating results from information, decision, and group theory, is shown to provide straight and unique answers to such questions and to deal easily with prior information and small samples.