This paper describes new theory associated with using untrained neural networks to conduct regularization. The idea has been explored empirically before, but the theory provides new insight into why these ideas work. Some of the proofs are derivative of proofs in other contexts, and the key assumption (analogous to the RIP) is strong, especially since in the deep image prior makes this assumption much less interpretable than it would be in a sparse recovery problem.