Permanent deformation in solids results from atoms not aligning with the external stress causing the deformation. Detecting such non-affine atomic rearrangements and connecting them to measurable mechanical effects is now shown to be feasible by means of high-energy X-ray diffraction.