A crystal lattice forms because atoms combine to form molecules, and molecules combine to form a lattice. Without outside interference the process works faultlessly. If energy is removed, the lattice deforms. Even if the deformation is non-isotropic, the deformation in any direction is related to the bond strengths in the lattice. When the energy returns, why would the factors that originally formed the lattice not cause it to return to its original form?

There never was nothing.