The temperature and pressure dependence of the thermal displacements and lattice parameters were obtained across the $\gamma \to \alpha$ phase transition of Ce using high-pressure, high-resolution neutron and synchrotron x-ray powder diffraction. The estimated vibrational entropy change per atom in the $\gamma \to \alpha$ phase transition, $\Delta S_{\mathrm{vib}}^{\gamma -\alpha }\approx (0.75\pm 0.15)k_B$, is about half of the total entropy change. The bulk modulus follows a power-law pressure dependence that is well described using the framework of electron-phonon coupling. These results clearly demonstrate the importance of lattice vibrations, in addition to the spin and charge degrees of freedom, for a complete description of the $\gamma \to \alpha$ phase transition in elemental Ce.