Impurity-induced structural distortion in SmB6 is suggested by Raman spectroscopy study on impurity-controlled samples. Significant differences in the Raman spectra of single crystals SmB6(6N), synthesized with 99.9999%-pure boron, and SmB6(3N), synthesized with 99.9%-pure boron, are detected. While no noticeable differences are detected in the X-ray diffraction of SmB6(3N) and SmB6(6N), all the Raman phonon modes of the T2g, Eg, and A1g of SmB6(3N) are clearly broader and shift to higher wavenumbers than those of SmB6(6N), and the T2g and Eg modes of SmB6(3N) show doublet features. Based on the high-pressure Raman measurements and phonon calculation in uniaxial compression model, we argue that small amount of impurities in SmB6(3N) is enough to induce anisotropic distortion in the B6 octahedra, leading to peculiar behaviors in Raman spectrum of SmB6(3N). Our results may present a clue for understanding the current puzzles about SmB6 of various origins with different impurities.