There often appears unwanted elasto-optical birefringence in optical devices such as electro-optical, magneto-optical modulators and sensors when they are manufactured and used. This kind of elasto-optical birefringence causes unwanted effect for optical device and needs to be removed. Based on the method of index ellipsoid analysis, we theoretically analyze elasto-optical effects in various point groups of optical crystals and glasses, and accordingly propose some methods to eliminate unwanted elasto-optical birefringences in optical crystals and glasses. Main conclusions show that for orthorhombic biaxial crystal and the light wave propagating along any one crystalline axis of the crystal, if there is no shearing stress and the two external normal stresses applied to the other two crystalline axes can keep a constant ratio related to crystal parameters including refractive index and photoelastic constants, then unwanted elasto-optical birefringence can be eliminated from relevant optical devices, typical crystals include potassium titanium oxide phosphate (KTiOPO4, KTP) crystal and rubidium titanium oxide phosphate (RbTiOPO4, RTP) crystal, which are usually used as electrooptic Q-switchers in laser systems. For all the uniaxial crystyals such as potassium dihydron phosphate (KH2PO4, KDP), beta-barium borate ( -BaB2O4, BBO) and lithiun niobate (LiNbO3, LN) crystals, cubic crystals of 43 m, 432, m3 m point groups such as bismuth germanate (Bi4Ge3O12, BGO) crystal, and optical glasses, if the two normal stresses applied to the x1- and x2- crystalline axes of the crystal are equal to each other and there is no shearing stress, or there exists only one normal stress applied to the x3- crystalline axis of the crystal, then for the light wave propagating along the x3- crystalline axis, unwanted elasto-optical birefringence in relevant optical devices can also be eliminated. The above-proposed method to remove unwanted elasto-optical birefringence is benificial to design, manufacture, and usage of related optical devices.