This study establishes a theoretical framework for realizing and dynamically controlling magnon and optical bistability in a hybrid cavity optomagnomechanical system composed of microwave cavity mode, magnon mode, phonon mode, and optical cavity mode. The objective is to investigate the synergistic interplay among self-Kerr nonlinearity, magnetostrictive effect, and radiation pressure induced optomechanical coupling in generating and modulating bistable behavior. Furthermore, this work aims to reveal the transient quantum state transition dynamics between bistable states. The system Hamiltonian incorporates magnetic dipole interaction between the magnon mode and microwave cavity mode, magnomechanical interaction between the magnon mode and phonon mode, and optomechanical interaction between the phonon mode and optical cavity mode. In addition, the self-Kerr nonlinearity of the magnon mode is considered. Numerical analysis of the system dynamics is conducted using quantum Langevin equations that include dissipation and input noise terms. Steady-state analytical solutions for the average magnon number and optical photon number are derived, revealing a bistable characteristic with three possible solutions. Numerical simulations are performed using experimentally feasible parameters, including coupling strengths, frequency detunings, and dissipation rates. Results indicate that both magnon and optical bistabilities are tunable. Specifically, adjusting the microwave cavity–magnon coupling efficiency enables modulation of the energy transfer efficiency from microwave to magnon, thereby altering the hysteresis window and excitation threshold of the magnon bistability. Tuning the magnon-phonon interaction can influence the energy transfer from magnon to phonon. A larger magnon-pump detuning enhances nonlinear frequency shifts, alters energy transfer pathways, broadens the hysteresis loop, and increases the magnon population on the upper branch of the bistable curve. Higher magnon dissipation rate hinder the accumulation of nonlinear effect, narrowing the bistability window and shifting the threshold to higher pump powers. For optical bistability, stronger optomechanical interaction reduce the effective cavity loss and weaken the nonlinear response to the pump field, leading to a decrease in the amplitude of bistability and a narrowing of the hysteresis loop. Increasing the optical cavity–pump detuning suppresses energy transfer efficiency, necessitating higher pump power to achieve the same photon number, thereby enhancing the prominence of the bistability. Elevating the optical cavity dissipation rate requires stronger driving to compensate for photon losses, resulting in a narrower hysteresis loop and a rightward shift of the threshold. Sharp vertical jumps observed in the bistability curves correspond to instantaneous transitions at critical driving points, enabling switch-like behavior. Moreover, transient dynamics obtained by numerically solving the Langevin equations reveal time evolution of magnon and photon numbers under nonequilibrium initial conditions. Within the bistability regime, the system exhibits quantum state transitions between low and high steady states. Transition rates are determined collectively by system parameters. Therefore, this study provides a theoretical platform for multi-parameter cooperative control of magnon and optical bistability. The tunability mechanisms are governed by the joint action of coupling strength, detuning, and dissipation rate. The controllability of bistability thresholds, hysteresis widths, and transient quantum state transition dynamics demonstrated in this work highlights the significant potential for applications such as tunable optical switches, quantum information processing devices, and fundamental studies of nonlinear quantum dynamics in hybrid system.