Magnetorheological (MR) bearings, with their field-controllable rheological properties, offer new possibilities for control of rotor instabilities. However, their nonlinear dynamic behaviors and the underlying physical mechanisms governing these instabilities remain insufficiently understood. This work develops a coupled MR bearing-rotor system model, where the oil film force is derived from a novel bilinear constitutive equation to capture the field-sensitive shear behaviors of MR fluids. Complex nonlinear dynamic behaviors including period doubling, quasi-period, and chaos are revealed, which emerge from the interaction between oil film vortex dynamics and magnetic excitation. The critical instability mechanism is identified from the evolution of intrinsic dynamic characteristics of MR bearings. When the whirl speed within the oil film reaches approximately half of the rotor speed, the damping force balances the destabilizing force, thereby defining a critical threshold beyond which the system transitions to instability. This threshold can be effectively tuned by adjusting the excitation current, which modifies the yield stress of MR fluids and consequently regulates the damping force. As a result, the nonlinear vibrations of oil whirl and whip can be suppressed, and the system stability can be significantly enhanced. These findings provide both theoretical insight and practical guidance for the design and control of MR bearing supported rotor systems.