Active switching of plasmons by an external magnetic field is demonstrated in a metal–ferromagnet–metal structure. The strong modulation, combined with possible all-optical magnetization reversal induced by femtosecond light pulses, opens the door to ultrafast magneto-plasmonic switching.