My standard cure for this is to get my 3-jaw puller on it tight and then pop the shaft of the puller with a smart blow from a 'live' hammer (not a deadblow -- I like a traditional ballpeen). If it's nice and tight and you pop it smartly, the problem becomes catching the stuff before it hits the floor.
If the key has become bound this will be more of a challenge. Someone asked if the wheel is loose or tight on the shaft . . . this is an important detail.