I'm really not trying to second guess you, but are you sure it's plastic? On my 350x and (85/86) 250r, it was definitely metal, and the setup looks almost identical in your picture. I guess regardless of material, I think you should be able to gently tap it out if you go from the outside with a relatively flat tool. I agree though, you don't want to break it! It would be a pain to have to make a new one. ABSOLUTE worst case scenario, if you found a junk 350x or (85/86) 250r caliper, you could harvest the metal one from those to use on yours. Given that the o-ring is the same size, the metal retainer from one of these should work.
- Frank
1984 200ES Big Red
1985 350X (x2)
1986 350X
1986 250SX
1984 Auto-X
1984 ATC70
1985 ATC70