My plan was to take a screw driver and just collapse one side and pry it out, does anyone see any problem with that? Thanks

Nope. That will work but make sure to make a sleeve for the inserting of the new seal. I think I got a chunk of exhaust pipe wrapped electrical tape around the edge for padding.