Ok end result. I drilled them out more and then beat with a hammer. Probably not the best or correct method but it did work. Now I've got to take off the 6 pin connector so I can fit the wire into the new arm. You need to remember to mark the wire placement in the connector. So once you rethread the wire you can put them back into the connector in the correct order.
Good times.