Getting paint of a carbon frame

Don't know how or who did it but there are some small flecks of white paint a few mm long on my matt carbon fork. The frames 4 years old and I'm not the best at washing it when I should but I still don't want the paint on it.
Any solutions for safely removing it from the fork without compromising the fork cos I'm building a house at the moment and can't get a new bike past the missus.