As a referee, you can decline to give a vouch at any point.
This may be because you either:
don't know the person who's sent the request
haven't known them for long enough
When you decline:
Vouchsafe will remove your information, and you will no longer get automatic reminders
The other person will be asked to choose a new referee