That would be fine! The guards are just there to ensure character safety so once they leave, the guards won't pursue. Repeated attempts, however, will result in the guards coming to find them and asking them to stop for their own safety.