You're trying to enforce a policy without much introspection or feedback on why that policy is useful. That's your problem, not the employees' actions.
@mary57 What i find so funny about this is if the only way they know whether you were at the office was because of a badge swipe proves your job doesn't need to be done at the office.