No, Plama, the women he cheated with weren't just sneaked in and out the side door, they had access to his friggin' computer and office files. They are security issues. Do I personally think they were spies? No, but for all any of us know they could have taken top-secret info home with them. They could blackmail him or sell it to "interested parties" sometime down the road.
Do you have any proof of this "access"? I know one was an Army officer, but I seriously doubt even she had the access you claim.