While the folders marked with classification levels aren't in and of themselves classified, they ARE considered sensitive, meaning there has to be accounting for them, and they are generally stored in a central place, usually in a secure area.
And no, not once in my career were empty folders with classification markings on them left just lying around. If there was a folder with a classification marking on it, or cover sheet, it usually had material in it. When the material was taken out, the folders or cover sheets were returned to the secure area for storage. We NEVER let them just lay around. Matter of fact, if a folder or cover sheet was found on someone's desk without anything in it, the first thing the supervisor asked was 1) why was the folder/cover sheet just lying around, and 2) where was the material that was attached to it? Questions were asked, and searches were made to make sure that the folder/cover sheet was in fact unused, and it was promptly returned to it's storage space. If nobody knew why it was there, everything stopped and a search was made to make sure that classified documents weren't just lying around uncovered. Usually, finding a classified cover sheet/folder without anything in it resulted in an ass chewing for the person who last had it in their possession for not properly returning it. So yeah, while empty folders and cover sheets aren't classified in and of themselves, they are still treated as sensitive, and steps are made to make sure they are accounted for.
Some even have serial numbers which are logged in so that proper accounting can be made for the material they contain.