Is it hard to tell the difference between a living organism and inanimate matter? No.
Of course it is. We often need lots of equipment and chemicals and education to do it. For thousands of years, our first and worst attempt at philosophy and science -- religion -- did not identify pathogens, or even coral as living creatures. We had to invent something better and apply our best minds to the task. Still today people study and work their entire lives to identify possibly one virus or eukaryote in their lifetimes. So no, you're ass backwards wrong.
And even if you were right, you're still wrong, because you are only speaking from objects and life you know about.
So, even when granted your incorrect premise, you're still wrong. That should serve to make you understand how wrong you are.
Also, we would have had a very hard time indeed deciding what was the FIRST life, if we could have monitored abiogenesis.
So, basically, you're wrong on every level possible.