Rumor has it that the entire human species is descended from a family group of about 600 breeding individuals who systematically removed all other human-like species as they spread across the globe.
Not sure about your numbers, but everything I've read on the subject of DNA analysis suggests that humankind was down to a very small number of human beings TWICE, not once.
The first time it happened, every surviving human being was the offspring of a single female.
The second time it happened (some thousands of years later) the only suriviving humans were the decendents of a single male.
So yes, your argument that our species is probably entirely the offspring of a very limited gene pool seems valid.
But since that time (roughly the last 40,000 years or so) the gene pool has probably expanded as helpful (or at least benign) genetic mutations have been introduced into the pool.
Generally speaking and over the longer run (multiple generations) , I think incestuous mating is a bad idea for the species.
Study of populations who suffered from limited mating propects for many generations seems to prove that contention.
Those populations tend to suffer from more genetically related diseases and conditions that the MUTTS (that represent the rest of humankind) do not.