Good communities means good users. As internet got flooded with normals through the 2000s, the good users had two options: Suffer the eternal september or move on with your life. Every public community on the internet today is vulnerable to normalfag lynch mobs. These mobs will rise up against you if you are in any way politically incorrect (impossible to avoid in any serious discussion these days) with only the slightest agitation, but they will also rise up if you are mean to low quality users (they'll complain about elitism, bullying, offensive language, meanness, people will try to troll you because they think you're a pretentious faggot etc).
So basically, if any joe shmoe on the internet can access your community, it will quickly turn to shit. See 4chan and forums with public registration. The more popular the forum, the worse this will get, since at some point the owners start caring about the income stream more than the pride they take in their community, and then they act with this mentality of trying to attract as many users as possible for the banner clicks/views, which means cater to the lowest common denominator (and fuck over anyone who isn't welcoming the lowest common denominator because they're concerned about board quality). The critical mass is different for everyone. Some sell out at the 10 users mark, some hold out for ages. Hotwheels so far has been remarkably commendable, but I have no doubt that he will eventually fall into the same "let's be more newbie friendly" trap as moot (either that or this site will die).
If registration is not as easy as creating a free account, it still won't make a difference if the barrier is trivial. See for example Something Awful ($10 fee for account) and demonoid (private tracker which occasionally opened registration and never banned anyone). Both started okay but quickly turned to shit in the same way that free forums do. The reason is that from the economics stand point, it really doesn't make a huge difference that you need to pay $10 or track down a demonoid invite first. Millions of people are still able to do it, and the same race to the bottom happens as soon as the dollar sign lights up in the admin's head.
If registration is actually difficult, but not very exclusive, the decline will be slow (and your individual users better be fucking quality contributors because there's not gonna be many of them one way or another) but still happens. See what.cd and similar private trackers that take themselves seriously (by enforcing rules, severely limiting invites, requiring a test). Even if you have 100 users on a tracker like what.cd, who's to say one of those doesn't turn out to be faggot and sell an invite to some richfag who doesn't give a fuck and start spreading it further to his other friends? Who's to say someone doesn't go full SJW and "break the news" in a Vice interview, leading to normalfags flooding in by the millions?
These days, an internet community needs to be locked down really tight to avoid being fucked with by the mainstream. The community must all know each other on some personal level (maybe not real identity but personality etc) and have some mutual assurance of integrity so that nobody betrays the community to outsiders. This is still possible online, but very difficult. So difficult, that at this point meatspace communities are just easier.
So all the competent, intelligent people have used their intelligence and talents to place themselves in a cushy career, where they interact with many others like themselves. They form informal, IRL communities with such people, communities that you cannot join online. Some people failed, either out of autism or sheer bad luck - you can sometimes find them in places like this board: Tragic cases where the person clearly belongs somewhere better, but for whatever reason ended up here instead.
Once you do get the community rolling, word inevitably gets out about a sekrit klub being run by these assholes who have the nerve to think they're better than you, and you should go troll there to teach them a lesson. This is after you actually do have your community functioning and being active - if you cannot solve this problem, you will see everything you have labored to build turn to shit thanks to the flood.