Never, if you are enjoying it! Just keep creating good quality content that is fun and engaging, and your audience will grow organically
Well done, you are correct!
Building a community can take time but is very worthwhile. Nobody should make you feel like giving up