Simplify Telescope for 4.0

[humphd]

Overview

Telescope is great. I use it all the time to keep up with the amazing work my open source students do. I love that hundreds of people have had a hand in creating it, and I hope to keep it going for a long time.

It's also way more complicated than it needs to be. We've been very permissive with technological experiments. If someone showed an interest in adding a feature or using a particular technology, we typically found a way to support them.

However, 99% of those hundreds of students don't maintain the project, and a handful of us are left to keep the lights on.

We need to look at ways to simplify things so that we can maintain what we have, with the people that we have.

We also need to make Telescope easier to run locally. It's way more complicated to run on your local machine than it needs to be, and this means that new devs are scared off or simply can't help, because their computers aren't powerful enough to run all of the containers.

Goals

• Use fewer containers
• Consolidate services down into fewer technologies
• Make the CI runs go faster and fail less often
• Make the project easier to develop
• Make the project easier to deploy

Ideas

I'm not sure what all the right moves are here. @manekenpix and I have discussed a few things, and it would be good to get a list going, so we can start filing/fixing things.

Here are some starting points:

• Remove Supabase and replace with Postgres + Prisma. #3839
• Replace Elasticsearch with Postgres/Prisma for full text serach/indexing #3841
• Use a single proxy, either nginx or traefik, but not both.
• Remove the Planet service (we don't need it anymore)
• Remove the mobile app

That might be a good start. What else do people see that we could do?

[RC-Lee]

I just want to talk a bit about Elasticsearch.

While I personally would love to keep playing around with Elasticsearch in Telescope, I admit we haven't really been unlocking its full potential. There is also the problem of getting students to be willing to dive into the huge amounts of documentation and learn how to use and maintain ES. Another part of maintaining is dealing with hardware and limited disk space, where I am assuming (have yet to read about) that ES has more overhead than its competitors.

In particular, I think we've been missing out using it for analytics, for example, the most searched keywords, most topics that have been written on about, highest blogging contributors (besides Prof :P), etc. We also have only been using ES for indexing posts only, where instead we could also index our documentation and add search to that (Though, our documentation is quite sparse). Another common use case for ES is indexing logs and analyzing performance. I had also been contemplating using ES for the Hacktoberfest analytics issue (but I also see ways of not using ES for it as well).

Admittedly, I'm not sure if we have that much text and users to be considered "large volumes of data" where Elasticsearch seems to have the edge at compared to other competitors. It might be interesting to see how Redis and Postgress handles full-text search, and see if there will be noticeable time and performance differences.

TLDR: I'd be happy if ES stays, but it might also be a good opportunity to learn about other search alternatives.

[humphd]

I thought about you @RC-Lee when I filed this issue. I feel like we never realized the potential of ES in our stack, and if I had had a few more students like you, it might have been worth digging into more deeply. However, there's only one @RC-Lee, and sadly, the passion for ES in Telescope seems to have been closely tied with you.

There's no doubt we could do more, a lot more. But the cost is also huge. Running ES in development takes a lot of resources, and since we only really need a thin slice of what it does, I'm tempted to say goodbye and find ways to live without it.

[sirinoks]

Honestly, the most diffucult part of Telescope, personally, was docker itself. Debugging containers and making sure they run has been the biggest issue for me.

Make the project easier to develop

I also want to add how we lack general documentation - as, the general picture/overview of our services. I'm trying to do this for Starchart, as I have a chance to actually follow all the new parts as they get developed. However, with Telescope it's different. Most people that made those parts are not around anymore, and we have a lot of stuff here. Not just microservices, but also different projects completely. Like the dependency one, or docusaurus.

Speaking of docusaurus.. Did we end up hosting it, or does it only run locally..?

[humphd]

https://www.unixsheikh.com/articles/sqlite-the-only-database-you-will-ever-need-in-most-cases.html has me thinking we should use sqlite!

[Kevan-Y]

It might be out of the scope of this, but I'm wondering if we should clean up a project that is not maintained/development is stopped. Like the mobile app.

I found there is not much need to have a mobile app for the telescope, don't see any usage for it at the moment. So no point in hosting it on Google Play or App Store since those services cost money.

[manekenpix]

It might be out of the scope of this, but I'm wondering if we should clean up a project that is not maintained/development is stopped. Like the mobile app.

I found there is not much need to have a mobile app for the telescope, don't see any usage for it at the moment. So no point in hosting it on Google Play or App Store since those services cost money.

I agree, let's get rid of the mobile app.

[humphd]

Yes, let's rip it out.

[TDDR]

Hey @humphd, are you wanting this done before the next round of students come through in January? Or will this be the focus for them?

I'd like to start contributing again and this would be good experience.

[humphd]

I'm not sure what I'm going to focus on in January yet, but this is a potential for sure. If you're looking to contribute to this, we'd love to have you.