Airbyte vs Meltano, OS Book, Kestra; ThDPTh #58
Airbyte vs Meltano, OS Book, Kestra; ThDPTh #58
I got a lot more feedback for both, the ThDPTh as well as the Thoughtful Friday last week, so thanks for that! Every piece of feedback helps me make this newsletter better. Keep it coming.
I’m Sven, I collect “Data Points” to help understand & shape the future, one powered by data.
Svens Thoughts
If you only have 30 seconds to spare, here is what I would consider actionable insights for investors, data leaders, and data company founders.
New data orchestrator called Kestra. There is an enterprise version behind it, it is quite mature and aims to be very scalable.
Airbyte vs. Meltano. Both tools still are not quite there yet if the target customer is a mature data engineering team. At least that is the feedback from the Kolibri Games team.
Open Source Book. Nadia Eghbal wrote a light must-read on how open source works.
Kestra a new data orchestrator
What: There seems to be a new data orchestrator in town called Kestra. With a focus on scalability and declarative setup. The first public release appeared on GitHub beginning of 2022. The tool is used in production at Leroy Merlin in a medium-sized deployment.
My perspective: I haven’t done a test drive yet, but the tool appears to already contain quite a few features, given that most of the work on it happened behind closed doors, something that I always find perplexing.
I am always amazed at how many tools are created simply out of the frustration of using Apache Airflow. I remember both prefect & dagster sharing a similar frustration with certain dimensions of Airflow. Although I find it hard to believe that for the deployment mentioned by the Kestra crew at Leroy Merlin, Airflow fails to deliver. Still, I always like it when someone uses declarative code to solve a problem, even though the question then becomes, whether the abstraction is also readable for more complex tasks. We’ll see how Kestra turns out!
The Open Source Book
What: Nadia Eghbal published a book on Open Source with a special look at GitHub in 2020.
My perspective: I enjoyed the book and would recommend it to everyone working inside the open-source space. It’s a light introduction but it brought a bunch of interesting concepts and foundational ideas behind open source to my attention. It’s not a deeply strategic perspective, but then again it doesn’t aim to be that.
Ressource: “Working in Public: The Making and Maintenance of Open Source Software”
What: Robert Offner from Kolibri games takes a deeper look into Airbyte and Meltano, and concludes with the note that for his specific use case, neither tool currently does the job much better than a hand-coded solution. But he is very optimistic about the future of these tools.
My perspective: It’s very interesting at which points Robert struggles to extract value from both of these tools.
In particular, Meltano, with the vision of being a DataOps OS should be just that: Easier to use than hand-coded anything. Apparently, it is not (yet).
I do follow Robert when he highlights that Airbyte has put more effort into making it easier to contribute and develop connectors, something that Meltano still fails to do. I’ve done a deeper dive into that topic and the issues at Meltano three months ago. It still seems like they aren’t moving aggressively into the platform business.
Let’s see where this is headed. Airbyte seems to aim to get all issues out of the way by Q3…
🎄 Thanks => Feedback!
Thanks for reading this far! I’d also love it if you shared this newsletter with people whom you think might be interested in it.
Data will power every piece of our existence in the near future. I collect “Data Points” to help understand & shape this future.
If you want to support this, please share it on Twitter, LinkedIn, or Facebook.
And of course, leave feedback if you have a strong opinion about the newsletter! So?
It is terrible | It’s pretty bad | average newsletter… | good content… | I love it!
P.S.: I share things that matter, not the most recent ones. I share books, research papers, and tools. I try to provide a simple way of understanding all these things. I tend to be opinionated. You can always hit the unsubscribe button!
Data; Business Intelligence; Machine Learning, Artificial Intelligence; Everything about what powers our future.
In order to unsubscribe, click here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue