🐰 #18 Hot OSS Data Projects, Agile Data Roadmaps, Data Mesh Exec. Brief; ThDPTh #18 🐰

What are the cool OSS data projects this year? How does a good data roadmap look like? And how to pitch the data mesh paradigm to the C-level.

(1)🔥 Hot OSS Data Projects

Pete Sodeling ran a Data Council Survey to check out the hot data projects for 2021 which features a bunch of interesting ones you should have a look at. The good old Apache Airflow and the transformation tool dbt are on the list, but also a few other interesting tools you might not have seen.

A few ones I’d like to highlight: Apache Superset, the OS enterprise-level BI tool released its version 1.0 this year. It has become extensible, integrable, and modular which makes it in my opinion the default option to look at if you’re considering enterprise-level BI tools. You might still go for something “smaller” sometimes, but even hosted versions are already available taking away the hassle of operating it.

Dagster an orchestrator for data DAGs is mentioned as is great-expectations which is a really interesting tool for profiling & testing data. The tool RudderStack also appears on the list which is a data integrator/reverse ETL tool. Take a look at the list if you got some time to spare.

What are the most popular OSS data projects of 2021?

We successfully ran our Data Council 2021 OSS Data Tools Community Survey …

petesoder.medium.com  •  Share

(2)🔮 Good Agile Roadmaps Examples

I’ve said it before, but I’ll say it again, data needs more proper product management. With that, it especially can need some proper agile roadmapping. After all, I think almost anything we do currently lives in an agile world. But what does a good agile roadmap look like? To me it has to have a few properties:

1. It has to be simple, ideally a plain list, because anything that’s “parallel” suggests you’re actually multi-tasking and that’s a pretty bad idea.

2. It has to be subject to change; ideally, something like 30–50% of the things inside each “part” of the roadmap should be exchanged once you finish that “part”

3. The time segment size should increase! So if you’re now in April, the roadmap could say “April, May, June” then it could say “Q3”, “Q4” and then “2022”.

4. The “task size” should increase/become rougher. There’s no need to really refining things that will likely not happen or under changed circumstances once you get there.

That’s how a proper agile roadmap should look like, and the current meltano roadmap looks exactly like that, so I really wanted to share it. And turns out, GitLab made this a practice they call the “nearsighted roadmap”.

Introduction | Meltano

Learn where to use Meltano, how Meltano is built, and where to get started. A great nearsighted roadmap!

meltano.com  •  Share

(3)🚀 ThoughtWorks Executive Brief on Data Meshes

Data Meshes are a true paradigm shift inside a whole organization. It’s not enough to involve the analytics department, at the very least you will have to involve all of the tech department in this shift.

But to get that level of buy-in is not that easy. I really like the new executive brief ThoughtWorks put out to address this level of communication. It’s a good presentation of why companies should shift to a data mesh starting with the high-level company & the consumer perspective, not just the “pain of the analytics department”.

ThoughtWorks Data Mesh Exec Brief

files.thoughtworks.com  •  Share

🎄 In Other News & Thanks

I managed to produce two articles this week and they fit right into the topics here:

• 4 Excellent Examples of Agile “Nearsighted” Roadmaps

• Data as Code — Achieving Zero Production Defects for Analytics Datasets

Thanks for reading this far! I’d also love it if you shared this newsletter with people whom you think might be interested in it.

P.S.: I share things that matter, not the most recent ones. I share books, research papers, and tools. I try to provide a simple way of understanding all these things. I tend to be opinionated. You can always hit the unsubscribe button!

By Sven Balnojan

Data; Business Intelligence; Machine Learning, Artificial Intelligence; Everything about what powers our future.

Tweet     Share

In order to unsubscribe, click here.

If you were forwarded this newsletter and you like it, you can subscribe here.

Powered by Revue