Open Source Data Reliability, Joyful Data Devs, TechRadar 26; ThDPTh #65
Open Source Data Reliability, Joyful Data Devs, TechRadar 26; ThDPTh #65
I’m Sven, and this is the Three Data Point Thursday. The email that helps you understand and shape the one thing that will power the future: data. I’m also writing a book about the data mesh part of that.
Another week of data thoughts:
Data reliability is becoming a thing
The joy of programming is important for data devs
Read the TechnologyRadar №26, now!
🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰
What: Five properties that will make coding joyful, and to that extent really productive! The five properties are composable, Unix philosophy, predictable, idiomatic, and domain-based. This quote sums up the idea really well.
“Have you ever cracked open an unfamiliar codebase and just known how to get around? The structure, the naming, the flow is obvious, familiar somehow. A smile appears on your face. “I’ve got this!” you think.”
And …
“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” — Refactoring, Martin Fowler with Kent Beck, 1996
My perspective: First of all, Dan put a lot of thought into this article. I like his well-structured approach to turning principles into properties while offering help for beginners and yet nuances for experts.
I also like the whole intent of the piece, he’s simply making the point that if you add any of the CUPID properties to your code (which will usually involve more reducting than adding), you and others will have an easier time working with it.
I think this makes CUPID in particular very attractive for data people, where strict rules and “principles” might not be the right place to start. Just take any one of the properties and add it to your codebase, your SQL, your ETL workload and you’ll likely be happier at the end of the day.
I for instance remember the day when I saw a data team that was building ETLs in a graphical user interface. The team built a third ETL process basically just ingesting REST API data. They realized, hey we can make this composable! Even within the graphical tool, they managed to provide a composable “REST API — template” which they used to then build on top off. Suddenly, adding a new ingestion process was “a that’s just a simple parameter added” and not “uh, that’s a whole new ETL job”. That’s composability at play — and it certainly seemed more fun.
I also remember when we changed from a “one tool to orchestrate everything” to a more “one tool, one purpose” strategy inside a data team I was part of. I even held a talk about exactly that idea called “Modern Cloud Data warehouses best practices”. Turns out, by decoupling and having one tool do one thing like only ingesting data, only displaying data, only transforming data, suddenly the whole work becomes super fast. Data quality increases, things were much easier to fix because usually, the error was just inside one of these components. That’s the Unix philosophy in action — and I can testify to that myself — it was much more fun!
I am sure you can also find good examples for the other three properties, but I’d encourage you to simply take it as is: a nice opportunity to increase the joy of working inside data and through that, the productivity of your team.
What: The Technology Radar is an opinionated content piece that curates technologies, platforms, languages, and techniques into four categories: Assess, Trial, Adopt, Hold.
My perspective: To put it into my words: This is always a must-read for everyone in technology, and it’s kind of like the guide to the future of software (& data). It’s published by ThoughtWorks and they very accurately shaped the software industry over the last 20 years.
I am already highlighting everything I find interesting on Twitter, but two things stand out. One is the data mesh! Zhamak finally got her book out and I am excited to read it. I already loved the first chapter about the inflection point (you might guess, I am also a fan of everything that Andy Grove wrote, including the “Inflection Point”). Oh yeah, and I also co-authored a book about that topic.
Second, ThoughtWorks is also noticing that stuff is going on with open-source. Especially open-source and data. There is more happening in general, more openness in general, and more commercial action in that space. It’s undirected for now, but it is happening and will be a good thing for everyone involved.
Now, go ahead and just dive right into it.
🎄 Thanks => Feedback!
Thanks for reading this far! I’d also love it if you shared this newsletter with people whom you think might be interested in it.
Data will power every piece of our existence in the near future. I collect “Data Points” to help understand & shape this future.
If you want to support this, please share it on Twitter, LinkedIn, or Facebook.
And of course, leave feedback if you have a strong opinion about the newsletter! So?
It is terrible | It’s pretty bad | average newsletter… | good content… | I love it!
P.S.: I share things that matter, not the most recent ones. I share books, research papers, and tools. I try to provide a simple way of understanding all these things. I tend to be opinionated. You can always hit the unsubscribe button!
Data; Business Intelligence; Machine Learning, Artificial Intelligence; Everything about what powers our future.
In order to unsubscribe, click here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue