Dagster 1.0, VAST, Democratic AI; ThDPTh #75

I’m Sven, and this is the Three Data Point Thursday. The email that helps you understand and shape the one thing that will power the future: data. I’m also writing a book about the data mesh part of that. 

Time to Read: 3 minutes

Another week of data thoughts:

• Data developer experience is important. Dagster is taking that seriously, with 1.0 and GA.

• Network security is a market that needs platforms & protocols, different from the ones the general data world is developing. VAST is on a mission to solve that.

• Human mechanism design is now done okayish by AIs.

🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰🐰

VAST - Visibility Across Space and Time

What: The company tenzir, a network security company, build an open-source protocol called “VAST”. It is meant to be used as an abstraction layer to connect to different network-related data sources & targets. 

My perspective: I have no idea who came up with the name but I love it. The security space, just like any other data space is lacking protocol layers and abstraction layers. So having the first version of an abstraction layer here being built as an open-source solution is an important step.

The data connection problems in the security space are very interesting because they are quite different than what you usually see in the data space, so a special purpose tool is definitely needed. 

In the usual data space, the problem is to:

• batch get data

• put batch data

• stream data (in either direction)

• send more complex queries & retrieve the results

from A -> B. 

In the security space, the additional problem is focused on two things:

• sending very specific but complex queries from A-> B (like search for this long list of IP addresses, crosscheck with the weekly access frequency)

• sending command/query combinations from A -> B (search for all the above, if you then see the IP address again, block access and alarm me)

If you’re interested in that sector, check out VAST.

Resource: https://github.com/tenzir/vast

Dagster 1.0

What: While I was on vacation, exciting stuff happened! Dagster hit 1.0, a stable version, and launched the cloud version into general availability (GA).

My perspective: I really enjoy the focus Nick Schrock and his team put on developer experience. The tag line is “.... platform that put the developer experience first”; They feature hybrid deployments meaning data can still reside in the customer’s network, which is great for everyone with security concerns (which should be everyone, period.). Fwiw, I discussed that part already a year ago in How to Become The Next 30 Billion $$$ Data Company.

Why is that so great? Because the focus is on the data developers’ experience, which is mostly neglected by large parts of the industry still. 

Resource: https://dagster.io/blog/dagster-cloud-ga-launch

Human-Centered Mechanism Design with AI

What: Democratic AI is a reinforcement learning system with humans in the loop that designs social mechanisms. Social mechanisms are preferred by a majority of humans. 

My perspective: This is just a cool piece of research, especially because I think mechanism design is quite helpful as a discipline in the business world we’re operating in where we truly are free to design mechanisms for other people on a large scale (anything that is called a “platform” for instance is a largely designed mechanism).

Resource: https://www.nature.com/articles/s41562-022-01383-x

What did you think of this edition?

-🐰🐰🐰🐰🐰 I love it, will forward!

-🐰🐰🐰  Average newsletter... 

-🐰 It is terrible  ( = I just made it to this link b.c. I was looking for the unsubscribe button)

Want to recommend this or have this post public?

This newsletter isn’t a secret society, it’s just in private mode… You may still recommend & forward it to others. Just send me their e-mail, ping me on Twitter/LinkedIn and I’ll add them to the list.

If you really want to share one post in the open, again, just poke me and I’ll likely publish it on medium as well so you can share it with the world.