Rendered at 20:59:42 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
msteffen 23 hours ago [-]
This is so cool. I spent years at Pachyderm (YC...something) banging our head against:
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.
icyfox 23 hours ago [-]
Cool thought! We have an internal version of map reduce implemented in waymark that worked pretty well (though uses Redis for its intermediate values so it's technically not an end-to-end integrated solution).
In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.
Where did you hit the most snags with caching?
msteffen 20 hours ago [-]
Well, it was just that the models didn't really line up.
To give a somewhat longer description of what Pachyderm does/did:
1. Connect data-scale version control with pipeline orchestrator.
2. Version control tells orchestrator which shards have changed since the last run and which ones haven't.
3. Orchestrator only processes changed shards from one run to the next, instead of all of them.
4. Profit.
That's fine for e.g. one mapreduce, but,
1. If an action's behavior changes, you can't re-use its prior results (usually—sometimes you're just adding print statements to debug a failing shard)
2. If the overall pipeline changes, you can't re-use results from the first change onward
3. Figuring out if the overall pipeline changed is not necessarily easy, and if you're always throwing out everything just in case, your product is not helping people.
So, the way Waymark uses a compiler for the orchestration layer would've worked so well with our model.
It's also just satisfying—once I learned in detail how Bazel/Nix work, I see a fit for that architecture pattern in anything remotely pipeline-y, and it drove me nuts that Airflow/Beam/Dataflow weren't really using it. So I'm very happy to see this for that reason as well. And, as if that weren't enough, it's so cool that it introduces a pure language at the coordination layer (which this type of architecture basically needs, hence the Nix language/Starlark) without actually requiring a new language, or even a new file. Amazing!
ajb 15 hours ago [-]
Yeah, that's an interesting use case. The problem for you with a compiler would be how to get it to work with all of the efficient python modules like pandas, script, numpy, pytorch etc that all implement their backend in C with intrinsics/GPU usage.
MOZGIII 14 minutes ago [-]
Well, this is something we've been, kind of, getting ready to tackle with some of the aspects of the internal design of the VM. In short, it would be pretty easy for us to just add an "instruction (sub)set" to work with GPU kernels or pytorch or etc - and then to implement it via whatever means fir best (a third-party crate, a GPU offload via vulkan or sth, heck - even spawning a python interpreter to handle some big operation where the overhead of python bringup is negligible in on the table).
The main issue so far is that our VM that does the workflow orchestration and the actions runtime are living in separate processes - but we even then, we could do tricks to share memory between them for more efficient / zero-copy data passing... That's not something we have right now though.
yhjz 11 hours ago [-]
Nice project! (And website!)
Great to see some experimentation in this space. I really like the idea of 'make this async function a background process' using a simple annotation.
What kind of IR do you use? I often wonder what a WASM based durable execution engine would look like. And what that pattern would unlock.
MOZGIII 2 hours ago [-]
Thanks!
We have two things, both custom: bytecode (what our VM uses) and IR (what Python is parsed into); there's no stable ABI yet since we practically don't need it, yet, IR is specified as a protobuf schema and bytecode is a specified via rust-native types in our framework for building instruction sets.
In principle, where WASM would go is it would be our compilation target instead of our own bytecode. We've thought about using WASM and, independent of that, about using LLVM compiler facilities. They don't have to go together, and, arguably, tapping into LLVM would be more interesting than WASM (I'm mentioning them together one of the intuitive perks from using WASM would seemingly be using WASM compilers too).
The thing is, we would, most likely, not have too much benefits from using LLVM or existing compilers from various languages into WASM; we compile things in a slightly different manner, and, I suspect, it would be possible to re-use some of the existing optimizations for control graph - but not the low-level ones.
As for WASM - we use a stackless VM, and WASM has an execution stack as a requirement - so we can't seemingly use WASM. That said, in practice we could use the same technique rust uses for compiling futures, and run on a WASM VM with a single stack frame holding our state machine and just doing everything in it. That's pretty close conceptually to how we have implemented in our custom VM now.
gavmor 21 hours ago [-]
Durable execution has been on my radar for a minute but, although I've had a few pubsub usecases here and there, I've never had much call for long-running jobs that weren't simply cheaper to restart, AFAIR.
With agentic workloads, though—especially trending towards high network calls and smaller models—I do find a lot of concurrency to be a pain in the ass to pick up where we left off. Is that what "durability" ameliorates?
I just wonder what I'm leaving on the table.
MOZGIII 26 minutes ago [-]
To add to the above - with Waymark, the durability of highly concurrent workflows specifically is where we shine - because under the hood we just have a state machine that we durable persist when it emits side-effects.
What this means is when you have an async fn that you spawn in a loop where each iteration also spawns a ton of async fns which in turn call actions - there's no separate bookkeeping to do parallel to the execution itself. No replays (unlike other tools like Temporal), meaning the thing just goes, as-is, and when it crashes it can load the state and resume from where it left off - without the need to match the state of the interpreter to the persisted state of execution graph. We just store the real, actual runtime state.
icyfox 21 hours ago [-]
Yep, that’s exactly it. imo the main value add for agents:
- Each LLM call and tool call becomes a checkpoint. Without durable execution you're pretty much stuck doing db saves/restores if you detect that this agent might have a previous context window. Agent frameworks are fundamentally just harnesses around the underlying model: you `while loop` until some server gives you some payload back, you add that to your context window list, and you keep going until it terminates. Durable execution moves that loop into the workflow body itself (a quick example here https://www.waymark.sh/python/tutorials/building-a-from-scra...).
- Long waits. Sometimes you do want your agent to wait for an arbitrary amount of time (ie. wait for this compilation job to finish - look again in 5mins, or maybe 5 hours). Durable execution makes sure that sleep timer will wake up regardless of what happens in the system state.
- Easier monitorability. Instead of having to sift through interleaved logs to figure out what was going on, you can see every large function call with a clean input/output contract. Useful especially in tracing down raised Exceptions and the data flow that it took to get there. Good for agents and regular code alike.
Of course, if your system is largely idempotent or your agents are mostly classifiers that are working in short internals - a simple db save/restore can work perfectly well too.
cgio 19 hours ago [-]
Durable execution and orchestration are essential in some specialised uses, such as regulatory reporting for financial institutions, or other snapshot based data pipelines. The transition is happening, and temporal and conductor are the usual tools there, but good tools are not always behind good solutions to put it lightly.
I like this one, as in the context of workloads like these, going with anything more complex than a Postgres is overkill.
mukeshsinghnyc 6 hours ago [-]
How is it different from temporal ? How code execution governance policy is handled to avoid issues like hogging face ?
slopnt 20 hours ago [-]
The idea makes sense and the work is great. Some feedback I have is that it becomes hard to try the project out when postgres is a mandatory dependency: I understand that there are scalability reasons why that choice was made, but it would be neat if sqlite was an option.
icyfox 20 hours ago [-]
We'll look into it! The db layer is already modeled basically like a plugin; should be pretty easy to add in whatever database support you want. Sign up to the newsletter on the website if you want the updates when it's ready :)
hgomersall 11 hours ago [-]
I completely understand that it might be because I've never suffered the pain associated with the thing you're purporting to solve, but I really don't understand what Waymark does.
I find this really common on HN - someone posts a project that addresses a problem and all the intro and docs assume you understand the problem and couch the discussion in somewhat domain specific language.
It might be interesting to me. It might even be relevant to me and I don't know it yet. The problem is I need to work really hard to get to the point of understanding that.
This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.
MOZGIII 39 minutes ago [-]
Oh, I get it. Let me give a simple example that hopefully explains waymark. Can't guarantee it'd be helpful to you, but beyond the website (we got nice animations and explainers there, you should check it out if you haven't) this might be helpful cause it comes from a well-known architectural problem.
Consider a microservices architecture (for the sake of example), where a user sign up needs to create an account in three services: auth-svc, avatars-svc and emails-svc.
Your options to do it are either an event-sourcing architecture where your services have to orchestrate together via somewhat-awkward message-passing, or write another reconciling service where you write the logic more-or-less directly, but with a ton of code-overhead to make the execution reliable.
Waymark would help with the second approach, because it soaks in all the durable execution complexity, and gives you a way to just write the logic code and handle the edge cases - with a fraction of the code and systems you'd need otherwise.
I'd even dare to say - a tool like waymark, the second approach is pretty much impractical altogether, hence why the event sourcing microserivce architectures dominate today.
But, yet again, that's just one example of where waymark would be useful. If you can write the code where you can split the orchestration logic from the execution of the individual side effects - then with waymark your orchestration code can be just the logic you need, without the extra steps to persist the progress - thus making it easier to reason and mental model simpler.
AnonHP 6 hours ago [-]
> all the intro and docs assume you understand the problem and couch the discussion in somewhat domain specific language.
It might be interesting to me. It might even be relevant to me and I don't know it yet. The problem is I need to work really hard to get to the point of understanding that.
> This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.
I agree with this. For Show HN posts, this would really help broaden the audience as well as lift up the audience’s knowledge and enthusiasm.
mukeshsinghnyc 6 hours ago [-]
Does it handle checkpoints corrections mid run?
cyberpunk 12 hours ago [-]
How many tasks per workflow for the 1500/sec benchmark?
19 hours ago [-]
damndan 15 hours ago [-]
so Temporal without the replay, Step Functions without the JSON -pretty slick
kubenest 7 hours ago [-]
looks a lot like temporal
mukeshsinghnyc 4 hours ago [-]
Agree I asked same question
_1tan 12 hours ago [-]
Any plans for Java?
actionfromafar 20 hours ago [-]
Dotnet CIL + sqlite please?
MOZGIII 1 hours ago [-]
Well, re sqlite, we're so reliant on postgres that it's hard to imagine sqlite right now. If we'd want to replace postgres with something, I'd rather look into redb and the like first...
As for CIL - we should add it at the "What should we build next?" section on the website :)
dangistan202611 21 hours ago [-]
This looks like something that will break easily with a newer version of Python or even with the use of third-party packages. It seems to couple too tightly with the user's application code.
dangistan202611 22 hours ago [-]
It is too async heavy as per its readme -- this is very off-putting. Not everyone cares for async. There is no usage shown with sync code.
MOZGIII 1 hours ago [-]
Well, yeah, today the sync code would be awkward to use - possible, but hacky.
With the way our current Python workers are implemented you'd have we'd have to place the sync code onto threads manually.
We could make it happen though, if there is a demand for sync code; we can allow actions to be non-async, and from that run them in sync mode - either with an sdk-level offload to threads off-of async reactor, or on a separate worker pools that run fork-per-action. From the perspective of our VM, we don't really have a difference there...
icyfox 22 hours ago [-]
MIT licensed, just clarified in the repo.
As for async - background jobs are better modeled as async because of their natural parallelism across nodes imo. Logic in actions themselves are free to be as synchronous as you want. Why the preference?
dangistan202611 22 hours ago [-]
I dispute the "better" assertion. Anyhow, I don't know how familiar or not you are with sync code, but it fully supports background jobs using queues with threads/interpreters/processes. It has been around for decades, far longer than async.
Also, it is okay to run something in the foreground. Not everything needs to run in the background. Sometimes it's better to wait and have assurance of completion.
There is no natural parallelism of async code in Python. The coroutines run in a single thread.
9dev 21 hours ago [-]
I don’t understand these kinds of strong emotions async python triggers in some people. It reminds me of the irrational hate towards systemd in the Linux world.
This is a platform for running workflows distributed on multiple machines, sometimes long-running and with lots of IO. Every single contender in the space advertises the same code, from temporal to hatchet to dagster. Because it makes sense to write it this way.
dangistan202611 20 hours ago [-]
Huh. I hate neither async nor systemd. There is nothing wrong with async, but if my code is sync, what am I to do?
Also, the claim made that something is "better modeled as async" is wholly unqualified.
whilenot-dev 14 hours ago [-]
> if my code is sync, what am I to do?
You'd use asyncio.to_thread[0].
I also don't quite get your complaint... asyncio provides a high-level concurrency primitive with coroutines, and that's a good enough primitive for workflow engines. Even with asyncio you could fork/spawn processes easily with aiomultiprocess[1], if you'd ever want or need to.
What's your use case for having more low-level concurrency primitives (threads/processes) available, having to deal with their synchronization, and running workflows sync?
No such thing is typically necessary. If modeled correctly, the need for synchronization is minimal.
With sync code, tasks run on different threads/interpreters/processes. Message passing via queues is used to communicate which tasks need to run. Many asyncio users seem to wholly unaware of how well it works. Go-lang doesn't even have or need async.
I actually like asyncio; I am just not a religious zealot about it as many here seem to be, also without awareness of the struggles they face with loop-hogging coroutines and multicore parallelism.
vancekai 10 hours ago [-]
[dead]
drrajvanshi 16 hours ago [-]
[flagged]
KMan2006 17 hours ago [-]
How do you handle code changes for workflows already in flight? Say an instance is sleeping inside a loop for three days and someone deploys a new version of that loop. Is the compiled bytecode pinned per instance, or do you map the old position onto the new code?
MOZGIII 1 hours ago [-]
TLDR: pinned per instance.
When we compile a workflow into bytecode we get an executable; we store it and assign it an executable id. In-flight workflows are technically VM instances, and they always pin to a concrete executable id. When the VM goes out of residency (i.e. evicted from a node that executes it and unloaded from memory) we will eventually gc the executable bytecode from memory if it's not used by any VM for a while - but it's kept in postgres so VM can be revived.
There are ways to make workflows upgradeable - but don't have this yet. Ideally, we'd want to build it once we have enough feedback on how people want to do this - cause it's really about choosing the trade-offs more than anything else.
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.
In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.
Where did you hit the most snags with caching?
To give a somewhat longer description of what Pachyderm does/did:
1. Connect data-scale version control with pipeline orchestrator.
2. Version control tells orchestrator which shards have changed since the last run and which ones haven't.
3. Orchestrator only processes changed shards from one run to the next, instead of all of them.
4. Profit.
That's fine for e.g. one mapreduce, but,
1. If an action's behavior changes, you can't re-use its prior results (usually—sometimes you're just adding print statements to debug a failing shard)
2. If the overall pipeline changes, you can't re-use results from the first change onward
3. Figuring out if the overall pipeline changed is not necessarily easy, and if you're always throwing out everything just in case, your product is not helping people.
So, the way Waymark uses a compiler for the orchestration layer would've worked so well with our model.
It's also just satisfying—once I learned in detail how Bazel/Nix work, I see a fit for that architecture pattern in anything remotely pipeline-y, and it drove me nuts that Airflow/Beam/Dataflow weren't really using it. So I'm very happy to see this for that reason as well. And, as if that weren't enough, it's so cool that it introduces a pure language at the coordination layer (which this type of architecture basically needs, hence the Nix language/Starlark) without actually requiring a new language, or even a new file. Amazing!
The main issue so far is that our VM that does the workflow orchestration and the actions runtime are living in separate processes - but we even then, we could do tricks to share memory between them for more efficient / zero-copy data passing... That's not something we have right now though.
Great to see some experimentation in this space. I really like the idea of 'make this async function a background process' using a simple annotation.
What kind of IR do you use? I often wonder what a WASM based durable execution engine would look like. And what that pattern would unlock.
We have two things, both custom: bytecode (what our VM uses) and IR (what Python is parsed into); there's no stable ABI yet since we practically don't need it, yet, IR is specified as a protobuf schema and bytecode is a specified via rust-native types in our framework for building instruction sets.
In principle, where WASM would go is it would be our compilation target instead of our own bytecode. We've thought about using WASM and, independent of that, about using LLVM compiler facilities. They don't have to go together, and, arguably, tapping into LLVM would be more interesting than WASM (I'm mentioning them together one of the intuitive perks from using WASM would seemingly be using WASM compilers too).
The thing is, we would, most likely, not have too much benefits from using LLVM or existing compilers from various languages into WASM; we compile things in a slightly different manner, and, I suspect, it would be possible to re-use some of the existing optimizations for control graph - but not the low-level ones.
As for WASM - we use a stackless VM, and WASM has an execution stack as a requirement - so we can't seemingly use WASM. That said, in practice we could use the same technique rust uses for compiling futures, and run on a WASM VM with a single stack frame holding our state machine and just doing everything in it. That's pretty close conceptually to how we have implemented in our custom VM now.
With agentic workloads, though—especially trending towards high network calls and smaller models—I do find a lot of concurrency to be a pain in the ass to pick up where we left off. Is that what "durability" ameliorates?
I just wonder what I'm leaving on the table.
What this means is when you have an async fn that you spawn in a loop where each iteration also spawns a ton of async fns which in turn call actions - there's no separate bookkeeping to do parallel to the execution itself. No replays (unlike other tools like Temporal), meaning the thing just goes, as-is, and when it crashes it can load the state and resume from where it left off - without the need to match the state of the interpreter to the persisted state of execution graph. We just store the real, actual runtime state.
- Each LLM call and tool call becomes a checkpoint. Without durable execution you're pretty much stuck doing db saves/restores if you detect that this agent might have a previous context window. Agent frameworks are fundamentally just harnesses around the underlying model: you `while loop` until some server gives you some payload back, you add that to your context window list, and you keep going until it terminates. Durable execution moves that loop into the workflow body itself (a quick example here https://www.waymark.sh/python/tutorials/building-a-from-scra...).
- Long waits. Sometimes you do want your agent to wait for an arbitrary amount of time (ie. wait for this compilation job to finish - look again in 5mins, or maybe 5 hours). Durable execution makes sure that sleep timer will wake up regardless of what happens in the system state.
- Easier monitorability. Instead of having to sift through interleaved logs to figure out what was going on, you can see every large function call with a clean input/output contract. Useful especially in tracing down raised Exceptions and the data flow that it took to get there. Good for agents and regular code alike.
Of course, if your system is largely idempotent or your agents are mostly classifiers that are working in short internals - a simple db save/restore can work perfectly well too.
I like this one, as in the context of workloads like these, going with anything more complex than a Postgres is overkill.
I find this really common on HN - someone posts a project that addresses a problem and all the intro and docs assume you understand the problem and couch the discussion in somewhat domain specific language.
It might be interesting to me. It might even be relevant to me and I don't know it yet. The problem is I need to work really hard to get to the point of understanding that.
This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.
Consider a microservices architecture (for the sake of example), where a user sign up needs to create an account in three services: auth-svc, avatars-svc and emails-svc.
Your options to do it are either an event-sourcing architecture where your services have to orchestrate together via somewhat-awkward message-passing, or write another reconciling service where you write the logic more-or-less directly, but with a ton of code-overhead to make the execution reliable.
Waymark would help with the second approach, because it soaks in all the durable execution complexity, and gives you a way to just write the logic code and handle the edge cases - with a fraction of the code and systems you'd need otherwise.
I'd even dare to say - a tool like waymark, the second approach is pretty much impractical altogether, hence why the event sourcing microserivce architectures dominate today.
But, yet again, that's just one example of where waymark would be useful. If you can write the code where you can split the orchestration logic from the execution of the individual side effects - then with waymark your orchestration code can be just the logic you need, without the extra steps to persist the progress - thus making it easier to reason and mental model simpler.
> This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.
I agree with this. For Show HN posts, this would really help broaden the audience as well as lift up the audience’s knowledge and enthusiasm.
As for CIL - we should add it at the "What should we build next?" section on the website :)
With the way our current Python workers are implemented you'd have we'd have to place the sync code onto threads manually.
We could make it happen though, if there is a demand for sync code; we can allow actions to be non-async, and from that run them in sync mode - either with an sdk-level offload to threads off-of async reactor, or on a separate worker pools that run fork-per-action. From the perspective of our VM, we don't really have a difference there...
As for async - background jobs are better modeled as async because of their natural parallelism across nodes imo. Logic in actions themselves are free to be as synchronous as you want. Why the preference?
Also, it is okay to run something in the foreground. Not everything needs to run in the background. Sometimes it's better to wait and have assurance of completion.
There is no natural parallelism of async code in Python. The coroutines run in a single thread.
This is a platform for running workflows distributed on multiple machines, sometimes long-running and with lots of IO. Every single contender in the space advertises the same code, from temporal to hatchet to dagster. Because it makes sense to write it this way.
Also, the claim made that something is "better modeled as async" is wholly unqualified.
You'd use asyncio.to_thread[0].
I also don't quite get your complaint... asyncio provides a high-level concurrency primitive with coroutines, and that's a good enough primitive for workflow engines. Even with asyncio you could fork/spawn processes easily with aiomultiprocess[1], if you'd ever want or need to.
What's your use case for having more low-level concurrency primitives (threads/processes) available, having to deal with their synchronization, and running workflows sync?
[0]: https://docs.python.org/3/library/asyncio-task.html#asyncio....
[1]: https://aiomultiprocess.omnilib.dev/en/stable/
No such thing is typically necessary. If modeled correctly, the need for synchronization is minimal.
With sync code, tasks run on different threads/interpreters/processes. Message passing via queues is used to communicate which tasks need to run. Many asyncio users seem to wholly unaware of how well it works. Go-lang doesn't even have or need async.
I actually like asyncio; I am just not a religious zealot about it as many here seem to be, also without awareness of the struggles they face with loop-hogging coroutines and multicore parallelism.
When we compile a workflow into bytecode we get an executable; we store it and assign it an executable id. In-flight workflows are technically VM instances, and they always pin to a concrete executable id. When the VM goes out of residency (i.e. evicted from a node that executes it and unloaded from memory) we will eventually gc the executable bytecode from memory if it's not used by any VM for a while - but it's kept in postgres so VM can be revived.
There are ways to make workflows upgradeable - but don't have this yet. Ideally, we'd want to build it once we have enough feedback on how people want to do this - cause it's really about choosing the trade-offs more than anything else.