All posts

July 15, 2026 · Christian Domínguez

Why We Built the horus-runtime

Why We Built the horus-runtime

Years working at a supercomputing center taught me one thing: almost nobody enjoys configuring HPC environments. Everyone just wants to get to the science.

I spent a lot of hours learning SLURM, debugging environments, and waiting on compute queues that never seemed to free up. None of that time had anything to do with the actual research questions I was trying to answer. It was pure overhead, and it was the same overhead for almost everyone I worked with, regardless of their field.

So I started asking a simple question: why does running a computational workflow require so much infrastructure knowledge? A researcher should be able to focus on their field, not on which version of conda happens to be installed on a compute node.

What was missing

I tried the existing workflow managers. Nextflow, Galaxy, and a handful of smaller tools. They're solid pieces of engineering, built by people who clearly understood the problem. But I kept running into the same pattern: another manual to read, another environment to configure by hand, another day lost before I could actually run anything.

Using the tool wasn't the hard part. Getting the tool to a state where it could run my workflow was the hard part. That's where the days went.

At some point it became clear that a workflow engine is really just a collection of chained commands you could run from a bash script if you had the patience. What was missing wasn't orchestration logic. It was a system that took care of the environment itself: installing dependencies, moving files to where they needed to be, picking the right compute target, without asking the researcher to know any of that in advance.

That's what led to horus-runtime.

Separating where, how, and what

horus-runtime diagram Every Horus task separates where it runs, how it runs, and what runs.

One design decision worth explaining in some depth, because it's the part that makes the rest of the system work: every task in Horus is broken into three independent questions.

Where does it run. This is the target. It could be your laptop, an HPC cluster reachable over SSH, or a cloud instance. The target answers only one question: which machine.

How does it run. This is the executor. Locally, over SSH, through a scheduler like SLURM. The executor answers how the task gets dispatched and managed once it's on the target.

What actually runs. This is the runtime. The environment, the code, the dependencies, the actual computation.

Most workflow tools bundle these three things together. The way you define a task on your laptop is tied to the way you define it on a cluster, so moving from one to the other usually means rewriting part of the workflow. By keeping where, how, and what as separate, swappable pieces, the same task definition can run in a completely different environment just by changing one of the three. Have an idle GPU machine sitting on the network? Point the task's target at it. Nothing else about the workflow needs to change.

This sounds like a small architectural choice, but it's the one that makes reproducibility possible in practice rather than in theory. The execution layer becomes part of the workflow's definition instead of something you have to remember to configure correctly every time.

Target swap Swapping targets doesn't require touching the rest of the task definition.

Moving data without thinking about it

Once a task can run on any target, a new problem shows up immediately: the task needs its input files to actually be there. Swapping executors is easy in the abstract, but someone still has to get the data to the right machine, and that's usually where people give up and go back to doing things by hand.

Horus handles this with what we call transfer strategies. The runtime looks at where the input lives and where the task is going to run, and picks the fastest available way to move it:

  • Local target: no transfer needed at all.
  • Same cluster: a direct copy.
  • Remote target: whatever transfer mechanism is available and fastest for that pair of locations.

Because the local and SSH executors both implement the same target protocol, this swap happens automatically. You don't write different data-movement code for different environments. The runtime figures it out.

Executor swap diagram Horus picks the fastest transfer strategy based on where data lives and where the task runs.

Plugins, not a fixed feature set

No two labs run the same tools. Every research group has at least one piece of software that's niche, internal, or held together with duct tape from a decade ago. A workflow engine that only supports a fixed list of integrations will always leave someone out.

So almost every part of horus-runtime is pluggable: Python environments, SSH targets, SLURM executors, logging backends, event transports. The things you'd expect to be built-in (local execution, SSH, common environment types) actually are plugins themselves, implementing the same core interfaces available to anyone extending the system. If your lab's tool doesn't have a plugin yet, writing one follows the same pattern as everything else in the runtime.

Where this is going

horus-runtime started with life sciences workflows in mind, because that's where I was working and where the pain was most obvious. But the same problem exists anywhere compute-intensive research happens: physics simulations, AI training and inference, materials science, climate modeling. None of those fields need a life-sciences-specific tool. They need the same thing biotech researchers need, which is infrastructure that gets out of the way.

The runtime is open source, and we're building a community workflow repository, Pantheon, so people don't have to start every pipeline from a blank file. We're also building a visual interface on top of horus-runtime for teams who want to run these workflows without touching a terminal at all, which is a separate piece we'll write about soon.

If you work with HPC or scientific computing and any of this sounds familiar, we'd like to hear from you. Try it, break it, open an issue, or tell us why it's the wrong approach. That feedback is what shapes where the runtime goes next.

Links: