Horus: the open-source engine that runs scientific workflows
Describe a pipeline once, in Python or YAML, and run it wherever the compute lives. Horus tracks every artifact a task produces, moves those artifacts between machines for you, and skips the work that is already done when you run it again. Built by researchers from the Barcelona Supercomputing Center, licensed under AGPL-3.0, and free to use with no account and no waitlist.
Install it
pip install horus-runtime
horus --helpPublished on PyPI as horus-runtime. Requires Python 3.13 or newer. Installs the horus command-line tool. Licensed under AGPL-3.0.
How the pieces fit together
Horus is one part of a small family of projects that share a name and a purpose. Here is what each one is, and which one you probably want.
Temple Compute
The company. A team of researchers from the Barcelona Supercomputing Center, building the tooling we wanted while running workflows on real clusters. Everything below comes from the same group.
Horus
The execution engine, and what people mean when they say Horus. It resolves the task graph, moves data between machines, and runs each step where you point it. Open source under AGPL-3.0, installed from PyPI as horus-runtime, driven from Python or the horus CLI. It needs no Temple Compute account.
Pantheon
The workflow library: production-ready scientific pipelines for Horus, from molecular dynamics setup to docking and virtual screening. MIT-licensed, so you can clone one and adapt it instead of starting from an empty file.
Temple Compute OS
The managed platform built on Horus, for teams who would rather not run a cluster. Same engine underneath; the difference is that provisioning, monitoring and scaling stop being your job. In early access, with paid tiers launching January 2027.
Plugins
Everything beyond local execution ships as its own package: container backends, schedulers, environment provisioning. You install the ones your infrastructure actually needs and nothing else.
What a Horus workflow is made of
A task answers three separate questions, and keeping them separate is what makes the rest work. Moving a stage from your laptop to a cluster is a change to one of the three, not a rewrite.
Runtime
what runsThe payload itself: the command, the Python callable, the environment it needs. Change the runtime and you change what executes, without touching where it lands.
ReferenceExecutor
how it runsThe dispatch mechanism. A subprocess on the machine in front of you, or a job handed to a scheduler that queues it and reports back. This is the layer that knows about job IDs and polling.
ReferenceTarget
where it runsThe machine, reached over an agentless channel: run a command, put a file, get a file, make a directory. Nothing has to be installed on the far side for Horus to drive it, only the binaries your command already names.
ReferenceArtifacts
Inputs and outputs are typed, addressable objects rather than filenames threaded through strings. Every artifact is materialised on disk as a real file or directory, which is what makes it cacheable, inspectable after a run, and movable between machines with no serialisation protocol of its own.
ReferenceTransfer strategies
When an edge in the graph crosses a machine boundary, Horus looks at where the input lives and where the task is about to run, and picks the cheapest way to bridge the two. When both ends share a filesystem there is nothing to move, and it does not move it.
ReferenceWhat the engine handles for you
- Resolves dependencies across the graph and runs independent tasks concurrently
- Fans a single task out over a collection, then gathers the results back in
- Skips completed tasks on a re-run, so a pipeline that failed at hour nine resumes at hour nine
- Lets each task pick its own target, so one stage can run locally and the next on a cluster
Learning Horus
The documentation is the reference. These are the pages worth reading in order, and roughly what each one gives you.
Or read a finished pipeline end to end in the workflow examples, each one a real multi-stage workflow with the code alongside it.
Extending Horus
Every building block in a workflow is a plugin, registered through a Python entry point and discovered automatically once installed. The engine ships with the local target; schedulers, containers and environment provisioning are separate packages you add when you need them.
Writing your own is a package with an entry point, never a fork of the engine. See the extension points.
Do I need Temple Compute OS?
No. If you already have a cluster or a cloud account and someone to look after it, Horus on its own is a complete answer, and it always will be. The engine is not a trial version of anything.
Temple Compute OS exists for the teams where that second half is the problem: nobody wants to own the scheduler config, the quotas and the provisioning. It runs the same engine, so nothing you write for Horus needs rewriting to move across, and workflow definitions export as self-contained archives if you move away again. See the tiers.
Getting help
Issues and contributions
Bugs, feature requests and pull requests live on the repository.
Open an issuePaid and enterprise support
Help designing or migrating workflows, and support bundled into the Enterprise tier.
Work with usQuestions
- What is Horus?
- Horus is an open-source workflow manager for scientific computing, built by Temple Compute. You describe a pipeline as a set of tasks, in YAML or Python, and Horus runs them in dependency order: resolving the graph, moving data between stages, and sending each stage to the compute you point it at. It is distributed on PyPI as the horus-runtime package.
- Is Horus the same thing as horus-runtime?
- Yes. Horus is the project; horus-runtime is the name of the Python package on PyPI and of the GitHub repository; horus is the command-line tool the package installs. Temple Compute OS is a separate, managed product built on top of the same engine.
- Does Horus have a Python API?
- Yes, and it is the primary interface rather than an afterthought. Workflows are ordinary typed Python objects, so a task can read files, call a library, or prompt the user, and the whole pipeline integrates with the rest of your stack. Simple command-only pipelines can be written in YAML instead. The full SDK reference is in the documentation.
- Where do I find Horus tutorials?
- The documentation has a quickstart that takes you from install to a first run, then guides on core concepts, writing workflows in Python, fan-out and fan-in, and extending the engine. The Pantheon repository and the workflow examples on this site are complete, real pipelines you can read end to end.
- How do I extend Horus with my own plugin?
- Every building block is a plugin: artifacts, tasks, runtimes, executors, targets, transfer strategies and middleware. You subclass the relevant base class and register it through a Python entry point, so once your package is installed Horus discovers it automatically. Adding support for new infrastructure is a package, never a fork. There is a template repository to start from.
- Where do I get support for Horus?
- The community Slack is the fastest route for questions, and bugs belong in GitHub issues on the horus-runtime repository. Temple Compute also offers paid consulting for teams who want help designing or migrating workflows, and support is included in the Enterprise tier of Temple Compute OS.
- Is Horus free, and what is the licence?
- The engine is free and needs no account or waitlist. horus-runtime is licensed under the GNU Affero General Public License v3.0, and the Pantheon workflow library is MIT-licensed. The paid tiers exist only for teams who would rather not run the infrastructure themselves.
Start with the engine
Install it, run the quickstart, and clone a workflow from Pantheon that looks like the one you need. If you would rather not run the infrastructure, ask for a Temple Compute OS invitation.