Cylc | The Cylc Workflow Engine Production workflow orchestration for cycling systems

Cylc is a general purpose workflow engine that also orchestrates cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialized to those domains.

Cylc is an active Open Source project on GitHub.
See Cylc Publications and Reports below.

Next Release: ()

0% 100%

Cylc is a decentralised, distributed, DAG / DCG scheduler.

Scheduler: Cylc can schedule inter-dependent collections of tasks (aka workflows) to run on platforms of your choice.
Decentralised: Cylc employs a decentralised architecture where each workflow execution is fully independent. This means that workflows are isolated for robustness and Cylc deployments scale horizontally.
Distributed: Cylc workflows can be distributed across a pool of servers to facilitate large multi-user setups. Workflows can distribute their workload across multiple platforms.
DAG: Cylc can schedule Directed Acyclic Graphs of tasks (i.e. workflows which do not cycle) similar to other tools.
DCG: Cylc's unique cycling model allows it to run cyclic workflows too. This makes Cylc especially powerful for cases where later repetitions of tasks depend on the output of earlier instances. Tasks can repeat on different intervals within the same workflow and complex inter-cycle dependencies can be defined between them.

Cylc Does Cycling Properly

Cylc unwinds the cycle loop to get a dependency graph with no barrier between cycles. This allows interleaved cycles for:

Fast catch-up from delays
Sustained high throughput off the clock
Seamless transitions between fast catch-up and clock-limited real-time scheduling

Powerful User Interfaces

Alongside a powerful command line interface, Cylc has a web-based GUI for monitoring, controlling and analysing workflows.

Monitor all your workflows from one place, even in a distributed deployment.
Tabbed interface with multiple views catering to different use cases.
View workflow logs, task output, timings and more.
View the workflow graph live.
Edit task definition on-the-fly.
Manually intervene with workflow execution at runtime.

Jupyter Lab Integration

Optional Jupyter Lab integration allows you to deploy the Cylc GUI and Jupyter Lab together, providing an all in one research platform ready to be deployed to on premise or cloud systems.

Jupyter Hub Integration

Deploy the Cylc GUI via Jupyter Hub to unlock institution-scale multi-user deployments with fine-grained authorisation allowing users to be granted permissions to view and control workflows running under other user accounts.

Teams can collaborate on shared projects.
Different levels of support can be granted different levels of control of production systems.

Why Use Cylc?

Production Ready

Cylc is used 24/7 in production weather forecasting, notorious for the size and complexity of its workflows.
Seamless transition from simulated time to real time for ease of development and testing.

Scaling

Each workflow gets a scheduler that runs as the user and relies only on the filesystem.
Schedulers can be distributed over a pool of hosts, and jobs over job platforms.
Individual workflows scale to thousands of tasks per cycle.
Deployments can scale infinitely.

Powerful UIs

A CLI that can operate on multiple workflows and tasks at once.
A web GUI with Jupyter extension.
A terminal UI with control capability.

Ease of Use

Cylc installs from Conda Forge or PyPI and works "out of the box" on a Linux host.
Small workflows are easy. Many features to help with large, complex workflows.
With no central scheduler or database server, Cylc has low admin overheads and a small security footprint.
Built in support for HPC resource managers like PBS and Slurm.

Multi-User Installations

Cylc leverages JupyterHub for authentication and to launch Cylc UI Servers.
Fine-grained authorization of control by other users.

Advanced Scheduling

Graph branching from optional task outputs.
Supports multiple flows in the same workflow, e.g. to re-run a past subgraph while the main flow continues.
Start a flow from anywhere in the depedency graph.
Cycle interleaving: this cut catch-up from delays from ~24 hours to ~30 minutes at NIWA.

Resilience

Configurable task retries.
Triggering off of all task states.
Comprehensive event handling.
Manual job control: poll, kill, or retrigger jobs.
Restart from latest state even if the scheduler was killed, and follow up on orphaned jobs.
Cylc is cluster-aware: it can interact with jobs via multiple platform hosts.

Publications and Reports

If you use Cylc please cite it in your publications and let us know if your site can be listed on this website.

Workflow Automation for Cycling Systems: The Cylc Workflow Engine
H. Oliver et al
Computing in Science & Engineering Vol 21, Issue 4, July/Aug 2019. DOI: 10.1109/MCSE.2019.2906593

Cylc: A Workflow Engine for Cycling Systems
Oliver et al., (2018).
Journal of Open Source Software, 3(27), 737. DOI: 10.21105/joss.00737

Citable DOI for Cylc code releases.

Old (2017): Cylc response to Assessment Report on Autosubmit, Cylc and ecFlow