ravanan: Reproducible computation using Guix and CWL
 
Arun Isaac

About me
 
▸ Guix contributor
▸ Guile hacker
▸ Bioinformatics postdoc at University College London

My problem
 
▸ Mountains of scientific data to analyze
▸ Submit HPC jobs to a batch system
▸ Jobs can take hours or days to complete
▸ Cache job results so I don't have to run them again

Common Workflow Language (CWL)
 
▸ Standards based workflow language specification
▸ Strongly typed language
▸ YAML based; easy to parse, generate and interpret
▸ Higher level and superior to writing bash scripts
▸ Automates bookkeeping of input/output files, temporary directory creation, etc.

Structure of a CWL workflow
 
▸ input values (strings, numbers, files, etc.) on one end
▸ output values (strings, numbers, files, etc.) on the other end
▸ a "graph" or "flow diagram" of interconnected steps between them

How ravanan runs CWL
 
▸ Convert CWL steps into unique `program-file' G-exp scripts in the store
▸ Run the G-exp scripts passing in the inputs (parameters, data files, etc.)
▸ Cache results addressing by the unique hash of the generated script
▸ Cache is guaranteed to never go stale
▸ Guix—the secret sauce to an ever-fresh cache
 
“If everything is reproducible, computation is premature optimization!”
— Hugo Buddelmeijer on the guix-science mailing list

Demo!                       

ccwl
 
▸ writing CWL workflows in YAML is not much fun!
▸ ccwl is a lispy/schemey syntax to write CWL
▸ a domain-specific language (DSL) implemented in Guile

Thank You!
 
https://forge.systemreboot.net/ravanan/
https://github.com/arunisaac/ravanan/
 
https://ccwl.systemreboot.net/
https://github.com/arunisaac/ccwl/