Lightweight Data Pipeline

This is a lightweight framework for setting up dependency-driven processing pipelines. The tool is essentially a configurable shell script (sh/bash), which can be included in other scripts and primarily provides a small number of utility functions for dependency checking and NFS-safe file locking for cluster processing.

While the framework is not very powerful (deliberately so), it is extremely simple and easy to apply. There is no need to provide any command wrappers. An existing processing script (or sequence of shell commands) can be turned into a data-driven, cluster-safe pipeline by adding just a few extra lines of shell code.

Execution Options

Download Now:


GNU General Public License (GPL)
Development Status:
Intended Audience:
Natural Language:
Operating System:
Programming Language:
Other Keywords:
pipeline framework