Lightweight Data Pipeline
This is a lightweight framework for setting up dependency-driven processing pipelines. The tool is essentially a configurable shell script (sh/bash), which can be included in other scripts and primarily provides a small number of utility functions for dependency checking and NFS-safe file locking for cluster processing.
While the framework is not very powerful (deliberately so), it is extremely simple and easy to apply. There is no need to provide any command wrappers. An existing processing script (or sequence of shell commands) can be turned into a data-driven, cluster-safe pipeline by adding just a few extra lines of shell code.
While the framework is not very powerful (deliberately so), it is extremely simple and easy to apply. There is no need to provide any command wrappers. An existing processing script (or sequence of shell commands) can be turned into a data-driven, cluster-safe pipeline by adding just a few extra lines of shell code.
Execution Options
Download Now:
Execute Now:
Specifications
Category:
License:
Associations
is from the makers of:
is required to run:
Recent Activity - Documents
Basic documentation posted by Torsten Rohlfing on Nov 15, 2013
Recent Activity - Forums
Welcome to Help posted by Torsten Rohlfing on Nov 8, 2013
Recent Activity - Files
lwdp-1_0_1.tar.gz posted by Torsten Rohlfing on Dec 3, 2013