Lightweight Data Pipeline

This is a lightweight framework for setting up dependency-driven processing pipelines. The tool is essentially a configurable shell script (sh/bash), which can be included in other scripts and primarily provides a small number of utility functions for dependency checking and NFS-safe file locking for cluster processing.

While the framework is not very powerful (deliberately so), it is extremely simple and easy to apply. There is no need to provide any command wrappers. An existing processing script (or sequence of shell commands) can be turned into a data-driven, cluster-safe pipeline by adding just a few extra lines of shell code.
Download Now Download Now
OR See All Files

Specifications

Category:Algorithm or Reusable Library
License:GNU General Public License (GPL)
Development Status:5 - Production/Stable/Mature
Domain:Domain Independent
Intended Audience:Developers
Natural Language:English
Operating System:POSIX/UNIX-like
Programming Language:sh/bash
Other Keywords: pipeline framework

Recent Activity

Show: 
File Activity

lwdp: 1.0.1 release

lwdp-1_0_1.tar.gz posted by Torsten Rohlfing on Dec 3, 2013

Document Activity

Main Folder documentation

Basic documentation posted by Torsten Rohlfing on Nov 15, 2013

Forum Activity

help forum

Welcome to Help posted by Torsten Rohlfing on Nov 8, 2013