Dependency checking and "safe" file locking

Type:
Class		 Category:
UNIX Admin
License:
GNU Library Public License		 Language:
TCL
 
Description:
This (ba)sh script defines some useful functions for dependency checking and file locking.

Dependency checking is implemented in function "needs_update", which is called with two or more parameters (all paths to files or directories). The first parameter is the target file, which depends on the remaining files listed as parameters. If one of the prerequisite files is more recent than the target files, then the function returns "true", i.e., the target file needs to be updated. Otherwise, the function returns "false", i.e., the target file is up to date w.r.t. the given dependencies.

Example:

if needs_update target source1 source 2; then
cat source1 source2 > target
fi

The file locking (which tries hard to be NFS-safe, although it isn't completely) is implemented in two functions:

"lockfile_create" creates a lock for a given path in the file system, which is typically a target file that is being generated. If the function returns "true", locking was successful and processing should proceed. If the function returns false, locking failed (e.g., because another process has already locked the same file), and processing should skip this target file.

If a lock was successfully acquired, it must subsequently be freed (i.e., deleted) by calling "lockfile_delete".

Example:

if lockfile_create target; then
echo "Hello!" > target
lockfile_delete target
fi

Finally, there is a combination function, "needs_update_and_lock," which is called exactly like "needs_update", but attempts to acquire a lock on the target file in case it needs to be updated.

Example:

if needs_update_and_lock target src1 src2; then
cat src1 src2 > target
lockfile_delete target
fi

Versions Of This Snippet::

Torsten Rohlfing
Snippet ID Download Version Date Posted Author Delete
91.0.02009-08-26 20:56Torsten RohlfingDelete

Download a raw-text version of this code by clicking on "Download Version"

 

Latest Snippet Version: :1.0.0

#!/bin/sh

##
##  Copyright 2007-2009 SRI International
##
##  This file is part of the Computational Morphometry Toolkit.
##
##  http://www.nitrc.org/projects/cmtk/
##
##  The Computational Morphometry Toolkit is free software: you can
##  redistribute it and/or modify it under the terms of the GNU General Public
##  License as published by the Free Software Foundation, either version 3 of
##  the License, or (at your option) any later version.
##
##  The Computational Morphometry Toolkit is distributed in the hope that it
##  will be useful, but WITHOUT ANY WARRANTY; without even the implied
##  warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
##  GNU General Public License for more details.
##
##  You should have received a copy of the GNU General Public License along
##  with the Computational Morphometry Toolkit.  If not, see
##  <http://www.gnu.org/licenses/>.
##
##  $Revision: 413 $
##
##  $LastChangedDate: 2009-08-07 19:28:54 -0700 (Fri, 07 Aug 2009) $
##
##  $LastChangedBy: torstenrohlfing $
##

##
## Helper functions
##

needs_update()
{
    local target=$1
    if test ! -e ${target}; then
	if test -e ${target}.gz; then
	    target=${target}.gz
	fi
    fi

    shift
    local sources="$*"
    for source in ${sources}; do
	if [ ! -e ${source} ]; then
	    if [ ! -e ${source}.gz ]; then
		echo "========================================================================"
		echo "MISSING SOURCE ${source}"
		echo "========================================================================"
		
		return 1
	    fi
	fi
	
	if [ -e ${source}.lock ]; then
	    echo "========================================================================"
	    echo "SOURCE LOCKED ${source}"
	    echo "========================================================================"
	    
	    return 1
	fi
    done

    local source

    if [ ! -e ${target} ]; then
	echo "========================================================================"
	echo "CREATE ${target}"
        echo "========================================================================"

	return 0
    fi

    for source in ${sources}; do
	if test ! -e ${source}; then
	    if test -e ${source}.gz; then
		source=${source}.gz
	    fi
	fi
	
	if test ${target} -ot ${source}; then
            echo "========================================================================"
	    echo "UPDATE ${target}"
            echo "DUE TO ${source}"
            echo "========================================================================"

	    return 0
	fi
    done

    return 1
}

# NFS-safe (hopefully) file locking.
lockfile_create()
{
    local lockfile=$1.lock
    local hostpid=`hostname`-$$

    if [ -e ${lockfile} ]; then
	# lockfile already exists, so clearly we were not the first
	return 1;
    fi

    mkdir -p `dirname ${lockfile}`
    echo ${hostpid} >> ${lockfile}

    if [ `head -n 1 ${lockfile}` != ${hostpid} ]; then
	# first one to write PID was not us
	return 1;
    fi

    trap "rm -f ${lockfile}; exit" INT TERM EXIT
    return 0;
}

lockfile_delete()
{
    local file=${1}
    local lockfile=${file}.lock

    if [ -f ${file} ]; then
	touch --no-create -r ${lockfile} ${file}
    else
	touch --no-create -r ${lockfile} ${file}.gz
    fi

    rm -f ${lockfile}
    trap - INT TERM EXIT
}

#
# Combine dependency checking with locking of target
#
needs_update_and_lock()
{
    if needs_update $*; then
	if lockfile_create $1; then
	    return 0
	fi
    fi
    return 1
}

Submit a new version

You can submit a new version of this snippet if you have modified it and you feel it is appropriate to share with others..