ZCDB - Read and write cdb files from Common Lisp

ZCDB is a Common Lisp library for reading and writing D. J. Bernstein's fast and simple cdb database file format. It is available under a BSD-style license. The latest version is 1.0.3, released on December 15th, 2011.

Download shortcut: http://www.xach.com/lisp/zcdb.tgz

Contents

Overview

ZCDB is for reading and writing from CDB files. CDB files are described in more detail on D.J. Bernstein's CDB information page. The CDB file format makes it easy to quickly retrieve data associated with an arbitrary binary key.

Once written, CDB files cannot be updated with new records. In ZCDB, a CDB file is written out with WITH-OUTPUT-TO-CDB and ADD-RECORD, then read with LOOKUP.

Example Use

Here is a simple way to convert a Unix system password file to a CDB file:

(defpackage #:pwdb
  (:use #:cl #:zcdb))

(defun username (password-line)
  (let ((end (position #\: password-line)))
    (subseq password-line 0 end)))

(defun string-octets (string)
  (babel:string-to-octets string :encoding :utf-8))

(defun octets-string (octets)
  (babel:octets-to-string octets :encoding :utf-8))

(defun make-password-cdb (password-file cdb-file temp-file)
  (with-open-file (stream password-file)
    (with-output-to-cdb (cdb cdb-file temp-file)
      (loop for line = (read-line stream nil)
            while line do
            (let ((username (username line)))
              (add-record (string-octets username)
                          (string-octets line)
                          cdb))))))

(defun user-info (username cdb-file)
  (let ((entry (lookup (string-octets username) cdb-file)))
    (when entry
      (octets-string entry))))

Then the database can be created and queried:

* (make-password-cdb "/etc/passwd" "/tmp/pwdb.cdb" "/tmp/pwdb.cdb.tmp")
#P"/tmp/pwdb.cdb"

* (user-info "root" "/tmp/pwdb.cdb")
"root:*:0:0:System Administrator:/var/root:/bin/sh"

Limitations

The CDB file format offers many opportunities for low-level interaction, including writing and reading records that don't fit in memory by writing or reading them incrementally. ZCDB currently only offers a simplified interface that works with keys and values that are fully loaded in memory as (unsigned-byte 8) vectors.

ZCDB does not provide any functions for serializing various data types (such as strings) to vectors. There are many other third-party libraries for that purpose.

The ZCDB Dictionary

ZCDB exports the following symbols.

with-output-to-cdb (cdb cdb-pathname temporary-pathname) &body body => |

Evaluates body with cdb bound to a cdb writer object. cdb may be used as the target of ADD-RECORD operations.

The cdb in progress is written to temporary-pathname. Any existing file with that pathname is overwritten. When writing completes successfully, temporary-pathname is renamed to cdb-pathname with CL:RENAME-FILE. For atomic operation, both files should be on the same filesystem.

add-record key value cdb => |

Adds a record for key and value to cdb, which should be a cdb writer object created in the dynamic scope of WITH-OUTPUT-TO-CDB.

key and value must both be vectors specialized to hold (unsigned-byte 8) data.

lookup key cdb => value

Looks up key in cdb and returns its value, or nil if no record in cdb has the given key.

key must be a specialized (unsigned-byte 8) vector. If the value is not null, it will also be a specialized (unsigned-byte 8) vector.

cdb must be either a pathname, or an input stream of element-type (unsigned-byte 8) that can be repositioned with CL:FILE-POSITION.

map-cdb function cdb => |

For each record in cdb, function is called with two arguments, the key vector and the value vector, respectively.

cdb must be either a pathname, or an input stream of element-type (unsigned-byte 8) that can be repositioned with CL:FILE-POSITION.

Feedback

If you have any questions or comments about ZCDB, please email me, Zach Beane.

License

Copyright © 2010 Zachary Beane

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

2010-09-21