| Main |

SQLite and Mac OS X

SQLite is quite a popular library, particularly on Mac OS X where it's actually used to implement some system features.

Unfortunately, some of the filesystems on OS X (AFP, for instance) don't implement advisory locking via the POSIX function fcntl(), which leads SQLite to report that the "database is locked" whenever you try to use it.

Now, SQLite is included with Mac OS X, and Apple have been kind enough to solve this problem for the version that they provide (3.1.3). Unfortunately, that version doesn't support some of the syntax that gets generated by newer versions of SQLAlchemy, which I've been evaluating as part of a new design for my company's website. So I needed to upgrade to a newer version to make it work, and since my home area is on an AFP share, I needed it to work on that.

The upshot is that I've ported Apple's fix from 3.1.3 to the latest version of SQLite (3.3.6). Here's a patch. To apply it, download the patch and the sqlite 3.3.6 sources (1.59MB). Then, in Terminal, do something like the following:

alastair$ tar xvzf ~/Desktop/sqlite-3.3.6.tar.gz
sqlite-3.3.6/
sqlite-3.3.6/art/
sqlite-3.3.6/art/tmp/
...
sqlite-3.3.6/publish.sh
alastair$ cd sqlite-3.3.6
alastair$ patch -p1 < ~/Desktop/sqlite-3.3.6-osx.patch

Then build it as normal. Note that if you run make test on an AFP volume, you'll spot that one or two of the tests will fail. I think this is because of the differences in semantics between POSIX and AFP byte-range locks. If you run them from a local volume instead, they succeed as expected.

If you want the new build to match the version supplied by Apple, remember to configure it with --enable-threadsafe and --enable-tcl to turn on thread-safety and Tcl support.

Trackbacks

TrackBack URL for this entry:
http://alastairs-place.net/movabletype/mt-tb.cgi/77.

Comments

When I attempted to apply the patch it ended like this:

$ cd sqlite-3.3.6
$ patch -pl patch: **** strip count l is not a number

The value displayed is a lower-case 'L', not a numeric '1'.
I proceeded to eyeball the patch file, looking for an instance where it would appear that a numeric '1' was intended, and a character 'l' appeared instead, but found nothing.
Is this a normal conclusion to the patch operation?

Also, for the absolute dweeb (that would be me), when you say "build it as normal", exactly what sequence of commands should I enter, and where do I set the --enable-threadsafe and --enable-tcl options (or does it prompt for overrides?) I assume it's just as described in the README:

mkdir bld ;# Build will occur in a sibling directory
cd bld ;# Change to the build directory
../sqlite/configure ;# Run the configure script
make ;# Run the makefile.
make install ;# (Optional) Install the build products

Correct?
Will the configure script prompt me for the overrides, or do I need to edit the configure script?
Or should I be editing the generic "Makefile.linux-gcc" makefile to specify these parms?

You've got the patch command wrong. The character after the -p is a number one (i.e. 1), not a lower-case letter 'L' (l). You should be able to see the difference in the Courier font that normally gets used for <kbd>, <pre> and <code> tags (1 vs l).

As regards building it, yes, you just follow the instructions in the README, so something like the following should work:

mkdir build
cd build
../sqlite/configure --enable-threadsafe --enable-tcl
make
sudo make install

The two changes relative to what you wrote are that the options go on the end of the configure command, and you'll want to use sudo to do the install (it'll ask for your password so it can write to restricted parts of the filesystem).

I know SQLite is used by Apple's Core Data framework. I was wondering if you know whether upgrading to the latest SQLite version will break the Core Data support in Cocoa or not. Hopefully the latest version is backward compatible enough to replace the older version which shipped with Tiger 10.4.

No, this won't affect Core Data, as long as you make sure that you install SQLite in /usr/local (or you can use Fink or DarwinPorts in which case they'll put it into their respective directory trees).

Basically, don't overwrite the system copy of SQLite; Apple haven't tested against other versions and you stand a good chance of breaking something or other if you do that.

It looks like Sqlite v 3.3.8 added support for this:

from: http://www.sqlite.org/changes.html#version_3_3_8

  • Added OS-X locking patches (beta - disabled by default)

Line 20 in src/os_unix.c is commented out but has this #define

/* #define SQLITE_ENABLE_LOCKING_STYLE 0 */

At line 1816 there are about 550 lines of code confitional compiled of which 300 are for dealing with AFP-STYLE LOCKING.

They also state however (www/lockingv3.tcl):

SQLite uses POSIX advisory locks to implement locking on Unix. On
windows it uses the LockFile(), LockFileEx(), and UnlockFile() system
calls. SQLite assumes that these system calls all work as advertised. If
that is not the case, then database corruption can result. One should
note that POSIX advisory locking is known to be buggy or even unimplemented
on many NFS implementations (including recent versions of Mac OS X)
and that there are reports of locking problems
for network filesystems under windows. Your best defense is to not
use SQLite for files on a network filesystem.

So it looks like somebody's been working on it but you'd have uncomment the define in os_unix.c and test it.

Post a comment

If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thank-you for your patience.

(Your e-mail address will not be displayed or included in any pages served on this site; nor will you get any spam as a result.)

A live preview of your comment will be displayed below. It should refresh automatically when you stop typing, but if not then the “Preview” button above will update it.

Live Comment Preview