edoneel daily - daily stuff on random subjects

MySQL Backend discussion

This is a day by day journal of adding a new backend to MySQL.

17 May 2006

Hmm, not so day by day...

Well, there are not good instructions for building a pluggable storage engine in MySQL 5.1 yet (as of today it's 5.1.9). That's the bad news. The good news is that it's now much easier to build a builtin storage engine with MySQL. There are very few files (configure.in/configure) and (sql/handler.h with one enum) on top of the files you need for your storage engine.

Therefore I've gone back to the built-in storage engine and it's here. You untar this file overtop of MySQL 5.1.9 and then you configure and make. Please read the README. Please do not use this in any sort of production environment. I have no idea how unstable this might make MySQL.

16 Mar 2006

Well, it wasn't too hard to port things to MySQL 5.1.7 Beta. First, things got simplier. We only have to modify the configure.in file in the top level directory. We had to change ha_eclstorage.[h|cc] to match some changes made internally in MySQL. The rest of the changes were just so that MySQL would recognize a new storage engine.

The one catch is that after you untar things you MUST

otherwise automake tries to run. I need to read more on automake but that can wait.

The configure line becomes ./configure --prefix=/usr/local/mysql --enable-assembler --with-mysqld-ldflags=-all-static --prefix=/usr/local/mysql --with-named-curses-libs=/usr/lib/libncursesw.so.5 --with-blackhole-storage-engine --with-eclstorage-storage-engine

16 Feb 2006

MySql

MySQL reference manual on Storage Engines. In the source, the example engine lives in sql/examples/ha_example.cc and the blackhole one lives (oddly enough) both in sql/ha_blackhole.cc and libmysqld/ha_blackhole.cc. Somewhere there's going to have to be some mechanism to include, or not, the different storage engines since you can specify --with-blackhole-storage-engine to configure. Ah, it seems that the magic occurs with a #define via configure and then some code in libmysqld/handler.cc. Hmmm...

23 Feb 2006

MySQL

I'm starting a new project to add a new storage engine to MySQL. I'll be using ECL - Embedded Common Lisp.

Why? Because I think it will be interesting. Also a number of the current storage engines have changed hands recently, so, this might help MySQL. Finally I think I can do something useful here with Lisp.

For now I'll be using version 5.0.18 and I'll be using the infrastructure that the NetBSD pkgsrc package system supplies. Why? I use NetBSD on an iBook and it's easy. One basic driver here is that I do have a life so I can't spend hours every day. This is also one of my drivers to use lisp

The entries related to this will be repeated in the link in the upper left of the page titled MySQL Storage Engine Discussion

So, let's get started. Step one for those of you following at home is to download the NetBSD packages collection, follow what ever installation instructions are needed for your OS, and then 'make install' databases/mysql5-server. Now we can start poking around.

Under work/mysql-5.0.18 you will see the full source tree. Handily we can work from an example which always makes things easier. Even handier someone thought to name one of the storage engines the blackhole one. While the source comment is "Dumbest named feature ever" it makes it very easy to find. Much easier than heap, sample, test, etc, which appear all over typical source trees.

A quick find . -name \*black\* gives us:

Also a quick find shows us that, for the most part, not too many files have to be modified. We'll have to write the new versions of ha_blackhole.[cc|h] of course to interface into ecl, and we have to write the actual code in ecl to do the work, and finally we'll have to make a change or two to some other files to allow our new storage engine to actually be used.

Next steps. First we have to come up with a name. I've poked a bit and ldb, cldb, and ecldb are all taken. So is repldb (for read, eval, print loop). I'll mull this a bit. Second we'll take baby steps to make this work. The first version will do nothing more than throw stuff away, ie, just like the blackhole engine. Then we'll advance all the way up to the memory engine functions where we'll store stuff, but, not on disk. Then, with luck, we'll be able to save stuff to disk and be quick about it.

  • and some files in mysql-test

    • Also a quick find shows us that, for the most part, not too many files have to be modified. We'll have to write the new versions of ha_blackhole.[cc|h] of course to interface into ecl, and we have to write the actual code in ecl to do the work, and finally we'll have to make a change or two to some other files to allow our new storage engine to actually be used.

    Next steps. First we have to come up with a name. I've poked a bit and ldb, cldb, and ecldb are all taken. So is repldb (for read, eval, print loop). I'll mull this a bit. Second we'll take baby steps to make this work. The first version will do nothing more than throw stuff away, ie, just like the blackhole engine. Then we'll advance all the way up to the memory engine functions where we'll store stuff, but, not on disk. Then, with luck, we'll be able to save stuff to disk and be quick about it.

    Update: I've decided that the name will be eclstorage

    24 Feb 2006

    MySQL

    Got the configuration, it seems, to work. I added or changed the following files:

    All of these are in eclstorage-20060223.tar.gz. I'd put them a patch file for the patch command if someone could tell me how to add a file with patch.

    Now one can run 'make' in the databases/mysql5-server directory and the right thing happens with configure.

    I did add the following two changes to the pkgsrc Makefile.

    To rerun the configure in pkgsrc you need to rm work/.configure_done before you rerun 'make'

    Do NOT run 'make clean' in the pkgsrc directory. Run 'gmake clean' in work/mysql-5.0.18.

    26 Feb 2006

    MySQL

    Now we have enough changes so that we've added a copy of the blackhole storage engine but with a new name, eclstorage. The following files are changed (beyond the ones from before)

    All of these are in eclstorage-20060226-3.tar.gz.

    16 Mar 2006

    MySQL

    Well, it wasn't too hard to port things to MySQL 5.1.7 Beta. First, things got simplier. We only have to modify the configure.in file in the top level directory. We had to change ha_eclstorage.[h|cc] to match some changes made internally in MySQL. The rest of the changes were just so that MySQL would recognize a new storage engine.

    The one catch is that after you untar things you MUST

    otherwise automake tries to run. I need to read more on automake but that can wait.

    The configure line becomes ./configure --prefix=/usr/local/mysql --enable-assembler --with-mysqld-ldflags=-all-static --prefix=/usr/local/mysql --with-named-curses-libs=/usr/lib/libncursesw.so.5 --with-blackhole-storage-engine --with-eclstorage-storage-engine

    These changes live in eclstorage-20060315.tar.gz

    Bruce O'Neel

    Last modified: Tue Mar 16 14:45:53 MET 2006