Deliverable 4

Description: In this deliverable, I am experimenting with the N-gram parser plugin for MySQL. This plugin is required if we are trying to use the full-text search of the MySQL for languages like Japanese, Chinese and Korean. The full-text search makes use of indices against some fields in the tables, which we can define at the time of creating a table. Let us consider following example for better understanding of the full-text search functionality in MySQL.
e.g. We have following table in MySQL:

Suppose user performs a search on 'Names' field in the above table with some part of the name say search for names having 'John' in them. Above table has only one record with names as 'Johnson'. It is possible to use wildcard or commands 'LIKE', but full-text searches are faster than any other methods of searching. The full-text searching is ideal for large databases that contain thousands or millions of rows.
Why do we need the full-text search plugin for the Japanese language? Why can't we use the existing full-text search functionality in MySQL?
The main difference between languages like English and Japanese is, the Japanese language has not word delimiters. There are no spaces in between any sentences written in the Japanese language. The MySQL full-text index search is word based index and it cannot be used for no word delimiter languages like Japanese and Chinese. This n-gram parser uses this plugin interface to implement a simple n-gram (bi-gram) fulltext index parser which can index no word delimiter laungage.

Instructions for compiling and installing the N-gram plugin:

• Download bigram plugin from: http://sourceforge.net/projects/mysqlftppc/
• Once you download the tar file, extract it under some folder
• Now run configure script. Use following command:
  Woody:mysqlftppc-bigram-1.6 sujata$ ./configure CFLAGS='-arch i686' LDFLAGS='-arch i686' CC='gcc -m32' CXX='g++ -m32' CHOST='i686-apple-darwin10.0.0' --target='i686-apple-darwin10.0.0' --build='i686-apple-darwin10.0.0' --host='i686-apple-darwin10.0.0' --with-mysql-config=/Applications/XAMPP/xamppfiles/bin/mysql_config
• Once you are done with the compilation, run following commands for installation:
  Woody:mysqlftppc-bigram-1.6 sujata$ make CFLAGS='-arch i686' LDFLAGS='-arch i686' CC='gcc -m32' CXX='g++ -m32'
  Woody:mysqlftppc-bigram-1.6 sujata$ sudo make install
• This will install the plugin libraries in /Applications/XAMPP/xamppfiles/lib/mysql/plugin. You will find the file with the name as 'libftbigram.so' under this folder.
• Once you are done with the installation, open my.cnf file from /Applications/XAMPP/xamppfiles/etc
• Search for '[mysqld]' in my.cnf file and add following lines in my.cnf file:
  # The MySQL server
  [mysqld]
  ft_min_word_len = 1
  plugin_dir = /Applications/XAMPP/xamppfiles/lib/mysql/plugin
• Restart the MySQL server since my.cnf file is modified.
• After restarting the MySQL server, connect to the MySQL server by following command:
  Woody:mysqlftppc-bigram-1.6 sujata$ mysql -uroot -p
  Enter password:
  Welcome to the MySQL monitor. Commands end with ; or \g.
  Your MySQL connection id is 8
  Server version: 5.1.37 Source distribution
  
  Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
  mysql>
• Now, type the command for installing plugin:
  mysql>INSTALL PLUGIN bigram SONAME 'libftbigram.so'
• To check whether the plugin is added successfully or not, you can use the following command:
  mysql> SHOW PLUGINS;
  You should get the records as follows:
  

Usage:

References:

• http://sourceforge.net/apps/mediawiki/mysqlftppc/index.php?title=Main_Page
• http://www.devarticles.com/c/a/MySQL/Getting-Started-With-MySQLs-Full-Text-Search-Capabilities/1/
• Compiling i386 dynamic library: http://discussions.apple.com/thread.jspa?messageID=10497361