Development

Exim4-LDAP 0.0.7 released

Exim4-LDAP 0.0.7 is a development release, which fix a lot of hidden bugs, more comprehensive clone of Qmail-LDAP individual account features, better documentation and coding syntax, and even more. I am confident that the code is stable enough for wider testing by the community, download now.

Spotlight changes include:

  • Add support with Qmail-LDAP accountStatus attribute.
  • Add support with Qmail-LDAP deliveryProgramPath attribute.
  • Activate home_directory in transport with relative patch support.
  • Add LDAP_MAILROOT support if homeDirectory is not absolute.
  • Debug program pipe transport.
  • Add sample LDIF for demo user demo@example.com.
  • Document installation guideline in INSTALL.

Please refer to /usr/share/doc/exim4-ldap/changelog.gz for complete changelog. Changes since 0.0.5:


Exim4-LDAP 0.0.5 initial released

Exim4-LDAP is my latest development progress. It is a Debian package that depends on exim4-heavy-daemon and slapd, which utilize Qmail-LDAP schema as backend database schema for seamless migration. Unlink Qmail-LDAP which required for manual source patching and compile, Exim4-LDAP just define additional authentication/router/transfer rules within configuration files. The ultimate goal of this research project is going to integrate with Samba-LDAP-PAM (for both Windows/Linux Single-Sign-On), RADIUS (for other services SSO) and even other services such as webmail. The project is now hosting in SourceForge.net with .deb available, download now.

Here I will draft some simple installation guideline which not yet included in source package document. I am now testing this with a newly installed Debian Lenny sandbox.


Mail sync hack for IT-School intranet mail with normal mail server

IT-School (http://myit-school.net/) is a school management system in Hong Kong, which target for intranet mail, calender, account management, and so on. BTW, for the old version (around 2004) of IT-School, its mail system is split into 3 parts:

  • Intranet mail: Mail from and to account will fetch from MySQL DB directly, mail will NEVER deliver though normal email server router BUT save into MySQL DB as HTML format DIRECTLY. Support individual user and group of users.
  • Outgoing internet mail: Mail will send though sendmail.
  • Incoming internet mail: Mail will first received by sendmail as normal email format. User will not able to browse external mail UNLESS they press a "Check email" link from the GUI. Once they press this link, a small window will pop-up, inform user that IT-School will fetch and convert the real email as its own format, and save into its MySQL DB as intranet mail.

The send out email action preform 2 part (/usr/local/it-school/www/php/intra/savemail.php3):

  1. Handle all intranet email transaction.
                    //        send to other user
                    intramail($mail_id);
  2. Send internet email though MTA.
                    //              send e_mail
                    //        check privilege
                    if ($support_sending_email && checkprivilege($userid, 'send_email')) {
                            send_email($mail_id);
                    }

How complicated it is! In order to migrate the old system to normal MTA + eGroupWare, I did some trick in IT-School's source code for:

  1. Hack the send_email() function (/usr/local/it-school/www/php/intra/imapfunction.php3), search out all internal users and groups, generate a public email for them (if user's ~/.forward exists and contain valid forward address), and send out another copy though normal MTA:
    diff -urpN intra.20090216/imapfunction.php3 intra/imapfunction.php3
    --- intra.20090216/imapfunction.php3 2009-02-13 10:00:42.000000000 +0800
    +++ intra/imapfunction.php3 2009-02-16 09:52:34.000000000 +0800
    @@ -174,6 +174,93 @@ function send_email($mail_id) {
       get_email_list($unprocess_bpgroupid, $bcc_puser, $bcc_pemail);
       $pemail_source = array();

    +  // XXX: Check and forward intra email as internet email to mail2.
    +
    +  global $db_database;
    +  global $db_host;
    +  global $db_user;
    +  global $db_password;
    +  $db = new Mysql_DB_Sql($db_database, $db_host, $db_user, $db_password);
    +
    +  // Fetch loginid from userid, generate mail2. address, and append to internet_* as public mail.
    +  foreach (array('recipient', 'cc', 'bcc') as $target) {
    +    $_loginids = array();
    +    $_email_address_array = array();
    +    foreach (explode(',', $sourcemail['internet_' . $target]) as $address) {
    +      $_email_address_array[$address] = $address;
    +    }
    +
    +    // Grep all indiviual user IDs.
    +    if (count($sourcemail[$target]['user']) > 0) {
    +      $sql = "SELECT loginid FROM ituser WHERE userid IN (" . implode(', ', $sourcemail[$target]['user']) . ")";
    +      $db->query($sql);
    +      while ($db->next_record()) {
    +        $_loginids[] = $db->f('loginid');
    +      }
    +    }
    +
    +    // Grep all userid from group, then replace as loginid.
    +    if (count($sourcemail[$target]['group']) > 0) {
    +      $sql = "SELECT userid FROM user_usergroup WHERE groupid IN ('" . implode("', '", $sourcemail[$target]['group']) . "')";
    +      $db->query($sql);
    +      $_userid = array();
    +      while ($db->next_record()) {
    +        $_userid[] = $db->f('userid');
    +      }
    +
    +      $sql = "SELECT loginid FROM ituser WHERE userid IN (" . implode(', ', $_userid) . ")";
    +      $db->query($sql);
    +      while ($db->next_record()) {
    +        $_loginids[] = $db->f('loginid');
    +      }
    +    }
    +
    +    $_user = '[a-zA-Z0-9_\-\.\+\^!#\$%&*+\/\=\?\`\|\{\}~\']+';
    +    $_domain = '(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.?)+';
    +    $_ipv4 = '[0-9]{1,3}(\.[0-9]{1,3}){3}';
    +    $_ipv6 = '[0-9a-fA-F]{1,4}(\:[0-9a-fA-F]{1,4}){7}';
    +
    +    // If convert required, do so.
    +    if (count($_loginids) > 0) {
    +      foreach ($_loginids as $_loginid) {
    +        $path = '/home/' . $_loginid . '/.forward';
    +        if ($handle = @fopen($path, 'r')) {
    +          while (!feof($handle)) {
    +            $buffer = fgets($handle, 4096);
    +            $buffer = trim($buffer);
    +            if (preg_match("/^$_user@($_domain|(\[($_ipv4|$_ipv6)\]))$/", $buffer)) {
    +              $_email_address_array[$buffer] = $buffer;
    +            }
    +          }
    +          fclose($handle);
    +        }
    +      }
    +      $sourcemail['internet_' . $target] = implode(',', $_email_address_array);
    +    }
    +  }
    +
    +  // Print debug message within header.
    +  $array = array(
    +    'internet_recipient_orig' => $internet_recipient_orig,
    +    'internet_recipient_new' => $internet_recipient_new,
    +    'loginids' => $loginids,
    +    'sourcemail' => $sourcemail,
    +    'recipient_puser' => $recipient_puser,
    +    'recipient_pemail' => $recipient_pemail,
    +    'cc_puser' => $cc_puser,
    +    'cc_pemail' => $cc_pemail,
    +    'bcc_puser' => $bcc_puser,
    +    'bcc_pemail' => $bcc_pemail,
    +    'pemail_source' => $pemail_source,
    +  );
    +  ob_start();
    +  var_dump($array);
    +  $contents = ob_get_contents();
    +  ob_end_clean();
    +  print "\n\n<!-- $contents -->\n\n";
    +
    +  // XXX: Check and forward intra email as internet email to mail2.
    +
    //  join them
       $email_address_array = $process_userid;
    //  $email_address_array = array_merge($process_userid, $recipient_pemail, $cc_pemail, $bcc_pemail);
  2. Trigger "Check email" action though JS periodically (/usr/local/it-school/www/php/intra/index.php3):
    diff -urpN intra.20090216/index.php3 intra/index.php3
    --- intra.20090216/index.php3 2009-02-15 19:02:37.000000000 +0800
    +++ intra/index.php3 2009-02-16 09:49:22.000000000 +0800
    @@ -254,6 +254,13 @@ if ($can_check_email) {
       echo 'email = window.open("check_email.php?"+Math.random(), "CheckEmail","status=1,width=200,height=150");';
         echo 'email.focus();';
       echo '}';
    +
    +  // XXX: Auto open "Check email" window for every 5mins.
    +  echo 'var fm_timerFolderStatus;';
    +  // refresh after 5min.
    +  echo 'var refreshTimeOut = 300000;';
    +  echo 'fm_timerFolderStatus = window.setInterval("check_email()", refreshTimeOut);';
    +  // XXX: Auto open "Check email" window for every 5mins.
    }
    ?>
    </SCRIPT>
  3. Close the pop-up window though JS automatically (/usr/local/it-school/www/php/intra/check_email.php):
    diff -urpN intra.20090216/check_email.php intra/check_email.php
    --- intra.20090216/check_email.php 2009-02-15 20:43:03.000000000 +0800
    +++ intra/check_email.php 2009-02-16 09:49:36.000000000 +0800
    @@ -57,6 +57,11 @@ function refresh_parent() {
    function goBack() {
       window.close();
    }
    +
    +// XXX: Auto close confirm window after 1s.
    +fm_goBack = window.setInterval("goBack()", 1000);
    +// XXX: end Auto close confirm window after 1s.
    +
    //-->
    </SCRIPT>
    <BODY>

Now any IT-School email transaction will be cloned to the new system :D

P.S. Thanks for the experience of hacking Drupal, this hack become much simple: it only cost for 2 working days. Some checking are cloned from Drupal driectly, e.g. email address validation, pipe console output as plain text, and even the skill for patching source. Well, skills will always become weapon, and become useful whenever you need for it :D


dev note: update text fields as nullable and no default

Prepare schema:

  1. Check target fields (http://edin.no-ip.com/node/166).
    a. No default + NOT NULL => can't accept NULL so valid value only (1 status only)
    b. Default with '' => remove
  2. Update schema with "NOT NULL => FALSE, (remove default)".
  3. Running update.php. Ensure schema is valid.
  4. Run related simpletest.

Once schema ready for expert install:

  1. ORACLE_NULL => NULL
  2. Before insert: check '' => NULL (!?)
  3. Before normal variable binding: '' => NULL
  4. check SQL syntax, change from = '' to IS NULL

Testing procedure:

  1. expert mode -> patch -> update
  2. normal mode -> patch -> update
  3. normal mode + all modules -> patch -> update
  4. patch -> install -> all modules


Dummy code for makeConstraintName()

<?php
$table
= 'menu_links';
$fields = array('menu_name', 'plid', 'expanded', 'has_children');
$name = 'menu_plid_expand_child';

// Clone from PostgreSQL _createIndexSql() handling.
$result = Database::getActiveConnection()->prefixTables('{' . $table . '}_' . $name . '_idx');
var_dump($result);

// Handle with makeIndexName().
$result = Database::getActiveConnection()->makeIndexName($table, $fields);
var_dump($result);
?>


BarCamp Hong Kong 2008: What Can I contribute for it?

barcamp Hong Kong 2008

A good invitation from Groups.Drupal (Hong Kong), about BarCamp Hong Kong 2008. I will try to share my research founding with Drupal 7.x + Enterprise DB backend, e.g. Oracle, DB2 and MSSQL; some tricky skill for integrate 3rd layout design with Drupal; also hope to gather with Hong Kong Drupaler and discover for our possibility.

Are you living in Hong Kong? Are you using or interesting with Drupal? Do you hope to meet more friends and share your idea? Is time to show up, and I will be there waiting for you. Let's meet on tomorrow :D


Wirtting query in reserved word safe: any possible solution?

Reserved word conflict is an always pain for multiple DB backend supported system: Drupal also face this for many years (e.g. http://drupal.org/node/371, since June 30, 2002). This is the most complicated bottleneck for daily maintenance and introduce any new DB backend to Drupal: whenever existing DB vendor change their standard, we need to follow it; whenever a new DB backend introduce, another set of reserved words are coming in, too.

There is many possible solution, e.g. prevent use of ANY reserved words based on requirement of ALL supported database engines (as like as Moodle's implementation); BTW, as this solution will left issue as ALWAYS OPEN, I am not going to discuss within this review.

Another possibility is the use of escape characters, e.g. `` for MySQL, "" for PostgreSQL/Oracle/DB2 and [] for MSSQL. This approach is widely used for many successful projects, e.g. phpmyadmin, phppgadmin, Oracle/DB2/MSSQL EM. The primary consideration is: How to implement this syntax for Drupal, which is simple and elegant for both end user and DB abstraction designer?

For sure that we will need some update for existing Drupal queries syntax, where the ultimate goal is query will finally present as following example before process by PHP database connection (e.g. SELECT batch FROM {batch} WHERE bid = :bid AND token = :token from line 15 of includes/batch.inc, CVS HEAD, rewrite with TNG DB standard). In case of MySQL, after translation it should looks like:

SELECT `batch` FROM `batch` WHERE `bid` = :bid AND `token` = :token

I will discuss different syntax change approach, based on: a) user experience, b) similar to ANSI coding style, c) backward compatibility, d) backend complexity, e) overall performance and f) workload for syntax conversion. A simple marking will be given for each item (e.g. good/functioning: 2; fine/not bad: 1; poor/not functioning: 0) so we can have some simple idea between different proposals.

Here is a quick rundown of my review:


Drupal 6.x + PDO_OCI: finally work together

After put down my multiple database backend research for Drupal among past 4 months, now finally get Drupal 6.x and PDO_OCI working together... It is all about my incorrect pdo_oci driver implementation... The progress is now available in here.

What is the bottleneck?

The main bottleneck of pdo_oci driver implementation is all about size of VARCHAR2 support under AL32UTF8. When working with AL32UTF8, as characters may store up to 2 or 3 bytes, the original maximum 4000 characters support of VARCHAR2 will cut off as 1333 (4000/3). If a single table coming with more than 1 column larger than 1333 characters, an annoying ORA-01461 error message MAY occur. This error message will not trigger when using oci8, but always happened with pdo_oci + Drupal 6.x (Siren 1.x)...

Any tries?

I have try some method to escape from this ORA-01461 error, including:

  1. Reduce most text field size mapping as VARCHAR2(1333), which only left text:big map with VARCHAR2(4000).
  2. Replace some fields as BLOB (e.g. fields storing serialized data) for unlimited storage size support, where restrict some other fields as varchar 256 or even smaller.

But above tries both face some critical error, and so need to rollback:

  1. Some fields (e.g. {menu_router}.access_arguments and {menu_router}.access_arguments) can't use VARCHAR2(1333) only, as they are storing serialized data; BTW, if mapping both to VARCHAR2(4000), ORA-01461 will occur.
  2. BLOB field need special handling (i.e drupal_write_record() and db_decode_blob()) which will complicate the I/O task, therefore not suitable for some special fields (e.g. again {menu_router}.access_arguments and {menu_router}.access_arguments). Since {menu_router} will finally store its rendered result in Drupal's cache system (already using BLOB field for storage), a DUPLICATED BLOB I/O handling result as a very fancy and buggy implementation.

Finally...

Well... Back to the main fantastic problem: as Drupal core hack are share among oci8 and pdo_oci version (backend independent), ~60% code and ~80% Oracle related programming logic are shared among oci8 and pdo_oci, where ~80% PDO related programming logic are also shared among pdo_pgsql and pdo_oci, I switch my focus to debug PHP's pdo_oci driver and my Drupal + pdo_oci driver implementation.

So, what is the problem? Christopher Jones on OPAL give me some hints: I have a wrong transaction handling in case of pdo_oci driver implementation. When working with Oracle + BLOB (or even not with BLOB) a transaction is always required, plus a try {} catch (Exception $e) {} in further more operation. Within my old implementation (trimmed version, share and clone between legacy and other PDO driver):

<?php
  $active_db
->beginTransaction();
 
// Execute the statement.
 
$result = $stmt->execute();
  if (
$result === FALSE) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
 
// Some PDO + BLOB stream I/O handling code.
  // On success, commit the transaction.
 
$active_db->commit();
?>

The correct implementation should be:

<?php
  $active_db
->beginTransaction();
  try {
   
// Execute the statement.
   
$result = $stmt->execute();
   
// Some PDO + BLOB stream I/O handling code.
    // On success, commit the transaction.
   
$active_db->commit();
  }
  catch (
Exception $e) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
?>

Conclusion

Even the research progress is functioning with Drupal 6.x, a lot of effort is still required. As Drupal 7.x will completely switch to PDO, where core database drivers (including MySQL and PostgreSQL) implementation are revamped with OOP, I still need to catch up my missing section among the past 4 months.

The patch for next generation is now RTBC, should I wait for its commit and propose my amendment afterward, or keep hard working and propose the amendment before everything shuttle down?


PHP debugging...

Finding way to debug PHP, since pdo_oci keep on crash with Drupal (Siren) installation and operation. Seems to be problem of BLOB insertion and deletion. P.S. same code base are function with oci8 (i.e. PHP and Oracle are both function) and pdo_pgsql (i.e. programming logic for Drupal + PDO is function)...

Remember to compile PHP with --enable-debug and --enable-xdebug options. Still don't know how to play with vim + Xdebug...

PHP + Xdebug:
http://www.php.net/manual/en/install.pecl.static.php
http://www.xdebug.org/
http://2bits.com/articles/using-vim-and-xdebug-dbgp-for-debugging-drupal...

PHP + Oracle:
http://blogs.oracle.com/opal/

PHP bug trace and issue thread:
http://bugs.php.net/bugs-generating-backtrace.php
http://bugs.php.net/search.php?cmd=display&search_for=pdo_oci&x=0&y=0


Siren: dev check list + bug list + footnote

http://192.168.93.21/
http://192.168.93.21/phpmyadmin/
http://192.168.93.21/phppgadmin/
https://192.168.93.21:1158/em/

http://192.168.93.21/projects/benchmarking/drupal-7.x-dev/
http://192.168.93.21/projects/benchmarking/siren-2.x-dev/

http://192.168.93.21/projects/benchmarking/drupal-7.x-dev-pdo_mysql/
http://192.168.93.21/projects/benchmarking/drupal-7.x-dev-pdo_pgsql/

http://192.168.93.21/projects/benchmarking/siren-2.x-dev-pdo_mysql/
http://192.168.93.21/projects/benchmarking/siren-2.x-dev-pdo_pgsql/
http://192.168.93.21/projects/benchmarking/siren-2.x-dev-pdo_oci/ (dev, 95%)
http://192.168.93.21/projects/benchmarking/siren-2.x-dev-pdo_sqlite/ (dev, 70%)
http://192.168.93.21/projects/benchmarking/siren-2.x-dev-pdo_ibm/ (dev, 80%)

http://192.168.0.88/
http://192.168.0.88/phpmyadmin/
http://192.168.0.88/phppgadmin/
http://192.168.0.88:1158/em/
http://192.168.0.88/projects/benchmarking/drupal-7.x-dev/
http://192.168.0.88/projects/benchmarking/siren-2.x-dev/

MySQL flush DB:

DROP DATABASE AL32UTF8;
CREATE DATABASE AL32UTF8;

PostgreSQL flush DB:

su - postgres
dropdb AL32UTF8
createdb -O root AL32UTF8

Oracle flush DB:

DROP USER ROOT CASCADE;

CREATE USER ROOT PROFILE "DEFAULT" IDENTIFIED BY "CHANGE" DEFAULT TABLESPACE "USERS" TEMPORARY TABLESPACE "TEMP" ACCOUNT UNLOCK;
GRANT "AQ_ADMINISTRATOR_ROLE" TO ROOT;
GRANT "AQ_USER_ROLE" TO ROOT;
GRANT "AUTHENTICATEDUSER" TO ROOT;
GRANT "CONNECT" TO ROOT;
GRANT "CTXAPP" TO ROOT;
GRANT "DBA" TO ROOT;
GRANT "DELETE_CATALOG_ROLE" TO ROOT;
GRANT "EJBCLIENT" TO ROOT;
GRANT "EXECUTE_CATALOG_ROLE" TO ROOT;
GRANT "EXP_FULL_DATABASE" TO ROOT;
GRANT "GATHER_SYSTEM_STATISTICS" TO ROOT;


Syndicate content