PHP

Mail sync hack for IT-School intranet mail with normal mail server

IT-School (http://myit-school.net/) is a school management system in Hong Kong, which target for intranet mail, calender, account management, and so on. BTW, for the old version (around 2004) of IT-School, its mail system is split into 3 parts:

  • Intranet mail: Mail from and to account will fetch from MySQL DB directly, mail will NEVER deliver though normal email server router BUT save into MySQL DB as HTML format DIRECTLY. Support individual user and group of users.
  • Outgoing internet mail: Mail will send though sendmail.
  • Incoming internet mail: Mail will first received by sendmail as normal email format. User will not able to browse external mail UNLESS they press a "Check email" link from the GUI. Once they press this link, a small window will pop-up, inform user that IT-School will fetch and convert the real email as its own format, and save into its MySQL DB as intranet mail.

The send out email action preform 2 part (/usr/local/it-school/www/php/intra/savemail.php3):

  1. Handle all intranet email transaction.
                    //        send to other user
                    intramail($mail_id);
  2. Send internet email though MTA.
                    //              send e_mail
                    //        check privilege
                    if ($support_sending_email && checkprivilege($userid, 'send_email')) {
                            send_email($mail_id);
                    }

How complicated it is! In order to migrate the old system to normal MTA + eGroupWare, I did some trick in IT-School's source code for:

  1. Hack the send_email() function (/usr/local/it-school/www/php/intra/imapfunction.php3), search out all internal users and groups, generate a public email for them (if user's ~/.forward exists and contain valid forward address), and send out another copy though normal MTA:
    diff -urpN intra.20090216/imapfunction.php3 intra/imapfunction.php3
    --- intra.20090216/imapfunction.php3 2009-02-13 10:00:42.000000000 +0800
    +++ intra/imapfunction.php3 2009-02-16 09:52:34.000000000 +0800
    @@ -174,6 +174,93 @@ function send_email($mail_id) {
       get_email_list($unprocess_bpgroupid, $bcc_puser, $bcc_pemail);
       $pemail_source = array();

    +  // XXX: Check and forward intra email as internet email to mail2.
    +
    +  global $db_database;
    +  global $db_host;
    +  global $db_user;
    +  global $db_password;
    +  $db = new Mysql_DB_Sql($db_database, $db_host, $db_user, $db_password);
    +
    +  // Fetch loginid from userid, generate mail2. address, and append to internet_* as public mail.
    +  foreach (array('recipient', 'cc', 'bcc') as $target) {
    +    $_loginids = array();
    +    $_email_address_array = array();
    +    foreach (explode(',', $sourcemail['internet_' . $target]) as $address) {
    +      $_email_address_array[$address] = $address;
    +    }
    +
    +    // Grep all indiviual user IDs.
    +    if (count($sourcemail[$target]['user']) > 0) {
    +      $sql = "SELECT loginid FROM ituser WHERE userid IN (" . implode(', ', $sourcemail[$target]['user']) . ")";
    +      $db->query($sql);
    +      while ($db->next_record()) {
    +        $_loginids[] = $db->f('loginid');
    +      }
    +    }
    +
    +    // Grep all userid from group, then replace as loginid.
    +    if (count($sourcemail[$target]['group']) > 0) {
    +      $sql = "SELECT userid FROM user_usergroup WHERE groupid IN ('" . implode("', '", $sourcemail[$target]['group']) . "')";
    +      $db->query($sql);
    +      $_userid = array();
    +      while ($db->next_record()) {
    +        $_userid[] = $db->f('userid');
    +      }
    +
    +      $sql = "SELECT loginid FROM ituser WHERE userid IN (" . implode(', ', $_userid) . ")";
    +      $db->query($sql);
    +      while ($db->next_record()) {
    +        $_loginids[] = $db->f('loginid');
    +      }
    +    }
    +
    +    $_user = '[a-zA-Z0-9_\-\.\+\^!#\$%&*+\/\=\?\`\|\{\}~\']+';
    +    $_domain = '(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.?)+';
    +    $_ipv4 = '[0-9]{1,3}(\.[0-9]{1,3}){3}';
    +    $_ipv6 = '[0-9a-fA-F]{1,4}(\:[0-9a-fA-F]{1,4}){7}';
    +
    +    // If convert required, do so.
    +    if (count($_loginids) > 0) {
    +      foreach ($_loginids as $_loginid) {
    +        $path = '/home/' . $_loginid . '/.forward';
    +        if ($handle = @fopen($path, 'r')) {
    +          while (!feof($handle)) {
    +            $buffer = fgets($handle, 4096);
    +            $buffer = trim($buffer);
    +            if (preg_match("/^$_user@($_domain|(\[($_ipv4|$_ipv6)\]))$/", $buffer)) {
    +              $_email_address_array[$buffer] = $buffer;
    +            }
    +          }
    +          fclose($handle);
    +        }
    +      }
    +      $sourcemail['internet_' . $target] = implode(',', $_email_address_array);
    +    }
    +  }
    +
    +  // Print debug message within header.
    +  $array = array(
    +    'internet_recipient_orig' => $internet_recipient_orig,
    +    'internet_recipient_new' => $internet_recipient_new,
    +    'loginids' => $loginids,
    +    'sourcemail' => $sourcemail,
    +    'recipient_puser' => $recipient_puser,
    +    'recipient_pemail' => $recipient_pemail,
    +    'cc_puser' => $cc_puser,
    +    'cc_pemail' => $cc_pemail,
    +    'bcc_puser' => $bcc_puser,
    +    'bcc_pemail' => $bcc_pemail,
    +    'pemail_source' => $pemail_source,
    +  );
    +  ob_start();
    +  var_dump($array);
    +  $contents = ob_get_contents();
    +  ob_end_clean();
    +  print "\n\n<!-- $contents -->\n\n";
    +
    +  // XXX: Check and forward intra email as internet email to mail2.
    +
    //  join them
       $email_address_array = $process_userid;
    //  $email_address_array = array_merge($process_userid, $recipient_pemail, $cc_pemail, $bcc_pemail);
  2. Trigger "Check email" action though JS periodically (/usr/local/it-school/www/php/intra/index.php3):
    diff -urpN intra.20090216/index.php3 intra/index.php3
    --- intra.20090216/index.php3 2009-02-15 19:02:37.000000000 +0800
    +++ intra/index.php3 2009-02-16 09:49:22.000000000 +0800
    @@ -254,6 +254,13 @@ if ($can_check_email) {
       echo 'email = window.open("check_email.php?"+Math.random(), "CheckEmail","status=1,width=200,height=150");';
         echo 'email.focus();';
       echo '}';
    +
    +  // XXX: Auto open "Check email" window for every 5mins.
    +  echo 'var fm_timerFolderStatus;';
    +  // refresh after 5min.
    +  echo 'var refreshTimeOut = 300000;';
    +  echo 'fm_timerFolderStatus = window.setInterval("check_email()", refreshTimeOut);';
    +  // XXX: Auto open "Check email" window for every 5mins.
    }
    ?>
    </SCRIPT>
  3. Close the pop-up window though JS automatically (/usr/local/it-school/www/php/intra/check_email.php):
    diff -urpN intra.20090216/check_email.php intra/check_email.php
    --- intra.20090216/check_email.php 2009-02-15 20:43:03.000000000 +0800
    +++ intra/check_email.php 2009-02-16 09:49:36.000000000 +0800
    @@ -57,6 +57,11 @@ function refresh_parent() {
    function goBack() {
       window.close();
    }
    +
    +// XXX: Auto close confirm window after 1s.
    +fm_goBack = window.setInterval("goBack()", 1000);
    +// XXX: end Auto close confirm window after 1s.
    +
    //-->
    </SCRIPT>
    <BODY>

Now any IT-School email transaction will be cloned to the new system :D

P.S. Thanks for the experience of hacking Drupal, this hack become much simple: it only cost for 2 working days. Some checking are cloned from Drupal driectly, e.g. email address validation, pipe console output as plain text, and even the skill for patching source. Well, skills will always become weapon, and become useful whenever you need for it :D


Timezone identifier error for phpMyAdmin 3.1.1 setup

For new phpMyadmin 3.1.1 with Debian Lenny, you may face the following error message during setup with /setup, as:

Runtime Notice in ./libraries/common.inc.php#272
date_default_timezone_get() [function.date-default-timezone-get]: It is not safe to rely on the system's timezone settings. Please use the date.timezone setting, the TZ environment variable or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'Asia/Hong_Kong' for 'HKT/8.0/no DST' instead

This is a support request that already reported to phpMyAdmin user fourm (#72909). In order to fix this error message you will need to retouch your php.ini setup, and specify a correct date.timezone information for your server.

In case of Debian Lenny, follow these procedure:

  1. vi /etc/php5/apache2/php.ini
  2. Search for date.timezone, it should by default as below:
    ; Defines the default timezone used by the date functions
    ;date.timezone =
  3. Change this line according to your location (read http://www.php.net/manual/en/timezones.php for more information), e.g.:
    ; Defines the default timezone used by the date functions
    date.timezone = "Asia/Hong_Kong"
  4. /etc/init.d/apache2 restart

Restart your phpMyAdmin setup procedure and the error message should now run away :D


Need to install flex-old for compile PHP CVS under Debian sid

I hope to hack pdo_oci so need to compile PHP CVS manually. BTW, official document mention that:

From this point onwards, installation is similar to the installation of one of the official packages with one main difference – you must have bison 1.28 or later and flex 2.5.4 to compile, because the pre-generated scanner and parser files are not included in CVS.

But Debian sid coming with flex 2.5.35 which is not support. So what can I do for it? Well... There is a package called as flex-old, which is 2.5.4a-7... Just run the following command and install it:

apt-get install flex-old


PHP 5.2.7RC1 is now available for public testing

According to news (2008-10-09) from group php.internals, PHP 5.2.7 should have a final release out in the next 2-3 weeks:

The first release candidate of 5.2.7 was just released for testing and can be downloaded here:

http://downloads.php.net/ilia/php-5.2.7RC1.tar.bz2 (md5sum: be7544212fe4a3ede775a3793ecd3967)

The Windows binaries should become available shortly as well, for now you can grab the win32 snapshots from snaps.php.net. This release has been long in coming, so the NEWS file looks quite impressive with an extensive number of bug fixes. I do not believe any of the fixes are bound to introduce any regression and the hope is to have a final release out in the next 2-3 weeks. To make this possible, please test this RC against your code base and report any problems that you encounter.

Ilia Alshanetsky
PHP 5.2 Release Master

Since the PDO PostgreSQL BLOB issues is now get fixed in CVS of both PHP 5.2.x and 5.3.x (read more), it should come with official release soon. Thanks god that as Drupal 7.x is now binding with PDO only, the PostgreSQL support status is now hopefully safe.

P.S. As I am now working with Drupal 7.x + Oracle, pdo_oci may also come with some hidden issue, too. I should speed that up in order to get it done :D


PHP 5.2.6 pdo_pgsql is buggy with BLOB INSERT/UPDATE

Long story short, PHP 5.2.6 pdo_pgsql is buggy with BLOB INSERT/UPDATE. Both NULL and empty string are saved as NULL when pass into PostgreSQL, and so programming logic may break.

It is now fixed in latest PHP CVS, both 5.2.x and 5.3.x. All packages newer than snapshot php5.2-200810130030.tar.gz. is safe from this issue. Please refer to PHP bug report for more information.

As Drupal 7.x is now revamped with DBTNG and using PDO as the only default support database driver, this is a critical issue for its PostgreSQL support status. Patch is submitted and pending for review. Please refer to Drupal bug report for more information.

How to check if my PHP is safe from this bug?

You can follow the bug reproduction guideline and its latest version of code snippet. Check your result and see if all logical comparison of original data and fetched value are return as TRUE.

What can I do with this bug?

If you are a Drupal 7.x core developer which focusing on PostgreSQL support status, please download PHP snapshot newer than php5.2-200810130030.tar.gz manually, and give a hand for the commit of related Drupal bug report. If you need some help for the manual PHP installation, please refer to my HOWTO for more information.

If you are normal Drupal user which will use PostgreSQL in coming future, please be patient and wait for the release of PHP 5.2.7 (it is now 5.2.7RC2). You may also keep your eyes on the update status of the Drupal bug report, and help the promote of using PHP 5.2.7 as minimal support version for Drupal 7.x public release.


Drupal 6.x + PDO_OCI: finally work together

After put down my multiple database backend research for Drupal among past 4 months, now finally get Drupal 6.x and PDO_OCI working together... It is all about my incorrect pdo_oci driver implementation... The progress is now available in here.

What is the bottleneck?

The main bottleneck of pdo_oci driver implementation is all about size of VARCHAR2 support under AL32UTF8. When working with AL32UTF8, as characters may store up to 2 or 3 bytes, the original maximum 4000 characters support of VARCHAR2 will cut off as 1333 (4000/3). If a single table coming with more than 1 column larger than 1333 characters, an annoying ORA-01461 error message MAY occur. This error message will not trigger when using oci8, but always happened with pdo_oci + Drupal 6.x (Siren 1.x)...

Any tries?

I have try some method to escape from this ORA-01461 error, including:

  1. Reduce most text field size mapping as VARCHAR2(1333), which only left text:big map with VARCHAR2(4000).
  2. Replace some fields as BLOB (e.g. fields storing serialized data) for unlimited storage size support, where restrict some other fields as varchar 256 or even smaller.

But above tries both face some critical error, and so need to rollback:

  1. Some fields (e.g. {menu_router}.access_arguments and {menu_router}.access_arguments) can't use VARCHAR2(1333) only, as they are storing serialized data; BTW, if mapping both to VARCHAR2(4000), ORA-01461 will occur.
  2. BLOB field need special handling (i.e drupal_write_record() and db_decode_blob()) which will complicate the I/O task, therefore not suitable for some special fields (e.g. again {menu_router}.access_arguments and {menu_router}.access_arguments). Since {menu_router} will finally store its rendered result in Drupal's cache system (already using BLOB field for storage), a DUPLICATED BLOB I/O handling result as a very fancy and buggy implementation.

Finally...

Well... Back to the main fantastic problem: as Drupal core hack are share among oci8 and pdo_oci version (backend independent), ~60% code and ~80% Oracle related programming logic are shared among oci8 and pdo_oci, where ~80% PDO related programming logic are also shared among pdo_pgsql and pdo_oci, I switch my focus to debug PHP's pdo_oci driver and my Drupal + pdo_oci driver implementation.

So, what is the problem? Christopher Jones on OPAL give me some hints: I have a wrong transaction handling in case of pdo_oci driver implementation. When working with Oracle + BLOB (or even not with BLOB) a transaction is always required, plus a try {} catch (Exception $e) {} in further more operation. Within my old implementation (trimmed version, share and clone between legacy and other PDO driver):

<?php
  $active_db
->beginTransaction();
 
// Execute the statement.
 
$result = $stmt->execute();
  if (
$result === FALSE) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
 
// Some PDO + BLOB stream I/O handling code.
  // On success, commit the transaction.
 
$active_db->commit();
?>

The correct implementation should be:

<?php
  $active_db
->beginTransaction();
  try {
   
// Execute the statement.
   
$result = $stmt->execute();
   
// Some PDO + BLOB stream I/O handling code.
    // On success, commit the transaction.
   
$active_db->commit();
  }
  catch (
Exception $e) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
?>

Conclusion

Even the research progress is functioning with Drupal 6.x, a lot of effort is still required. As Drupal 7.x will completely switch to PDO, where core database drivers (including MySQL and PostgreSQL) implementation are revamped with OOP, I still need to catch up my missing section among the past 4 months.

The patch for next generation is now RTBC, should I wait for its commit and propose my amendment afterward, or keep hard working and propose the amendment before everything shuttle down?


PHP debugging...

Finding way to debug PHP, since pdo_oci keep on crash with Drupal (Siren) installation and operation. Seems to be problem of BLOB insertion and deletion. P.S. same code base are function with oci8 (i.e. PHP and Oracle are both function) and pdo_pgsql (i.e. programming logic for Drupal + PDO is function)...

Remember to compile PHP with --enable-debug and --enable-xdebug options. Still don't know how to play with vim + Xdebug...

PHP + Xdebug:
http://www.php.net/manual/en/install.pecl.static.php
http://www.xdebug.org/
http://2bits.com/articles/using-vim-and-xdebug-dbgp-for-debugging-drupal...

PHP + Oracle:
http://blogs.oracle.com/opal/

PHP bug trace and issue thread:
http://bugs.php.net/bugs-generating-backtrace.php
http://bugs.php.net/search.php?cmd=display&search_for=pdo_oci&x=0&y=0


Still fighting with annoying ORA-01461 + pdo_oci...

As like as case before, Siren + pdo_oci is still buggy with ORA-01461 during installation. Well, since pdo_oci doesn't update since 3 years ago (http://pecl.php.net/package/PDO_OCI), for sure that the case will not have any change...

Anyway, this page give a very detail idea of the problem (http://littledan.itpub.net/post/6400/270539). When checking with script provided in document, I found that some table is now buggy with the problem:

If you make sure that there is only 1 LONG and no VARCHAR > 1333 bytes, OR just 1 VARCHAR > 1333 bytes in the table, you cannot hit this problem. The following query will give you all the tables with such a combination of columns:

SELECT * FROM
(SELECT TABLE_NAME, OWNER, count(*) NUM
FROM DBA_TAB_COLUMNS
WHERE DATA_TYPE='LONG'
OR (( DATA_TYPE='VARCHAR2'
or DATA_TYPE='CHAR'
or DATA_TYPE='NVARCHAR2'
or DATA_TYPE='NCHAR')
AND DATA_LENGTH > 1333)
AND OWNER NOT IN
('SYS','SYSTEM','SH','OLAPSYS','MDSYS','WKSYS','ODM','XDB','WMSYS')
GROUP BY TABLE_NAME, OWNER)
WHERE NUM > 1;

Both tables watchdog, menu_router and node_type need some modification. The solution maybe a bit simple (from other point of view, very complicated): change those text:big field into blob. This will require for both blob checking for INSERT/UPDATE and after SELECT before use.

Update: the above idea is not work, since sometime fields are using for GROUP BY where BLOB don't support for. As an alternative, I am now trying to restrict most text fields as max 1333, where only text:big as 4000. Therefore some data size limitation to normal storage, and also need to carefully specify which ONE field should be text:big. Seem more simple in solution, but need time to verify the impact...

Giving a try right now...


Apache2.2 + PHP5.2 + pgsql/pdo_pgsql from sketch on Debian sid HOWTO

This simple HOWTO will guide you about how to setup Apache2.2 + PHP5.2 + pgsql/pdo_pgsql from sketch. Compile all package from tarball can give you the maximum flexibility of functionality, e.g. you can enable both mysql/mysqli/pgsql/oci8/pdo_mysql/pdo_pgsql/pdo_oci within single installation.

Before start, I will assume you have Debian sid and Postgresql8.3 installed correctly, which will not detail within this document. In case of Debian, Postgresql8.3 installation is just as simple as a single command:

apt-get install postgresql-8.3 postgresql-client-8.3 postgresql-contrib-8.3 postgresql-common

After installation, you should also initialize your user accounts and database. Please refer to my other article for more information.

This HOWTO is highly similar as my other article which target for install Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch on Debian sid. Therefore some duplicated section will directly refer to there, and only mention those different in case for PostgreSQL in here.


Apache2.2 + PHP5.2 + mysql/mysqli/pdo_mysql from sketch on Debian sid HOWTO

This simple HOWTO will guide you about how to setup Apache2.2 + PHP5.2 + mysql/mysqli/pdo_mysql from sketch. Compile all package from tarball can give you the maximum flexibility of functionality, e.g. you can enable both mysql/mysqli/pgsql/oci8/pdo_mysql/pdo_pgsql/pdo_oci within single installation.

Before start, I will assume you have Debian sid and MySQL5 installed correctly, which will not detail within this document. In case of Debian, MySQL5 installation is just as simple as a single command:

apt-get install mysql-server-5.0 mysql-client-5.0 mysql-common

This HOWTO is highly similar as my other article which target for install Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch on Debian sid. Therefore some duplicated section will directly refer to there, and only mention those different in case for MySQL in here.


Syndicate content