Blogs

BarCamp Hong Kong 2008: What Can I contribute for it?

barcamp Hong Kong 2008

A good invitation from Groups.Drupal (Hong Kong), about BarCamp Hong Kong 2008. I will try to share my research founding with Drupal 7.x + Enterprise DB backend, e.g. Oracle, DB2 and MSSQL; some tricky skill for integrate 3rd layout design with Drupal; also hope to gather with Hong Kong Drupaler and discover for our possibility.

Are you living in Hong Kong? Are you using or interesting with Drupal? Do you hope to meet more friends and share your idea? Is time to show up, and I will be there waiting for you. Let's meet on tomorrow :D

Wirtting query in reserved word safe: any possible solution?

Reserved word conflict is an always pain for multiple DB backend supported system: Drupal also face this for many years (e.g. http://drupal.org/node/371, since June 30, 2002). This is the most complicated bottleneck for daily maintenance and introduce any new DB backend to Drupal: whenever existing DB vendor change their standard, we need to follow it; whenever a new DB backend introduce, another set of reserved words are coming in, too.

There is many possible solution, e.g. prevent use of ANY reserved words based on requirement of ALL supported database engines (as like as Moodle's implementation); BTW, as this solution will left issue as ALWAYS OPEN, I am not going to discuss within this review.

Another possibility is the use of escape characters, e.g. `` for MySQL, "" for PostgreSQL/Oracle/DB2 and [] for MSSQL. This approach is widely used for many successful projects, e.g. phpmyadmin, phppgadmin, Oracle/DB2/MSSQL EM. The primary consideration is: How to implement this syntax for Drupal, which is simple and elegant for both end user and DB abstraction designer?

For sure that we will need some update for existing Drupal queries syntax, where the ultimate goal is query will finally present as following example before process by PHP database connection (e.g. SELECT batch FROM {batch} WHERE bid = :bid AND token = :token from line 15 of includes/batch.inc, CVS HEAD, rewrite with TNG DB standard). In case of MySQL, after translation it should looks like:

SELECT `batch` FROM `batch` WHERE `bid` = :bid AND `token` = :token

I will discuss different syntax change approach, based on: a) user experience, b) similar to ANSI coding style, c) backward compatibility, d) backend complexity, e) overall performance and f) workload for syntax conversion. A simple marking will be given for each item (e.g. good/functioning: 2; fine/not bad: 1; poor/not functioning: 0) so we can have some simple idea between different proposals.

Here is a quick rundown of my review:

傻Pooh北海道遊記 : Part 3

P1010183.JPGP1010183.JPGP1010214.JPGP1010214.JPGP1010214.JPGP1010214.JPGP1010300.JPGP1010300.JPGP1010328.JPGP1010328.JPGP1010416.JPGP1010416.JPGP1010418.JPGP1010418.JPGP1010420.JPGP1010420.JPGP1010426.JPGP1010426.JPGP1010465.JPGP1010465.JPGP1010467.JPGP1010467.JPGP1010474.JPGP1010474.JPG

Forward a message but still leave a copy on the server?

I am using Debian etch for production, plus Exim4 with per user ~/Maildir support. Since I hope to forward each user message to their own public email for backup, Procmail FAQ give me a useful example.

First of all, check if your Exim4 have ~/.procmailrc supported, which usually already there. Find the follow code snippet from /etc/exim4/exim4.conf.template:

#####################################################
### router/700_exim4-config_procmail
#####################################################

procmail:
  debug_print = "R: procmail for $local_part@$domain"
  driver = accept
  domains = +local_domains
  check_local_user
  transport = procmail_pipe
  # emulate OR with "if exists"-expansion
  require_files = ${local_part}:\
                  ${if exists{/etc/procmailrc}\
                    {/etc/procmailrc}{${home}/.procmailrc}}:\
                  +/usr/bin/procmail
  no_verify
  no_expn

#####################################################
### end router/700_exim4-config_procmail
#####################################################

Work around together with Debian's procmail example (/usr/share/doc/procmail/examples/), just create a ~/.procmailrc as below:

# Please check if all the paths in PATH are reachable, remove the ones that
# are not.

PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:.
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$MAILDIR/from
LOCKFILE=$HOME/.lockmail

:0c
! someone@example.com

Send an email to this user, check the mail log with tail -f -n 50 /var/log/exim4/mainlog and you will find that message already forward to your target public email address, with a saved local copy :-)

A new day, a new life, a next generation

Yesterday is my last working day for HKU CITE. After 3 annual holidays, I will in a new position on Monday. It is a secondary school and I will work as an IT Officer. Most duty and position are similar as now, but coming with better location, job title, working hour and also salary.

Not much thing can take away when leaving a working place; but skill and experience is something no one can take away from you, too. I was research for new technology under pay, also learn more when working in a larger company; and the most important is, I learn about what is/how to be a good manager, when compare my freelance project management skill with my department head. Thanks for HKU CITE, thanks for the support within the past year.

Still keep on researching Drupal + multiple database within the past month. After revoke most development skill and foot print, official HP for Siren is online, Siren 1.3 (besides Drupal 6.3) is released, pdo_oci transaction problem is solved, and keep on tracing Drupal 7.x development foot print. Up to now, I have already trace official CVS up to 2008-06-01, and hope to speed it up, too. Hopefully will able to propose patch for official soon.

UPDATE: already trace Drupal core development up-to-date! Will start studying crell and chx's patch on tomorrow. Already catch a lot of bugs :-)

Life will never stop; you will never know rather your decision is correct until tomorrow, and never know what will be happened on the next day, too. Today is what and where we can try our best for, so just focus on it :-)

PostgreSQL 8.3 on Debian sid mini-HOWTO

This mini-HOWTO will guide you though the installation of PostgreSQL 8.3 on Debian sid. After install initial packages, you will need to create and configure both user account and database, which Debian installation script will not do for you.

Before start I will assume you have a complete installed and functional Debian sid box on hand. If you have no idea about this, please refer to my other article for more details.

This document is a refine and trim down version of my legacy artical which target for Debian etch. Most procedure are shared so you may also able to apply them in case of Debian etch.

Prevent '*Zone.Identifier*' files generate by WinXP SP3 and Samba3

My Samba is working fine before but some days ago, it keep on generating some '*Zone.Identifier*' files, e.g. DskPerf.zip:Zone.Identifier:$DATA, when copy file from WinXP SP3 to Samba. I can't delete them from both Windows and Linux, and I don't even understand why it come out suddently...

Finally, openSUSE forum give me a workable solution (http://forums.opensuse.org/network-internet/387779-samba-file-transfer-c...). Just add the following line to smb.conf:

vfs objects = streams_xattr

After restart Samba no more '*Zone.Identifier*' files will be generated :-)

Drupal 6.x + PDO_OCI: finally work together

After put down my multiple database backend research for Drupal among past 4 months, now finally get Drupal 6.x and PDO_OCI working together... It is all about my incorrect pdo_oci driver implementation... The progress is now available in here.

What is the bottleneck?

The main bottleneck of pdo_oci driver implementation is all about size of VARCHAR2 support under AL32UTF8. When working with AL32UTF8, as characters may store up to 2 or 3 bytes, the original maximum 4000 characters support of VARCHAR2 will cut off as 1333 (4000/3). If a single table coming with more than 1 column larger than 1333 characters, an annoying ORA-01461 error message MAY occur. This error message will not trigger when using oci8, but always happened with pdo_oci + Drupal 6.x (Siren 1.x)...

Any tries?

I have try some method to escape from this ORA-01461 error, including:

  1. Reduce most text field size mapping as VARCHAR2(1333), which only left text:big map with VARCHAR2(4000).
  2. Replace some fields as BLOB (e.g. fields storing serialized data) for unlimited storage size support, where restrict some other fields as varchar 256 or even smaller.

But above tries both face some critical error, and so need to rollback:

  1. Some fields (e.g. {menu_router}.access_arguments and {menu_router}.access_arguments) can't use VARCHAR2(1333) only, as they are storing serialized data; BTW, if mapping both to VARCHAR2(4000), ORA-01461 will occur.
  2. BLOB field need special handling (i.e drupal_write_record() and db_decode_blob()) which will complicate the I/O task, therefore not suitable for some special fields (e.g. again {menu_router}.access_arguments and {menu_router}.access_arguments). Since {menu_router} will finally store its rendered result in Drupal's cache system (already using BLOB field for storage), a DUPLICATED BLOB I/O handling result as a very fancy and buggy implementation.

Finally...

Well... Back to the main fantastic problem: as Drupal core hack are share among oci8 and pdo_oci version (backend independent), ~60% code and ~80% Oracle related programming logic are shared among oci8 and pdo_oci, where ~80% PDO related programming logic are also shared among pdo_pgsql and pdo_oci, I switch my focus to debug PHP's pdo_oci driver and my Drupal + pdo_oci driver implementation.

So, what is the problem? Christopher Jones on OPAL give me some hints: I have a wrong transaction handling in case of pdo_oci driver implementation. When working with Oracle + BLOB (or even not with BLOB) a transaction is always required, plus a try {} catch (Exception $e) {} in further more operation. Within my old implementation (trimmed version, share and clone between legacy and other PDO driver):

<?php
  $active_db
->beginTransaction();
 
// Execute the statement.
 
$result = $stmt->execute();
  if (
$result === FALSE) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
 
// Some PDO + BLOB stream I/O handling code.
  // On success, commit the transaction.
 
$active_db->commit();
?>

The correct implementation should be:

<?php
  $active_db
->beginTransaction();
  try {
   
// Execute the statement.
   
$result = $stmt->execute();
   
// Some PDO + BLOB stream I/O handling code.
    // On success, commit the transaction.
   
$active_db->commit();
  }
  catch (
Exception $e) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
?>

Conclusion

Even the research progress is functioning with Drupal 6.x, a lot of effort is still required. As Drupal 7.x will completely switch to PDO, where core database drivers (including MySQL and PostgreSQL) implementation are revamped with OOP, I still need to catch up my missing section among the past 4 months.

The patch for next generation is now RTBC, should I wait for its commit and propose my amendment afterward, or keep hard working and propose the amendment before everything shuttle down?

PHP debugging...

Finding way to debug PHP, since pdo_oci keep on crash with Drupal (Siren) installation and operation. Seems to be problem of BLOB insertion and deletion. P.S. same code base are function with oci8 (i.e. PHP and Oracle are both function) and pdo_pgsql (i.e. programming logic for Drupal + PDO is function)...

Remember to compile PHP with --enable-debug and --enable-xdebug options. Still don't know how to play with vim + Xdebug...

PHP + Xdebug:
http://www.php.net/manual/en/install.pecl.static.php
http://www.xdebug.org/
http://2bits.com/articles/using-vim-and-xdebug-dbgp-for-debugging-drupal...

PHP + Oracle:
http://blogs.oracle.com/opal/

PHP bug trace and issue thread:
http://bugs.php.net/bugs-generating-backtrace.php
http://bugs.php.net/search.php?cmd=display&search_for=pdo_oci&x=0&y=0

傻Pooh北海道遊記 : Part 2

P1010029.JPG: 我真係"貌美如花啦!!哇呵呵~~~P1010029.JPG: 我真係"貌美如花啦!!哇呵呵~~~P1010063.JPG: 石獅下的Pooh仔~~好威猛喔^o^, 呵呵~~~~P1010063.JPG: 石獅下的Pooh仔~~好威猛喔^o^, 呵呵~~~~P1010066.JPG: 咦?有座神社喎, 睇下係咪來先.....P1010066.JPG: 咦?有座神社喎, 睇下係咪來先.....P1010073.JPG: 我都來許個願先!(希望我日日都咁靚仔~~~)P1010073.JPG: 我都來許個願先!(希望我日日都咁靚仔~~~)P1010089.JPG: 外面睇都幾靚wo~~~P1010089.JPG: 外面睇都幾靚wo~~~P1010090.JPG: 原來係札幌護國神社啊!大家請進......P1010090.JPG: 原來係札幌護國神社啊!大家請進......P1010110.JPG: 燒魚仔餅啊~~好味好味~~~(好在我個口大過佢!!哈哈~~~)P1010110.JPG: 燒魚仔餅啊~~好味好味~~~(好在我個口大過佢!!哈哈~~~)P1010119.JPG: 可以嚮小樽三角市場食到咁新鮮ge刺生飯,好幸福啊~~P1010119.JPG: 可以嚮小樽三角市場食到咁新鮮ge刺生飯,好幸福啊~~P1010129.JPG: 好野!今日可以去小樽行街街啦~~P1010129.JPG: 好野!今日可以去小樽行街街啦~~P1010135.JPG: 咦~~佢地咪係我失散左唔知幾年ge親人!!(人地識你架咩=.=!!?)P1010135.JPG: 咦~~佢地咪係我失散左唔知幾年ge親人!!(人地識你架咩=.=!!?)P1010137.JPG: 來福全家福大合照先~~~P1010137.JPG: 來福全家福大合照先~~~P1010138.JPG: 我幾時生左咁多BB ge!?(嗯嗯, 都有我咁靚仔)P1010138.JPG: 我幾時生左咁多BB ge!?(嗯嗯, 都有我咁靚仔)

Syndicate content