PHP

Drupal 6.x + PDO_OCI: finally work together

After put down my multiple database backend research for Drupal among past 4 months, now finally get Drupal 6.x and PDO_OCI working together... It is all about my incorrect pdo_oci driver implementation... The progress is now available in here.

What is the bottleneck?

The main bottleneck of pdo_oci driver implementation is all about size of VARCHAR2 support under AL32UTF8. When working with AL32UTF8, as characters may store up to 2 or 3 bytes, the original maximum 4000 characters support of VARCHAR2 will cut off as 1333 (4000/3). If a single table coming with more than 1 column larger than 1333 characters, an annoying ORA-01461 error message MAY occur. This error message will not trigger when using oci8, but always happened with pdo_oci + Drupal 6.x (Siren 1.x)...

Any tries?

I have try some method to escape from this ORA-01461 error, including:

  1. Reduce most text field size mapping as VARCHAR2(1333), which only left text:big map with VARCHAR2(4000).
  2. Replace some fields as BLOB (e.g. fields storing serialized data) for unlimited storage size support, where restrict some other fields as varchar 256 or even smaller.

But above tries both face some critical error, and so need to rollback:

  1. Some fields (e.g. {menu_router}.access_arguments and {menu_router}.access_arguments) can't use VARCHAR2(1333) only, as they are storing serialized data; BTW, if mapping both to VARCHAR2(4000), ORA-01461 will occur.
  2. BLOB field need special handling (i.e drupal_write_record() and db_decode_blob()) which will complicate the I/O task, therefore not suitable for some special fields (e.g. again {menu_router}.access_arguments and {menu_router}.access_arguments). Since {menu_router} will finally store its rendered result in Drupal's cache system (already using BLOB field for storage), a DUPLICATED BLOB I/O handling result as a very fancy and buggy implementation.

Finally...

Well... Back to the main fantastic problem: as Drupal core hack are share among oci8 and pdo_oci version (backend independent), ~60% code and ~80% Oracle related programming logic are shared among oci8 and pdo_oci, where ~80% PDO related programming logic are also shared among pdo_pgsql and pdo_oci, I switch my focus to debug PHP's pdo_oci driver and my Drupal + pdo_oci driver implementation.

So, what is the problem? Christopher Jones on OPAL give me some hints: I have a wrong transaction handling in case of pdo_oci driver implementation. When working with Oracle + BLOB (or even not with BLOB) a transaction is always required, plus a try {} catch (Exception $e) {} in further more operation. Within my old implementation (trimmed version, share and clone between legacy and other PDO driver):

<?php
  $active_db
->beginTransaction();
 
// Execute the statement.
 
$result = $stmt->execute();
  if (
$result === FALSE) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
 
// Some PDO + BLOB stream I/O handling code.
  // On success, commit the transaction.
 
$active_db->commit();
?>

The correct implementation should be:

<?php
  $active_db
->beginTransaction();
  try {
   
// Execute the statement.
   
$result = $stmt->execute();
   
// Some PDO + BLOB stream I/O handling code.
    // On success, commit the transaction.
   
$active_db->commit();
  }
  catch (
Exception $e) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
?>

Conclusion

Even the research progress is functioning with Drupal 6.x, a lot of effort is still required. As Drupal 7.x will completely switch to PDO, where core database drivers (including MySQL and PostgreSQL) implementation are revamped with OOP, I still need to catch up my missing section among the past 4 months.

The patch for next generation is now RTBC, should I wait for its commit and propose my amendment afterward, or keep hard working and propose the amendment before everything shuttle down?

PHP debugging...

Finding way to debug PHP, since pdo_oci keep on crash with Drupal (Siren) installation and operation. Seems to be problem of BLOB insertion and deletion. P.S. same code base are function with oci8 (i.e. PHP and Oracle are both function) and pdo_pgsql (i.e. programming logic for Drupal + PDO is function)...

Remember to compile PHP with --enable-debug and --enable-xdebug options. Still don't know how to play with vim + Xdebug...

PHP + Xdebug:
http://www.php.net/manual/en/install.pecl.static.php
http://www.xdebug.org/
http://2bits.com/articles/using-vim-and-xdebug-dbgp-for-debugging-drupal...

PHP + Oracle:
http://blogs.oracle.com/opal/

PHP bug trace and issue thread:
http://bugs.php.net/bugs-generating-backtrace.php
http://bugs.php.net/search.php?cmd=display&search_for=pdo_oci&x=0&y=0

Still fighting with annoying ORA-01461 + pdo_oci...

As like as case before, Siren + pdo_oci is still buggy with ORA-01461 during installation. Well, since pdo_oci doesn't update since 3 years ago (http://pecl.php.net/package/PDO_OCI), for sure that the case will not have any change...

Anyway, this page give a very detail idea of the problem (http://littledan.itpub.net/post/6400/270539). When checking with script provided in document, I found that some table is now buggy with the problem:

If you make sure that there is only 1 LONG and no VARCHAR > 1333 bytes, OR just 1 VARCHAR > 1333 bytes in the table, you cannot hit this problem. The following query will give you all the tables with such a combination of columns:

SELECT * FROM
(SELECT TABLE_NAME, OWNER, count(*) NUM
FROM DBA_TAB_COLUMNS
WHERE DATA_TYPE='LONG'
OR (( DATA_TYPE='VARCHAR2'
or DATA_TYPE='CHAR'
or DATA_TYPE='NVARCHAR2'
or DATA_TYPE='NCHAR')
AND DATA_LENGTH > 1333)
AND OWNER NOT IN
('SYS','SYSTEM','SH','OLAPSYS','MDSYS','WKSYS','ODM','XDB','WMSYS')
GROUP BY TABLE_NAME, OWNER)
WHERE NUM > 1;

Both tables watchdog, menu_router and node_type need some modification. The solution maybe a bit simple (from other point of view, very complicated): change those text:big field into blob. This will require for both blob checking for INSERT/UPDATE and after SELECT before use.

Update: the above idea is not work, since sometime fields are using for GROUP BY where BLOB don't support for. As an alternative, I am now trying to restrict most text fields as max 1333, where only text:big as 4000. Therefore some data size limitation to normal storage, and also need to carefully specify which ONE field should be text:big. Seem more simple in solution, but need time to verify the impact...

Giving a try right now...

Apache2.2 + PHP5.2 + pgsql/pdo_pgsql from sketch on Debian sid HOWTO

This simple HOWTO will guide you about how to setup Apache2.2 + PHP5.2 + pgsql/pdo_pgsql from sketch. Compile all package from tarball can give you the maximum flexibility of functionality, e.g. you can enable both mysql/mysqli/pgsql/oci8/pdo_mysql/pdo_pgsql/pdo_oci within single installation.

Before start, I will assume you have Debian sid and Postgresql8.3 installed correctly, which will not detail within this document. In case of Debian, Postgresql8.3 installation is just as simple as a single command:

apt-get install postgresql-8.3 postgresql-client-8.3 postgresql-contrib-8.3 postgresql-common

After installation, you should also initialize your user accounts and database. Please refer to my other article for more information.

This HOWTO is highly similar as my other article which target for install Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch on Debian sid. Therefore some duplicated section will directly refer to there, and only mention those different in case for PostgreSQL in here.

Apache2.2 + PHP5.2 + mysql/mysqli/pdo_mysql from sketch on Debian sid HOWTO

This simple HOWTO will guide you about how to setup Apache2.2 + PHP5.2 + mysql/mysqli/pdo_mysql from sketch. Compile all package from tarball can give you the maximum flexibility of functionality, e.g. you can enable both mysql/mysqli/pgsql/oci8/pdo_mysql/pdo_pgsql/pdo_oci within single installation.

Before start, I will assume you have Debian sid and MySQL5 installed correctly, which will not detail within this document. In case of Debian, MySQL5 installation is just as simple as a single command:

apt-get install mysql-server-5.0 mysql-client-5.0 mysql-common

This HOWTO is highly similar as my other article which target for install Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch on Debian sid. Therefore some duplicated section will directly refer to there, and only mention those different in case for MySQL in here.

Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch on Debian sid HOWTO

This simple HOWTO will guide you about how to setup Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch. Compile all package from tarball can give you the maximum flexibility of functionality, e.g. you can enable both mysql/mysqli/pgsql/oci8/pdo_mysql/pdo_pgsql/pdo_oci within single installation.

Before start, I will assume you have Debian sid and Oracle 11gR1 installed correctly, which will not detail within this document. If you really need some help for that, please refer to my other article for more indeed guideline.

pdo_db2 face difficult with LOWER() - SQL0418N

Finally, I give up Drupal 7.x + ibm_db2 development. It is because ibm_db2's BLOB INSERT/UPDATE/SELECT handling are too complicated... So let's skip it and dig to pdo_ibm directly, as like as case of SQLite ;-)

Most likely, pdo_ibm is just a simple clone of pdo_pgsql implementation, with different connection string handling (surly, we need different schema.*.inc and common.*.inc to serve database dependent differences). Up to this point, I am able to implement install.pdo_ibm.inc without any difficulties.

BTW, we still can't escape from some basic DB2 limitation: SQL0418N with SELECT + LOWER(), e.g. we use LOWER() for username compare. To solving this, we may use CAST().

IMHO, I would like to revamp our programming logic, rather than get rid this limitation with some hybrid implementation. This problem may left for other developers: it seems to be a problem of cross database query implementation, but not a duty of PDO driver development ;-(

Here are some reference message for SQL0418N:

An annoying ORA-01461 error from pdo_oci...

When I am working with pdo_oci development of Siren 1.x (Drupal 6.x), I face a critical problem: pdo_oci is not able to complete the Drupal installation procedure. BTW, as oci8 now share around 90% similar programming logic as that of pdo_oci, it works fine...

Since oci8 and pdo_oci drivers are assumed as interchangeable (they share both common.oracle.inc and schema.oracle.inc, so ~60% codes are share), I try to install with oci8 in order to get the DB up, then further more use pdo_oci during running. It looks all fine (and even pass ab test), but face a critical problem when accessing admin/build/modules (here is one of the error message, the others are similar):

user warning: OCIStmtExecute: ORA-01461: can bind a LONG value only for insert into a LONG column (/usr/local/src/php-5.2.5/ext/pdo_oci/oci_statement.c:146) query: INSERT INTO "menu_router" ("path", "load_functions", "to_arg_functions", "access_callback", "access_arguments", "page_callback", "page_arguments", "fit", "number_parts", "tab_parent", "tab_root", "title", "title_callback", "title_arguments", "type", "block_callback", "description", "position", "weight", "file") VALUES (:l5d0bafc60, :l5d0bafc61, :l5d0bafc62, :l5d0bafc63, :l5d0bafc64, :l5d0bafc65, :l5d0bafc66, :l5d0bafc67, :l5d0bafc68, :l5d0bafc69, :l5d0bafc610, :l5d0bafc611, :l5d0bafc612, :l5d0bafc613, :l5d0bafc614, :l5d0bafc615, :l5d0bafc616, :l5d0bafc617, :l5d0bafc618, :l5d0bafc619) in /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/common.oracle.inc on line 194.

When pressing F5, the case become even worse:

Fatal error: Uncaught exception 'PDOException' with message 'There is already an active transaction' in /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc:182 Stack trace: #0 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc(182): PDO->beginTransaction() #1 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc(113): _db_query(Array) #2 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/common.oracle.inc(194): db_query('INSERT INTO [{w...', Array) #3 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/common.inc(3292): db_query_insert('watchdog', Array) #4 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/modules/dblog/dblog.module(130): drupal_write_record('watchdog', Object(stdClass)) #5 [internal function]: dblog_watchdog(Array) #6 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/module.inc(450): call_user_func_array('dblog_watchdog', Array) #7 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/bootstrap in /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc on line 182

Fatal error: Exception thrown without a stack frame in Unknown on line 0

I guess the first ORA-01461 error is all due to pdo_oci mishandling on VARCHAR2(4000). When you search in google, you may found that similar errors were happened once before in JDBC, on 10gR2 and etc... As DB can setup successfully by oci8, there shouldn't be the bug of DB configuration nor programming logic. I will try to report this later ;-(

P.S. the interchangeable of drivers can also apply to MySQL's (mysql, mysqli and pdo_mysql) and PostgreSQL's (pgsql and pdo_pgsql). They are both functioning correctly ;-)

Finally, handle pdo_oci BLOB INSERT/UPDATE successfully...

Finally I give a hack at the end of Siren 1.x (Drupal 6.x) install.pdo_oci.inc, and able to complete the task for BLOB insertion with Oracle + PDO. Don't laugh at this minor research progress: since I just consider to give up the implementation of CLOB, but replace it with the use of BLOB, a successful insertion of BLOB in pdo_oci can ensure we will able to ship Drupal 7.x with PDO supporting, even including Oracle support ;-)

Technical references:
http://netevil.org/uuid/4365876a-cee9-3009-7726-365876a51802
http://www.php.net/manual/en/ref.pdo.php#pdo.lobs

Here are some code snippets, at the tail of install.pdo_oci.inc (I will soon remove them before commit it to Siren's CVS, so just keep in here as reference):

DB2 VARCHAR with 4000 characters logical limitation

Even though DB2 don't explicitly indicate the limitation of VARCHAR as like as that for Oracle (Oracle will only allow for VARCHAR2 with max 4000 characters; above this limitation, please consider to use CLOB with TB-scale storage), it is still logically limited as 4000 characters.

A minor footnote from IBM DB2:

The functions in the SYSFUN schema taking a VARCHAR as an argument will not accept VARCHARs greater than 4 000 bytes long as an argument. However, many of these functions also have an alternative signature accepting a CLOB(1M). For these functions, the user may explicitly cast the greater than 4 000 VARCHAR strings into CLOBs and then recast the result back into VARCHARs of desired length.

This should be a good example why CLOB is something necessary for most database implementation. We are not always as lucky as case of MySQL and PostgreSQL, which we are able to work with TEXT for (nearly) unlimited size. On the other hand, we have CLOB, which usually required for special handling, e.g. database dependent INSERT/UPDATE/DELETE abstraction, decode CLOB as like as BLOB when fetching data, etc.

Since we are now having drupal_write_record() for high level programming abstraction, and (may) have INSERT/UPDATE/DELETE API for driver level abstraction, CLOB should be something much simpler than case of before. Moreover, this logic is now proved as function with Siren: Siren is able to work with Oracle CLOB, on the other hand don't introduce complicated abstraction to MySQL and PostgreSQL.

Syndicate content