Oracle

Drupal 6.x + PDO_OCI: finally work together

After put down my multiple database backend research for Drupal among past 4 months, now finally get Drupal 6.x and PDO_OCI working together... It is all about my incorrect pdo_oci driver implementation... The progress is now available in here.

What is the bottleneck?

The main bottleneck of pdo_oci driver implementation is all about size of VARCHAR2 support under AL32UTF8. When working with AL32UTF8, as characters may store up to 2 or 3 bytes, the original maximum 4000 characters support of VARCHAR2 will cut off as 1333 (4000/3). If a single table coming with more than 1 column larger than 1333 characters, an annoying ORA-01461 error message MAY occur. This error message will not trigger when using oci8, but always happened with pdo_oci + Drupal 6.x (Siren 1.x)...

Any tries?

I have try some method to escape from this ORA-01461 error, including:

  1. Reduce most text field size mapping as VARCHAR2(1333), which only left text:big map with VARCHAR2(4000).
  2. Replace some fields as BLOB (e.g. fields storing serialized data) for unlimited storage size support, where restrict some other fields as varchar 256 or even smaller.

But above tries both face some critical error, and so need to rollback:

  1. Some fields (e.g. {menu_router}.access_arguments and {menu_router}.access_arguments) can't use VARCHAR2(1333) only, as they are storing serialized data; BTW, if mapping both to VARCHAR2(4000), ORA-01461 will occur.
  2. BLOB field need special handling (i.e drupal_write_record() and db_decode_blob()) which will complicate the I/O task, therefore not suitable for some special fields (e.g. again {menu_router}.access_arguments and {menu_router}.access_arguments). Since {menu_router} will finally store its rendered result in Drupal's cache system (already using BLOB field for storage), a DUPLICATED BLOB I/O handling result as a very fancy and buggy implementation.

Finally...

Well... Back to the main fantastic problem: as Drupal core hack are share among oci8 and pdo_oci version (backend independent), ~60% code and ~80% Oracle related programming logic are shared among oci8 and pdo_oci, where ~80% PDO related programming logic are also shared among pdo_pgsql and pdo_oci, I switch my focus to debug PHP's pdo_oci driver and my Drupal + pdo_oci driver implementation.

So, what is the problem? Christopher Jones on OPAL give me some hints: I have a wrong transaction handling in case of pdo_oci driver implementation. When working with Oracle + BLOB (or even not with BLOB) a transaction is always required, plus a try {} catch (Exception $e) {} in further more operation. Within my old implementation (trimmed version, share and clone between legacy and other PDO driver):

<?php
  $active_db
->beginTransaction();
 
// Execute the statement.
 
$result = $stmt->execute();
  if (
$result === FALSE) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
 
// Some PDO + BLOB stream I/O handling code.
  // On success, commit the transaction.
 
$active_db->commit();
?>

The correct implementation should be:

<?php
  $active_db
->beginTransaction();
  try {
   
// Execute the statement.
   
$result = $stmt->execute();
   
// Some PDO + BLOB stream I/O handling code.
    // On success, commit the transaction.
   
$active_db->commit();
  }
  catch (
Exception $e) {
   
// Some error handling code.
   
$active_db->rollBack();
    return
FALSE;
  }
?>

Conclusion

Even the research progress is functioning with Drupal 6.x, a lot of effort is still required. As Drupal 7.x will completely switch to PDO, where core database drivers (including MySQL and PostgreSQL) implementation are revamped with OOP, I still need to catch up my missing section among the past 4 months.

The patch for next generation is now RTBC, should I wait for its commit and propose my amendment afterward, or keep hard working and propose the amendment before everything shuttle down?

Still fighting with annoying ORA-01461 + pdo_oci...

As like as case before, Siren + pdo_oci is still buggy with ORA-01461 during installation. Well, since pdo_oci doesn't update since 3 years ago (http://pecl.php.net/package/PDO_OCI), for sure that the case will not have any change...

Anyway, this page give a very detail idea of the problem (http://littledan.itpub.net/post/6400/270539). When checking with script provided in document, I found that some table is now buggy with the problem:

If you make sure that there is only 1 LONG and no VARCHAR > 1333 bytes, OR just 1 VARCHAR > 1333 bytes in the table, you cannot hit this problem. The following query will give you all the tables with such a combination of columns:

SELECT * FROM
(SELECT TABLE_NAME, OWNER, count(*) NUM
FROM DBA_TAB_COLUMNS
WHERE DATA_TYPE='LONG'
OR (( DATA_TYPE='VARCHAR2'
or DATA_TYPE='CHAR'
or DATA_TYPE='NVARCHAR2'
or DATA_TYPE='NCHAR')
AND DATA_LENGTH > 1333)
AND OWNER NOT IN
('SYS','SYSTEM','SH','OLAPSYS','MDSYS','WKSYS','ODM','XDB','WMSYS')
GROUP BY TABLE_NAME, OWNER)
WHERE NUM > 1;

Both tables watchdog, menu_router and node_type need some modification. The solution maybe a bit simple (from other point of view, very complicated): change those text:big field into blob. This will require for both blob checking for INSERT/UPDATE and after SELECT before use.

Update: the above idea is not work, since sometime fields are using for GROUP BY where BLOB don't support for. As an alternative, I am now trying to restrict most text fields as max 1333, where only text:big as 4000. Therefore some data size limitation to normal storage, and also need to carefully specify which ONE field should be text:big. Seem more simple in solution, but need time to verify the impact...

Giving a try right now...

100% CPU Usage Overhead running EM DBConsole 11g on Debian sid

My Oracle 11gR1 gone crazy with 100% CPU loading when working with Debian sid. This page provide a functional solution:

Shutdown OEM, login as SYSMAN user and restart the provisioning daemon by executing the two packaged procedures

SYSMAN> execute MGMT_PAF_UTL.STOP_DAEMON

PL/SQL procedure successfully completed.

SYSMAN> execute MGMT_PAF_UTL.START_DAEMON

PL/SQL procedure successfully completed.

Start OEM again and the problem is gone.

WARNING: You must take this on your own responsibility!

So here is my detail procedures. For shutdown OEM (login as user oracle where Oracle 11gR1 installed in /u01/app/oracle/product/11.1.0/db_1/bin):

su - oracle
cd /u01/app/oracle/product/11.1.0/db_1/bin
./emctl stop dbconsole

Login as SYSMAN:

./sqlplus SYSMAN

Then execute the following PL/SQL commands:

SQL> execute MGMT_PAF_UTL.STOP_DAEMON
SQL> execute MGMT_PAF_UTL.START_DAEMON

Restart OEM:

./emctl start dbconsole

Check the CPU loading and it should be fine now :-)

Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch on Debian sid HOWTO

This simple HOWTO will guide you about how to setup Apache2.2 + PHP5.2 + OCI8/PDO_OCI from sketch. Compile all package from tarball can give you the maximum flexibility of functionality, e.g. you can enable both mysql/mysqli/pgsql/oci8/pdo_mysql/pdo_pgsql/pdo_oci within single installation.

Before start, I will assume you have Debian sid and Oracle 11gR1 installed correctly, which will not detail within this document. If you really need some help for that, please refer to my other article for more indeed guideline.

Oracle Database 11g Release 1 on Debian sid HOWTO

This is a simple HOWTO, for installing Oracle Database 11g Release 1 on Debian sid.

But why we need this HOWTO? Since Oracle only officially support Red Hat Enterprise Linux, SUSE Enterprise Linux and other enterprise level Linux distribution; BTW, there shouldn't be any problem if you hope to use with other distribution, which just need some special configuration and install required dependent packages. On the other hand, some dependent packages are obsoleted in Debian etch, but only available in oldstable or sid, so install with sid should be better than etch + hybrid sid packages.

"Hey man! Please stop for a while! I have no idea about how to working with Debian sid!"... Ooops... If this is your case, my other article should be your cup of tea. Go back now and let's enjoy with that for a bit while :-)

Before start you will need to prepare your Debian with:

  • A complete Debian sid installation with X11.
  • Static public IP address (or you may bind to localhost during installation).
  • Functional hostname, e.g. hardcode it within /etc/hostname and /etc/hosts.

An annoying ORA-01461 error from pdo_oci...

When I am working with pdo_oci development of Siren 1.x (Drupal 6.x), I face a critical problem: pdo_oci is not able to complete the Drupal installation procedure. BTW, as oci8 now share around 90% similar programming logic as that of pdo_oci, it works fine...

Since oci8 and pdo_oci drivers are assumed as interchangeable (they share both common.oracle.inc and schema.oracle.inc, so ~60% codes are share), I try to install with oci8 in order to get the DB up, then further more use pdo_oci during running. It looks all fine (and even pass ab test), but face a critical problem when accessing admin/build/modules (here is one of the error message, the others are similar):

user warning: OCIStmtExecute: ORA-01461: can bind a LONG value only for insert into a LONG column (/usr/local/src/php-5.2.5/ext/pdo_oci/oci_statement.c:146) query: INSERT INTO "menu_router" ("path", "load_functions", "to_arg_functions", "access_callback", "access_arguments", "page_callback", "page_arguments", "fit", "number_parts", "tab_parent", "tab_root", "title", "title_callback", "title_arguments", "type", "block_callback", "description", "position", "weight", "file") VALUES (:l5d0bafc60, :l5d0bafc61, :l5d0bafc62, :l5d0bafc63, :l5d0bafc64, :l5d0bafc65, :l5d0bafc66, :l5d0bafc67, :l5d0bafc68, :l5d0bafc69, :l5d0bafc610, :l5d0bafc611, :l5d0bafc612, :l5d0bafc613, :l5d0bafc614, :l5d0bafc615, :l5d0bafc616, :l5d0bafc617, :l5d0bafc618, :l5d0bafc619) in /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/common.oracle.inc on line 194.

When pressing F5, the case become even worse:

Fatal error: Uncaught exception 'PDOException' with message 'There is already an active transaction' in /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc:182 Stack trace: #0 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc(182): PDO->beginTransaction() #1 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc(113): _db_query(Array) #2 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/common.oracle.inc(194): db_query('INSERT INTO [{w...', Array) #3 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/common.inc(3292): db_query_insert('watchdog', Array) #4 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/modules/dblog/dblog.module(130): drupal_write_record('watchdog', Object(stdClass)) #5 [internal function]: dblog_watchdog(Array) #6 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/module.inc(450): call_user_func_array('dblog_watchdog', Array) #7 /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/bootstrap in /mnt/hswong3i/project/drupal/drupal-6.x-siren-1.x/includes/database.pdo_oci.inc on line 182

Fatal error: Exception thrown without a stack frame in Unknown on line 0

I guess the first ORA-01461 error is all due to pdo_oci mishandling on VARCHAR2(4000). When you search in google, you may found that similar errors were happened once before in JDBC, on 10gR2 and etc... As DB can setup successfully by oci8, there shouldn't be the bug of DB configuration nor programming logic. I will try to report this later ;-(

P.S. the interchangeable of drivers can also apply to MySQL's (mysql, mysqli and pdo_mysql) and PostgreSQL's (pgsql and pdo_pgsql). They are both functioning correctly ;-)

Finally, handle pdo_oci BLOB INSERT/UPDATE successfully...

Finally I give a hack at the end of Siren 1.x (Drupal 6.x) install.pdo_oci.inc, and able to complete the task for BLOB insertion with Oracle + PDO. Don't laugh at this minor research progress: since I just consider to give up the implementation of CLOB, but replace it with the use of BLOB, a successful insertion of BLOB in pdo_oci can ensure we will able to ship Drupal 7.x with PDO supporting, even including Oracle support ;-)

Technical references:
http://netevil.org/uuid/4365876a-cee9-3009-7726-365876a51802
http://www.php.net/manual/en/ref.pdo.php#pdo.lobs

Here are some code snippets, at the tail of install.pdo_oci.inc (I will soon remove them before commit it to Siren's CVS, so just keep in here as reference):

Tips and scripts for checking PHP5 OCI8/PDO_OCI + Oracle 11gR1 installation

In case of Oracle 11gR1, database name is no longer such simple as SID, but need to change into something call "Global Database Name" in "name.domain" format. So the testing script for OCI8 and PDO_OCI need to have some change, too.

Let's say we have a host call "localhost.localdoman" with a database SID as "AL32UTF8", its global database name should be "AL32UTF8.localdomain". In case of checking OCI8:

<?php
  $db_conn
= ocilogon("HR", "CHANGE", "//localhost.localdomain/AL32UTF8.localdomain");
 
$cmdstr = 'SELECT "LAST_NAME", "SALARY" FROM "HR"."EMPLOYEES"';
 
$parsed = ociparse($db_conn, $cmdstr);
 
ociexecute($parsed);
 
$nrows = ocifetchstatement($parsed, $results);
  echo
"<html><head><title>Oracle PHP Test</title></head><body>";
  echo
"<center><h2>Oracle PHP Test</h2><br>";
  echo
"<table border=1 cellspacing='0' width='50%'>\n<tr>\n";
  echo
"<td><b>Name</b></td>\n<td><b>Salary</b></td>\n</tr>\n";
  for (
$i = 0; $i < $nrows; $i++ ) {
    echo
"<tr>\n";
    echo
"<td>" . $results["LAST_NAME"][$i] . "</td>";
    echo
"<td>$ " . number_format($results["SALARY"][$i], 2). "</td>";
    echo
"</tr>\n";
  }
  echo
"<tr><td colspan='2'> Number of Rows: $nrows</td></tr></table><br>";
  echo
"<em>If you see data, then it works!</em><br>";
  echo
"</center></body></html>\n";
?>

In case of checking PDO_OCI:
<?php
  $db_conn
= new PDO('oci:dbname=//localhost.localdomain/AL32UTF8.localdomain', 'HR', 'CHANGE');
 
$cmdstr = 'SELECT "LAST_NAME", "SALARY" FROM "HR"."EMPLOYEES"';
 
$stmt = $db_conn->query($cmdstr);
 
$results = $stmt->fetchAll();
 
$stmt = $db_conn->query("SELECT COUNT(*) AS nrows FROM ($cmdstr)");
 
$nrows = $stmt->fetch(PDO::FETCH_OBJ)->NROWS;
  echo
"<html><head><title>Oracle PHP Test</title></head><body>";
  echo
"<center><h2>Oracle PHP Test</h2><br>";
  echo
"<table border=1 cellspacing='0' width='50%'>\n<tr>\n";
  echo
"<td><b>Name</b></td>\n<td><b>Salary</b></td>\n</tr>\n";
  for (
$i = 0; $i < $nrows; $i++ ) {
    echo
"<tr>\n";
    echo
"<td>" . $results[$i]["LAST_NAME"] . "</td>";
    echo
"<td>$ " . number_format($results[$i]["SALARY"], 2). "</td>";
    echo
"</tr>\n";
  }
  echo
"<tr><td colspan='2'> Number of Rows: $nrows</td></tr></table><br>";
  echo
"<em>If you see data, then it works!</em><br>";
  echo
"</center></body></html>\n";
?>

Syndicate content