Classically, web services are created using one of two methods, typically referred to as bottom-up or top-down. Using the bottom-up approach, the developer writes java classes (usually in the form of EJBs, or Enterprise JavaBeans) that perform certain business functions then abstracts the design up into a Web Services Description Language (WSDL) document, which is a high-level XML document that is used to describe the web service. It is this document that might be used in a Universal Description, Discovery, and Integration (UDDI) registry, which is a directory used to advertise the availability of web services.

On the other hand, the top-down approach begins with a WSDL document that is created by a business analyst or architect. From the WSDL and its associated XML schemas the developer creates java classes that implement the schema objects (usually in the form of Apache XMLBeans or Java Architecture for XML Binding (JAXB) classes) and then writes web service methods on top of those classes that implement the business operation. While there is hot debate over which design method is best, suffice to say that there are very good reasons to use one method over another. For example, if the system running the web service does not already have its business objects and logic represented by EJBs, then the top-down approach is simpler and quicker to implement.

Modifying OracleAS Libraries

In OracleAS 10g, a web service is installed as an Oracle Container for J2EE (OC4J). Because each OC4J is a discretely running program within the application server, it is possible to use JAX-WS on a single OC4J without affecting the integrity of other web services running on the system.

First, the JAX-WS java library files must be installed as shared libraries. In addition, if the JAXB java classes will be used to convert XML schemas described in the WSDL to java classes then an XML transformation library will be necessary as well; for example, Saxon. Implementing these as shared libraries means that the same files may be used by multiple web services in multiple OC4J instances without reloading the libraries into server memory more than once.

Second, the new OC4J must have several Oracle libraries de-registered so they do not conflict with the JAX-WS software. The easiest way to do this is to deploy the web service with a deployment plan. Whatever method is used, the following default Oracle libraries must be de-registered: oracle.jwsdl, oracle.ws.client, oracle.ws.jaxrpc, and oracle.xml.

Summary

Although the removal of default Oracle libraries will make unavailable a few of the Oracle web service enhancements, the transition from JAX-RPC to JAX-WS web service technologies may be advantageous in certain situations. In the future, expect to see Oracle combine the standard java application server features of its WebLogic Suite, which already supports JAX-WS, with the enhanced administration features of the Oracle Application Server software.

In late 2008 I had the opportunity to meet Dennis Brandl, member of the ISA-SP88 committee and World Batch Forum (WBF) Working Group. The context of the meeting was to help a global pharmaceutical manufacturing company use B2MML to communicate with auxiliary systems, including the company's LIMS. While the application of how B2MML might be used to express and communicate ERP data and decisions was clear, the task of teaching the LIMS system how to consume this data and return communication using B2MML was less clear. The company had already settled on using web services whenever possible, so my job was to work out the specific technologies that might be used to accomplish the task.

Challenge

After reviewing the B2MML schemas, it became clear that the level of possible detail would be overwhelming. Further, while the schemas presented data using rational and coherent relationships, the overall data model looked nothing like LIMS data. For example, in a LIMS system, lots and samples are frequently created using information pulled from different parts of the system: a sample may require an attribute stored on the material, a lot may require an attribute stored on the lot plan, etc. When modeled into B2MML, the data required to log a LIMS lot might be spread out over several schemas, which made the creation and update of lots complex. To make the matter even more difficult, the LIMS needed to communicate back to the ERP using B2MML, which meant that the mapping of ERP data to LIMS data had to be bi-directionally accurate.

Solution

To solve this dilemma, I chose to write an XML schema that described the SQL*LIMS lot and included the relationships to essential LIMS information such as lot dispositions, formulations, specifications, etc. With the LIMS data model fully described in an XML schema, I chose to use Extensible Stylesheet Language Transformations (XSLT) to convert B2MML XML data into SQL*LIMS XML data. Therefore, when a web service received B2MML data from the ERP, an XSLT Stylesheet was used to transform the data into the LIMS' schema and it was written to a table in the Oracle XML Database, which was added to the database running the LIMS system.

This solution allowed all custom LIMS code to be read and written to a single data specification. In other words, instead of programming the LIMS to look for "ERP Datum A" and place it into "LIMS Datum C", all code was simply written to read and write data to known locations. If the ERP system chose to move "ERP Datum A" to "ERP Datum D", no LIMS API code needed to be modified. Instead, the XSLT Stylesheet was simply updated and placed on the LIMS web services host. Since changes to an XSLT would not be considered a "code change", only brief acceptance testing would be conducted to verify that the flow of data was not interrupted between systems. Additionally, the ERP system could start sending new data at any time and the LIMS project could work on its own timeline without being concerned that the change in data would adversely affect the overall system.

Conclusion

I am not aware of any LIMS system that has integrated B2MML communication. However, with a little help from web services and an XML schema mapping program, I believe any LIMS system could take advantage of this rich resource.

Although the MRP portion of the interface was deployed to a few sites in the United States and Europe for both SQL*LIMS v4.1 and v5, it soon became clear that the highly customized nature of the interface was slow and too difficult to support. The entire system was single-threaded, which meant that all transactions – both inbound and outbound – had to be processed one at a time. And, data was being transferred using an obtuse and hard-coded format; it was impossible to introduce new data into the system without causing a significant amount of technical rework and software re-release.

Fresh Start
Soon after having completed the initial deployment of the first global LIMS instance in 2005, team members began hearing rumors that the company now wished to replace their aging MRP system. Although there were only two software packages that were finally considered for the new system, the company choose Oracle E-Business Suite (EBS). EBS is an incredibly robust system that would replace not only their original MRP system, but would implement new ERP software for financials, procurement, and several other aspects of running a global manufacturing company.

Therefore, the challenge given to me was how to interface SQL*LIMS v5 to an Oracle EBS? What data formats should be considered? What software should be used?

Considerations
Even during the interface planning stages, I was given several options for implementing the interface. First, the new ERP could connect directly to our existing interface software, which meant that data would come in and out of the LIMS exactly as it had been planned back in 2002. Second, we could use a new data format and have the data placed either on the LIMS file system via FTP or into the LIMS database in a temporary table. Or third, we could choose to re-implement the interface using web services.

My very strong recommendation to this company was to use web services. Using this option meant that the LIMS system could take advantage of XML-based data exchange. This was very important to me because it meant that data being transferred between systems would be self-describing. No longer would changes in data definition break the interface or cause the re-release of interfacing components. Also, it meant that the interface could take advantage of the multi-threaded nature of a web server; transactions not requiring first-in, first-out processing could be executed in parallel, which would significantly increase the speed of the entire system.

New Direction
SQL*LIMS v5 does not support web services, so whatever architecture was chosen had to extend the SQL*LIMS architecture in a manner that was consistent with the original application and would cause the least number of customizations to the vendor system. While many possible solutions were investigated, the following major components were chosen:

• Web services would be written to send and receive data. Each data transaction would have its own web service endpoint and would be packaged separately so that future enhancements would be easy to implement. This allowed the solution to leverage the SQL*LIMS application tier by running the web services from the Oracle Application Server, or from any production quality application server such as Oracle Weblogic, Sun Glassfish, etc.
• Data would be transferred between the LIMS and ERP system using the Business To Manufacturing Markup Language (B2MML). These XML schemas are explicitly designed to implement the ANSI/ISA 95 family of standards, which help define the data that must move between ERP and manufacturing systems. By using open-standard XML schemas, the company was able to leverage the knowledge and experience of other manufacturing organizations and professionals all over the world.
• An interface monitor would be added to run as a Windows service on the SQL*LIMS Business Logic Server (BLS), which is where the core of the SQL*LIMS monitors execute. The new monitor would process transaction data waiting in the queue table in the LIMS database, create lots, update lots, and send system notifications as required. Although adding a monitor introduced new custom code, it also allowed the solution to use existing architectural knowledge and training as a foundation for implementation.
• A single Oracle form was added to the SQL*LIMS Utilities menu that allowed users to review inbound and outbound transactions and review transactions that had triggered errors. The form allowed the user to identify which transactions were legitimate errors, and which could be re-processed successfully.

Summary
I am often asked to make LIMS customizations. The first question I ask is, "Why?" Customizations can mean that a vendor product becomes unsupportable in the present and may remove it from the vendor upgrade path in the future. However, with the correct experience and knowledge, changes to a vendor product can be made to augment, rather than modify, existing functionality. These are the best customizations; these are the customizations that show pay-off for the time I spent and for the customer who must now support the system for the next several years. The keys are to use industry-standard technologies and methodologies and to leverage existing architecture, experience, and training.

In this article, I will describe a solution that can be used to automatically generate completed APHIS 2008 PDF forms directly from the SQL*LIMS v5 software.

Standard SQL*LIMS Reporting
My previous article, “XML Reporting in SQL*LIMS”, explains how reports work in the core SQL*LIMS product. In brief, a report is requested from the SQL*LIMS client, sent to the Oracle Reports server in the Oracle AS software, and the Oracle Report output is sent to the appropriate destination, the screen, a printer, a file, etc.

Design Decisions
During design, several issues needed to be address by this solution:
1. How to extract report data from the database
2. How to get the data into the federally-provided APHIS 2008 PDF form.
3. What to do with the completed form.

Selecting Data
Like the previous XML reporting solution, all data was selected using a PL/SQL package. As before, this offered a greater range of algorithmic possibilities than provided in Oracle Reports, and placed all logic into a single location that was more easily maintained than an Oracle Report.

Using the data provided by the PL/SQL data extract package, this solution required that the Oracle Report output data in the Adobe XFDF format. Similar to XML, this output format was easily reproducible in Oracle Reports, and allowed for the solution to remain consistent with the standard report execution method provided in the core SQL*LIMS software, the Run Events form, or automatic execution via status action.

Placing XFDF data on the PDF
Because data formatted to the Adobe XFDF specification can be manually imported into an Adobe PDF form, the next issue to be addressed was how to cause this data importation to be automatically triggered. Fortunately, the iText java library provided the perfect solution for programmatically inserting XFDF data into an existing PDF form.

The iText library was used to create and implement a new Pluggable Destination in the Oracle AS server. This allowed the server to receive the XFDF data created by the Oracle Reports client, import the data into the APHIS PDF file using the iText java library, and save the completed PDF file to a user-selected folder on the server. Each file name is made unique by lot and specification name, as well as site location, and a report creation timestamp.

Pluggable Destinations in SQL*LIMS
As before, a simple tweak to the EVENT_DISTRIBUTION_CLASS system parameter in SQL*LIMS allowed for events to be configured to the “PdfForm” destination. This modification to the software only allows the event to be executed in server mode, but this is actually a limitation intrinsic to the Oracle Forms software, which does not yet support Oracle Reports Pluggable Destinations in its internal APIs.

Summary
As with any good technical solution, all of the technology in this solution is transparent to the SQL*LIMS user: The Veterinary Biologics event is executed using the Run Events form and a PDF file containing the requested LIMS data appears in a shared folder within seconds. And, while the solution was implemented specifically for the APHIS 2008 Veterinary Biologics report, the Pluggable Destination that was created to augment the Oracle AS server was created to be generic enough to work with future reports that require PDF form templates.

Just because a person wishes to build a better mousetrap does not mean that they dislike mousetraps! I like mousetraps, personally. We had a really large Norwegian rat that decided to reside with us when we lived outside of Detroit, Michigan. (We’re clean people, really!) But for whatever reason, this rat just decided that our house was cozy, pleasant, and friendly - whatever it is that rats go for when choosing a domicile - and he took up residence. In fact, one night he chewed a large hole in the wall not more than four feet from the desk chair where I sat every day doing LIMS consulting. So, as you can imagine, what I needed was a really good, rather large, mousetrap.

Buying a mousetrap is a pretty simple affair. I drove to Home Depot and in less than five minutes picked out the best looking trap I could find. I can tell you what never crossed my mind: I never once considered building my own mousetrap, not once. Why? Because my goals were very straight forward, I wanted one dead rat. No mess, no frills, no desire to wine and dine my rat before death, just one dead rat.

Somewhere out there are people who dream of better mousetraps. You know, the do-it-yourself engineer type of person who really enjoys thinking up new, cleaner, more humane, better mousetraps. I have nothing against these people, but their desires were not mine: they wish to improve the design; I wished to use the design. More power to them! Yes, my rat was dead within a week, but maybe the next generation of mousetraps will be better. Maybe I’ll just push a button and every rodent within a hundred yards will simply disappear into vapor…I don’t know…whatever that new-and-improved mousetraps will do.

Let me make my point: Dreaming of a better LIMS is my job…it’s what I do, and in some twisted way, I find it rewarding. Don’t hold it against me! I love the products that are on the market now, I look forward to new versions, improved interfaces, faster performance, standards based technology, etc. Please don’t think that because I pine for a better product, process, or whatever, that I dislike what’s out there. As I said above, it’s really not in my best interest to upset the apple cart, challenge the status quo, complain, disturb the balance, make a stink, stir things up…

…but a man’s gotta dream!

Normally, SQL*LIMS implements all reports using Oracle Reports Developer for both data selection and report layout. This solution is consistent with the core technology, but has several disadvantages. Foremost, Oracle Reports Developer is a confusing and unstable application. Any developer that has spent significant time creating Oracle reports can tell you the application crashes several times a day. Sometimes it crashes when copying and pasting layout objects, sometimes it crashes for no reason at all. Whatever the case, it’s frustrating to loose work, and seasoned developers learn to save their work obsessively. Also, Oracle Reports allows the developer to place programmatic triggers in many places throughout the application. Starting with the report properties form before and after triggers, programmatic logic can be executed in the data model, before and after triggers on the report body, and on every object in the report layout. The result is a confusing amalgam of logic that is spread out across dozens and dozens of program units, and executes in a very complex order, sometimes repeatedly. For all of its power, Oracle Reports Developer turns a simple reporting task into an advanced task, and an advanced reporting task into rocket science. For a complex report like a Certificate of Analysis, this solution becomes too cumbersome to maintain, not to mention the CPU cost of causing most of that processing to occur not in the Oracle Database server, but in the user’s client.

The following solution architecture was not my first choice. Oracle has a product called Oracle XML Publisher that I believe would produce the same result as the customization described in this article. However, when we received requirements for a flexible output reporting solution from a customer recently, it was clear that requiring the purchase and implementation of Oracle XML Publisher would not be an option, since the decisions regarding architecture had already been made based on the software required by SQL*LIMS version 5.0.1. This architecture consists of Oracle 10g Database and Oracle 10g Application Server, which includes the Oracle Forms server to implement the SQL*LIMS software forms, and Oracle Reports server to implement SQL*LIMS reports.

Design Decisions

During design several questions needed to be answered:

1) How to extract data according to complex business rules in a manner that was quick to execute, yet simple to understand during software maintenance?
2) How to get the selected data into an XML format?
3) How to change XML data into a user-friendly report layout?
4) How to change a user-friendly report layout into a format that could not be modified by the end user?

Selecting Data

We need to remove data selection out of Oracle Reports. The logical choice given the architecture is to place all data selection criteria into a database package. This is advantageous because it makes the implementation of business rules for selected data more transparent. Think of data selection in Oracle Reports like a system of tree roots; although it starts in one place in the data model, it quickly branches out into a complex system of tiny logical branches as the report executes group, frame, and item formatting triggers. Placing data selection in a database package forces the data selection algorithm into a single, straight line. Also, the package executes in the database, meaning that the user’s client is relieved of this computational task.

Another of the big advantages of using a database package is that the data selection algorithm can be modified without accidentally affecting report layout. In other words, the maintenance of business rules as implemented by report data selection is isolated from the rest of the solution, and may be modified by any developer with PL/SQL knowledge.

Formatting Data to XML

In order to format the data as XML, Oracle Reports Developer is used to create PL/SQL selection blocks in the data model. Each query returns a reference cursor from a function in the database package, and each field returned by the reference cursor is assigned a proper XML tag according to the desired reporting XML schema. The report does not require any layout, since when Oracle Reports executes to an XML format destination only XML data is output according to the order in which queries appear in the data model.

Placing XML formatting in an Oracle Report also allows this custom solution to stay within the SQL*LIMS v5 architecture, since Oracle Reports is already used for reports. So, the report is executed in SQL*LIMS either interactively using the Run Events form, or automatically using a server status action, and SQL*LIMS executes the Oracle Report, which uses the database package to select data, and outputs the data into XML.

Making XML User-Friendly

The next decision is how to make XML user-friendly. Fortunately, the World Wide Web Consortium, or the W3C, has created XSL, which stands for Extensible Stylesheet Language. In simple terms XSL is a family of standards that define the semantics for working with, presenting, and manipulating XML data. One member of this family is XSL-FO, which allows XML data to be converted to a paginated, user-friendly format. Because this is a fairly complex transformation, it is necessary to use a helper application such as Altova StyleVision. This application takes XML data and allows it to be placed on the page in a user-friendly manner. The following is a screenshot from the Altova website.

Click image to see full sized

For a developer that is comfortable with using an integrated development environment (IDE), this application turns stacks of XML data into simple fields that may be placed on the page similar to using Oracle Reports Developer, which for the purpose of comparison is shown in the following screenshot.

Click image to see full sized

Converting XML to PDF

The final component in this customization is to convert the XSL-FO transformation into PDF. This is the most technical part of the system, and requires both custom software and a modification to the Oracle Reports server.

While the ability to transform XML into a new markup format, XSL-FO, is a step in the right direction, a FO processor must be used to convert XSL-FO into PDF. For this, we must rely on Apache FOP.

A FOP, which stands for formatting objects processor, takes XSL-FO markup and turns it into a paginated format such as RTF or PDF. While there are several FOPs on the market, most notably Oracle's own XML Publisher, we required a FOP that could be plugged directly into the SQL*LIMS v5 architecture and would not adversely affect previously made financial decisions. Apache FOP is both free, and as described below, could be plugged directly into the Application Server as an Oracle Reports Pluggable Destination.

Oracle 10g Application Server introduced Pluggable Destinations, which allow a developer to create and implement a custom destination in the Oracle Reports server. As a proof of concept, Oracle delivered the standard FTP destination using this new feature. Custom destinations are implemented as java servlets and are executed by Oracle Reports as an integrated part of the Application Server architecture.

It may occur to the astute technician that using an Oracle Reports Pluggable Destination to transform XML to PDF using XSL-FO is not consistent with the architecture. This is accurate: this is a change in format not a change in destination. However, the limitation is that Oracle Reports does not yet, nor will it likely ever, implement "Pluggable Formats" for Oracle Reports. Therefore, the only option is to output the report as XML and convert to PDF during the implementation of the destination.

Creating a Pluggable Destination that uses Apache FOP is fairly straight-forward for a knowledgeable java developer. Both are written in java, and the newest version of Apache FOP is thread safe, an absolute necessity when executing in a servlet environment. The final java jar is placed in the Oracle Reports jlib folder along with all font and metrics files, which enables the FOP to embed all fonts into the PDF document. During transformation, Apache FOP uses an XSL-FO file created by Altova StyleVision and the XML data that Oracle Reports stores in the cache to transform XML to PDF. After transformation, the servlet copies the PDF file out of the cache to a shared folder on the SQL*LIMS application tier. This folder is shared to permissible users as necessary.

Pluggable Destinations in SQL*LIMS

The standard SQL*LIMS version 5 software does not support Oracle Reports Pluggable Destinations. However, a simple tweak to the EVENT_DISTRIBUTION_CLASS system parameter allows events to be configured to the "Transform" destination. This modification to the SQL*LIMS software only allows the event to be executed in server mode, but this is actually a limitation intrinsic to the Oracle Forms software, which does not yet support Oracle Reports Pluggable Destinations in its internal APIs. Conveniently, the custom destination is passed through the SQL*LIMS Event Monitor without incident, and sent directly to the Oracle Reports server for processing.

Summary

All of this technology is transparent to the SQL*LIMS user. The Certificate of Analysis event is executed using standard methods (Run Events form or status action), and the PDF report appears in the user's file share using a filename that is specific to the laboratory, lot, protocol, and date and time. If a change in business requirements is necessary, only the database package is affected. If a change in report layout, page size, or selected columns is necessary, only the StyleVision transform file is affected. In the final implementation, users are allowed to select from one of several report formats, meaning that the same data can be presented in a layout that is sensitive to language, market, region, or specification.

Best of all, the new functionality is based on international standards, with a bare minimum of modifications made to the vendor-provided SQL*LIMS software.

What’s up with a conference starting at 8:00AM with an one hour breakfast? Do these people not have homes with kitchens? Here’s the thing: I’m busy enough that I don’t actually read the schedule when I attend these things, so when I arrive at 7:50AM I expect things to start in 10 minutes. Instead, I got the chance to sit around and observe people, find out who’s who, and generally exercise my social muscle for a full 80 minutes.

People are fascinating and the technical crowd is no different. You have several types of people that will attend a technical conference, but I'd like to pick on only those with which I came into contact. The first person I sat down next to was a "Self-Trained," which is a person that learns everything on his or her own. I’m a self-trained, so this type is one I’m pretty familiar with: learned the technology during some historical task when a one-off solution was chosen by the employer and then became the resident "expert". My experience is that the older the Self-Trained, the more arcane the technology. The reason for this is simple: self-training takes lots of time, so keeping up with the "newest" thing becomes hard to do without outside motivation.

The next person I met was at the breakfast table. I noticed him immediately because he was actually wearing the Oracle baseball cap that every attendee received at the registration desk. He was also wearing – are you sitting down? – a Microsoft monogrammed oxford shirt. Who does this? The "Conference Junkie", that’s who! This type walks through local malls wearing a Cisco hat, carries an IBM duffle bag, and writes with an oversized Seibel ink pen.

The final type I will mention is the "Casual Technologist", which was represented at this conference by the two DBAs sitting at my table. Coming from a very large local government contractor, these guys mostly talked about how they intended to spend the day at one of the Disney World theme parks instead of learning about the newest Oracle technologies. To prove the point, they packed up their belongings and left the conference before the 9AM start time.

The title of this conference was "Compliance to the Core", which immediately piqued my interest. Being a LIMS consultant offers regular opportunities to deal with compliance issues. However, this type of compliance is different from CFR part 11 compliance. To get a handle on this type of compliance, think of the following hypothetical event: An executive from a major energy or financial company sends out false or inaccurate data in order to make the financial position of the company look more solid to shareholders and the financial community. From a culpability point of view, the company itself must be able to show that the timeliness and quality of the internal data available to that executive is not blamed for the release of inaccurate data. Enter "business intelligence."

Although the push is somewhat new to Oracle, the idea of business intelligence is very old: "Summarize for me all the data I have permission to see in a way I can understand and make certain it’s up to date!" Sound easy? Wrong! Typically, I see people reviewing only the data they can get, regardless of whether they should get it, or whether it is the right data, or whether they have the mathematical skill to summarize the data in ways that are accurate or meaningful. So, functional business intelligence is very good news, since it is an attempt by Oracle to place data storage, warehousing, mining, transformation, reporting, security, and dashboard-like summarization all under a unified application interface. This is excellent news for the LIMS industry, since it means that big, corporate businesses are increasingly interested in things that the average pharmaceutical company is interested in, which is, secured data delivered reliably.

I'm looking forward to seeing what affect this new push on business intelligence has on the LIMS industry long term. Is it possible, for example, that this more interactive front end could replace the somewhat inflexible reporting interfaces currently offered in several LIMS programs?

There are certain security aspects that I will skim over in this article: 1) using FTP with secured user access, 2) document storage and access, and 3) access to the interface itself. Each of these aspects is important to overall system security, but I will not take the time to demonstrate secure solutions in this article.

To setup the system, we'll create two system defaults: FTP - HOST and FTP - PORT. These indicate the server and port (typically port 21) to which files will be sent. Because we need to collect system defaults programmatically, we will plan to use the sys_default function that was described in the Creating an Email Interface in SQL*LIMS article.

Here's the code that we'll use for the PL/SQL FTP interface with line numbers for clarity:

1  CREATE OR REPLACE PROCEDURE send_ftp(name_in IN VARCHAR2,
2        file_in IN VARCHAR2) AS
3     v_cmd VARCHAR2(4);
4     v_datconn utl_tcp.connection;
5     v_conn utl_tcp.connection;
6     v_port_add PLS_INTEGER;
7     v_port_dec PLS_INTEGER;
8     v_reply VARCHAR2(512);
9     v_stat PLS_INTEGER;
10 BEGIN
11    v_conn := utl_tcp.open_connection(sys_default('FTP - HOST'),
12       sys_default('FTP - PORT'));
13    
14    WHILE NVL(v_cmd,'x') != '221 ' LOOP
15       IF v_cmd = '220-' THEN
16          v_stat := utl_tcp.write_line(v_conn, 'USER anonymous');
17       ELSIF v_cmd = '331 ' THEN
18          v_stat := utl_tcp.write_line(v_conn, 'PASS me@here.com);
19       ELSIF v_cmd = '230 ' THEN
20          v_stat := utl_tcp.write_line(v_conn, 'PASV');
21       ELSIF v_cmd = '227 ' THEN
22          v_reply := substr(v_reply, instr(v_reply, '(',1,1)+1,
23             instr(v_reply, ')',1,1) - instr(v_reply,'(',1,1)-1);
24          v_port_dec := substr(v_reply, instr(v_reply,',',1,4)+1,
25             (instr(v_reply,',',1,5) - (instr(v_reply,',',1,4)+1)));
26          v_port_add := substr(v_reply, instr(v_reply,',',1,5)+1,
27             length(v_reply) - instr(v_reply,',',1,5));
28          v_datconn := utl_tcp.open_connection(
29             sys_default('FTP - HOST'),
30             (v_port_dec * 256)+v_port_add);
31          v_stat := utl_tcp.write_line(v_conn, 'STOR ' || name_in);
32       ELSIF v_cmd = '125 ' THEN
33          v_stat := utl_tcp.write_text(v_datconn, file_in);
34          utl_tcp.close_connection(v_datconn);
35       ELSIF v_cmd = '226 ' THEN
36          v_stat := utl_tcp.write_line(v_conn, 'QUIT');
37       END IF;
38       v_stat := utl_tcp.read_line(v_conn, v_reply);
39       v_cmd := SUBSTR(v_reply, 1, 4);
40    END LOOP;
41    
42    utl_tcp.close_connection(v_conn);
43 EXCEPTION WHEN OTHERS THEN
44       dbms_output.put_line('ERROR: ' || SQLERRM);
45       utl_tcp.close_all_connections();
46 END send_ftp;

Lines 11-12 use the System Default function to return the FTP host and port number and the UTL TCP Package Open Connection function to open a connection to the server.

Lines 14 and 40 open and close a loop to step through the server's responses. This is necessary because FTP servers return a varying number of text lines when responding to certain commands. Lines 15 through 37 contain an IF block that contains each of the commands to which we need to respond.

Line 16 and 18 send the username and password to the server. These values would normally be stored in a secure location, and would not use an anonymous login.

Line 20 tells the FTP server to enter passive mode, which allows the client to contact the server with a new data port.

Lines 22 through 27 calculate the port on which data will be sent. The standard FTP protocol operates on two ports, the command port, which is typically 21, and the data port, which can be any port above 1024. Because servers are frequently contacted from outside the current network, traffic on the command port usually passes through firewalls and routers easily. Once the connection is made, the FTP client specifies the passive mode and calculates a new, unprivileged, or private, port on which to send the file data.

Line 28 through 30 opens the data port, and line 31 tells the server to store the incoming data as a specific file name.

Lines 33 through 34 send the file data and close the data port.

Line 36 tells the FTP server that we intend to quit, and line 42 carries it out.

To test the interface, start an FTP server that accepts anonymous connections and modify the FTP - HOST and FTP – PORT SQL*LIMS system defaults to contain the server information. Open SQL*PLUS as the SQL*LIMS Owner and execute the following:

exec send_ftp('test.txt', 'Very cool FTP transfer, I must say!');

Look in the root directory of the FTP server for a file named test.txt, it should contain the sentence, "Very cool FTP transfer, I must say!" If it does not, try logging into the FTP server from a command prompt using the information stored in the FTP - HOST and FTP – PORT SQL*LIMS system defaults. This will verify that the server does, in fact, accept anonymous connections.

Overall, our FTP client is decently prepared for standard text file transfer, but lacks robustness. If we were looking to add features, we would allow the procedure to accept the file as an Oracle CLOB, support binary transfer modes, look to see if the filename already exists on the destination system, more adroitly handle common error conditions, etc. Finally, we’d want to build in some kind of queuing mechanism so that if the destination server were unavailable for some reason we wouldn’t loose our data. That would not make us happy!

The "S" reality describes our primary reality; it is the context from which our world is observed, and the basis from which we think our thoughts. Based on this description, you can determine that this reality is intangible since I described it not as "our world" and "our thoughts", but as "the context" and "the basis". For the sake of this explanation, I will describe the "S" reality as nonphysical, powerful, thoughtful, and valuing, and will give examples of each.

NONPHYSICAL: The "S" reality cannot be detected by the five senses but is self-evident. It has neither size, shape, weight, color, flavor, order, nor texture. For example, a thought of or a wish for a new AMD Athlon 64 X2 4400+ are trivial examples of things that are undetectable by the senses, but are real in every sense of the word. They – the thought and wish - can be described clearly, communicated to and experienced by others, and yet cannot be physically sensed. In another example, my physical body can be said to house "me", yet if I were dissected, you would not find "me". Somehow, although a "me-ness" is localized in my physical body, it is not located there. Similarly, the "S" reality is localized to all things, but not located there.

POWERFUL: The "S" reality has unlimited power and energy. For example, if you look around, you probably see a room full of objects that came to be there because of the feelings, ideas, and wills of one or more persons. If you look to any road you will see a car that was created by the power and energy of the "S" reality…the unabashed persistence of a group of people determined to make a metal and plastic object capable of moving on land faster than another living thing. Do you doubt for one second that it took a tremendous amount of creative energy to create this thing? Cars don't grow on trees, so of course it did! The "S" reality contains enough energy for every one of us to build extraordinary things.

THOUGHTFUL: The "S" reality is the primary context for rational thought. What is a human being but a large collection of focused cognition? Our brains simply direct experience, knowledge, and will upon the various subjects and matters that concern them. Furthermore, no physical thing has thought; even my own brain, if placed outside of the body that describes my "me-ness", does not have thought. So, while a thought can be described by chemistry, it is not simply chemical; while "I" can be described by biology, I am not simply biological.

VALUING: From the "S" reality and it's thoughtful context described above we receive the drive to determine value. Based partly on feeling, the ability to concentrate rational thought on personal choices is provided through this reality. Without it, we have neither the context for rational thought nor the creative energy to affect our value choices on the world around us.

Although I have only briefly described several aspects of the "S" reality, I find it completely compelling. What could each of us do if we had 1) the assurance that our primary reality was integral to our being, 2) the unlimited creative energy to affect the world around us, 3) a firm context for rational thought, and 4) the context from which to choose value in our actions? A lot, that's what!

Now comes the part where I tell you my anonymous word and the title of the book I'm reading; or I don't. No, I don't think I will. Instead, I leave that search to you. Here are some hints:

1) The word appears in the J&R Mission statement

2) A reverse lookup on the word might be: Of, relating to, consisting of, or having the vital principle or animating force present within living beings; not tangible or material.

Good luck in the search.

1  CREATE OR REPLACE PROCEDURE send_memo(notify_in IN VARCHAR2,
body_in IN VARCHAR2) AS
2     v_document_id NUMBER;
3     v_r_status NUMBER;
4  BEGIN
5     naip_utility.counter(v_document_id, 'SERIAL', v_r_status);
6  
7     INSERT INTO nai_documents
8        (document_id, document_version, file_class,
9       document_clob, status, userstamp, timestamp)
10    VALUES
11       (v_document_id, 1, 'INTERNAL',
12       body_in, 'NEW', USER, SYSDATE);
13 
14    INSERT INTO nai_memos (userstamp, receiver, timestamp,
15       document_id, status, origin)
16    VALUES (REPLACE(USER, 'OPS$'), notify_in, SYSDATE,
17       v_document_id, 'NEW', 'Send Memo Procedure');
18    commit;
19 EXCEPTION
20    WHEN OTHERS THEN
21       dbms_output.put_line('ERROR: ' || SQLERRM);   
22 END send_memo;
23 /

Line 7 returns a document ID from the serial counter and uses the core SQL*LIMS Utility package to do so. This ID associates a master memo record with a detail record.

Lines 7-12 insert the memo into the Documents table, use the standard file_class of INTERNAL, and a status of NEW. When the user selects the memo using the Read Memos form, the status automatically changes to READ. This feature sets the SQL*LIMS memo apart from internet email because the storage and receipt of the notification is guaranteed unlike internet email, whose storage and delivery is not guaranteed.

Lines 14-17 insert the user notification records into the Memos table. This will cause the SQL*LIMS status bar to display a message to the user that a notification is waiting to be read. The origin column is set to “Send Memo Procedure”, although any value may be used.

So, we've reviewed the code; let's give this a test run. Log into SQL*Plus using the LIMS Owner account for the SQL*LIMS instance. This account is the user that owns the LIMS instance, for example ops$penlims. At the SQL*Plus prompt enter:

exec send_memo(REPLACE(USER, 'OPS$'),
'Your tests look terrific, you get a raise.');

SQL*Plus responds "PL/SQL procedure successfully completed", and the LIMS Owner will immediately receive a SQL*LIMS memo that can be reviewed using the Read Memo form.

Using the new procedure in the application is simple: when an event occurs in a status action or custom program that should result in a user notification, execute the send_memo procedure and supply the LIMS User and the text of the notification.

We now have SQL*LIMS email and memo interfaces. For a more robust solution, the two procedures may be combined into a single messaging interface that sends notifications to either email or the memo system based on either programmatic input or a system flag such as a System Default.In my next article, I will describe how to create a simple FTP interface that can be used in a SQL*LIMS status action to send a file to another server.

To setup the system, we'll create two system defaults, EMAIL - SMTP SERVER and EMAIL - SMTP PORT. The server default indicates the name of the SMTP server used by your company to send email, and the port default indicates the TCP port number on which the server is listening. These values are typically something like "mail.jandrconsult.com" and "25", so these will be the example values. We also need to store a user's email address, so use the Define User Classes form and the Address tab in SQL*LIMS to create a new class called EMAIL. Next, use the Define Users form and User Information block on the Names tab to store the LIMS Owner's email address. For example:

Seq    Address Class      Address Line

1      EMAIL              Jeff.Vannest@jandrconsult.com 

Now to make the design as modular as possible, create a function and procedure for the collection of email addresses and system defaults. Here are some very simple examples:

CREATE OR REPLACE PROCEDURE create_email_list(notify_in IN VARCHAR2,
	send_to_name_out OUT VARCHAR2, send_to_noname_out OUT VARCHAR2)
AS
BEGIN
	SELECT '"' || last || ', ' || first ||
	'" ' || addr_line, addr_line
	INTO send_to_name_out, send_to_noname_out
	FROM nais_addresses a, nais_usernames u
	WHERE u.userid = a.userid
		AND a.userid = REPLACE(UPPER(notify_in), 'OPS$', NULL)
		AND a.class = 'EMAIL'
		AND rownum = 1;
END create_email_list;
/

CREATE OR REPLACE FUNCTION sys_default(name_in VARCHAR2)
	RETURN VARCHAR2
AS
	v_response nais_system_defaults.value%TYPE;
BEGIN
	SELECT VALUE
	INTO v_response
	FROM nais_system_defaults
	WHERE UPPER(name) = UPPER(name_in);
	RETURN v_response;
END sys_default;
/

Of course, these examples need things like good error handling, code comments, and input parameter checking, but you get the idea. Here is the send email procedure with line numbers added for clarity:

1  CREATE OR REPLACE PROCEDURE send_email(body_in IN VARCHAR2,
2        subject_in IN VARCHAR2, notify_in IN VARCHAR2)
3  AS
4     v_full_body VARCHAR2(10000);
5     v_owner_name VARCHAR2(1000);      
6     v_owner_noname VARCHAR2(1000);      
7     v_mail_conn utl_smtp.connection;   
8     v_reply utl_smtp.reply;      
9     v_send_to_name VARCHAR2(1000);      
10    v_send_to_noname VARCHAR2(1000);      
11    v_smtp_server nai_system_defaults.value%TYPE;
12 BEGIN
13    create_email_list(notify_in, v_send_to_name, v_send_to_noname);
14    create_email_list(USER, v_owner_name, v_owner_noname);
15
16    v_smtp_server := sys_default('EMAIL - SMTP SERVER');
17    v_mail_conn := utl_smtp.open_connection(v_smtp_server,
sys_default('EMAIL - SMTP PORT'));
18
19    v_reply := utl_smtp.helo(v_mail_conn, v_smtp_server);
20    v_reply := utl_smtp.mail(v_mail_conn, v_send_to_noname);
21    v_reply := utl_smtp.rcpt(v_mail_conn, v_owner_noname);
22
23    v_full_body := 'To: ' || v_send_to_name || CHR(10) ||
24       'From: "' || USER || '" ' || v_owner_noname || CHR(10) ||
25       'Subject: ' || subject_in || CHR(10) ||
26       'Content-Type: text/html' || CHR(10) || CHR(10) ||
27       '<HTML><BODY><P><PRE>' || CHR(10) ||
28       body_in ||
29       '</PRE></P></BODY></HTML>';
30
31    v_reply := utl_smtp.data(v_mail_conn, v_full_body);
32    v_reply := utl_smtp.quit(v_mail_conn);
33
34 END send_email;
35 /

Lines 13 and 14 use the create_email_list created earlier to create the proper email address lines that the SMTP server expects. You'll notice that each call to the procedure returns the user's name and email address twice, once with the name included, and once without. These are intentional and will help the user see the name and email address of the sending and receiving users normally when receiving an email message. Line 14 uses the Oracle keyword "USER", which indicates that we assume this procedure will be executed by the LIMS Owner.

Lines 16 and 17 retrieved the name of the SMTP server name stored as a system default, and opened a connection to the mail server using the port specified by the appropriate system default.

Lines 19, 20 and 21 are used to introduce ourselves to this particular SMTP server, who comes from a long line of aristocratic servers and appreciates the formality.

Lines 23 through 29 construct the email message. Notice that simple HTML tags are used in lines 27 and 29 to wrap the body of the email message, which is consistent with the content type shown in line 26. If all of this technical detail sounds foreign don't be concerned; simply know that using pre-formatted HTML in the message body allows the text to be formatted for easy reading.

Finally, line 31 sends the full message text and line 32 says, "Good day, sir, I had a wonderful chat. My regards to your lovely wife."

So, we've reviewed the code; let's give this a test run. Log into SQL*Plus using the LIMS Owner account for the SQL*LIMS instance. This account is the user that owns the LIMS instance, for example ops$penlims. At the SQL*Plus prompt enter:

exec send_email(USER, 'Hello LIMSWorld!',
'Your tests look terrific, you get a raise.');

SQL*Plus responds "PL/SQL procedure successfully completed", and in a few minutes, the LIMS Owner will receive an email message at the address defined in the Define Users form. If this is not the response, then review the values of the SMTP system defaults.

Using the new procedure is simple: when an event occurs in a status action or PL/SQL event that should notify a user, execute a call to the send_email procedure and supply the LIMS User to which the message should be sent, the subject of the email, and the text body of the email.

In my next article, I'll describe how to create a simple PL/SQL interface to the SQL*LIMS memo system.

Tied with using a task list with four out of ten responses was finding time to work without interruptions. In Joel Spolsky's Joel on Software he writes, "We all know that knowledge workers work best by getting into 'flow', also known as being 'in the zone', where they are fully concentrated on their work and fully tuned out of their environment. They lose track of time and produce great stuff through absolute concentration. This is when they get all of their productive work done. Writers, programmers, scientists, and even basketball players will tell you about being in the zone." In general, J&R consultants get into the zone by working outside of normal business hours. Current customers may find this surprising, since when you call us we're always available. There are two tricks to being available. First, even if a consultant does not work during normal business hours, he is normally still at home, and therefore available to answer the phone. Second, many J&R consultants work well over 40 hours a week, meaning that although they are working after business hours, many of them also work during business hours! The other response related to finding uninterrupted time to work was to eliminate distractions actively. This means more than simply looking for quite time, it means controlling the home environment so as not drain concentration from the present task.

The final two productivity tips were to create a "real" office at home and to take regular breaks. Creating a real office means that consultants pay for nice, comfortable office furniture rather than using the desk mom and dad gave them for their 13th birthday. Taking regular breaks may mean finding calming, thought-free activities around the house that allow for time to reorganize thoughts and clear the mind for especially complex tasks.

So if you visit the home office of a random J&R Consultant, you will likely find a "real" office with comfortable furniture, a task list that has been prioritized and broken down into sub-tasks, and a distraction-free environment that allows the consultant to work without interruption.

And coffee. That's my one and only tip for both technology and productivity and health and sanity and long life: lots and lots of coffee.

Groove Virtual Office is a tightly bound collection of software tools that allows all members of a team to see and interact with the same documents, lists, and records. For example, our main J&R workspace currently includes the following tools: Files, Calendar, Discussion, Tasklist, Knowledge Base, Training Records, Corporate Policies, Media Repository, Mission Statement, and Pictures. Similar to a corporate intranet, each of these tools contain files and records that are particularly relevant to some aspect of our business; however, that is where the similarities end. The most notable difference is that all of the files and records that are available when connected to the Internet (and other users) are also available when a network is unavailable, like on the airplane that is my current office. I would estimate that each J&R person has five to seven discrete Groove workspaces in which they participate.

The use of multiple monitors connected to one or more computers was a technology tip that surprised me. I must admit that I have a single monitor, and was astounded to discover that more than half of my co-workers not only use multiple monitors, but also claim that it is a very important part of the work environment. When connected to one computer, the native operating system is frequently allowed to handle the desktop. When connected to multiple computers, a program called Synergy is frequently used to manage the monitors. Rob did a good job introducing Synergy, so I will not reiterate its benefits.

The last two technology tips were a software program called Microsoft OneNote, and having adequately equipped computing resources.

OneNote is an electronic notebook program, and is a new addition to the Microsoft Office suite and Microsoft SharePoint services. Picture a stack of legal notebooks, each clearly tabbed by category and topic. Now imagine that stack of data being stored electronically; this gives you an idea of OneNote’s power to organize your life. One of the best uses of OneNote I've seen is conducting meetings with all users connected to a shared notebook. When used in this manner, every user can add, remove, or modify text on the fly. I find this very useful, since I'm more apt to thoroughly understand meeting minutes if I can see them being created during the meeting than if I browse them (usually several days) after the meeting is conducted.

Having adequately equipped computing resources was also recognized as being important, especially having a computer with enough memory, a fast enough CPU, and a broadband cable connection. I agree with each of these, and regularly replace my core computer system every 18 months or so. This is even relatively cost effective if you are able to create and maintain your own system from individual parts. In my opinion, this usually beats the heck our of any service contract I have had to deal with.

Well, there you have it: Groove and OneNote for application productivity, and multiple monitors connected to well-equipped computers; these are the J&R greatest hits for technical efficiency.

About two weeks later I noticed something: I could always find the tape. You see, in a bookstore, you can rarely find the boxing tape. The reason that you can never find the tape is because it is used constantly, so it's a moving target. But because you can never find the boxing tape, the staff keeps exactly 47 rolls of boxing tape stashed around the backroom and behind the cash register. So, it’s not just one roll you cannot find, its 47 rolls you cannot find. After following the consultants' advice, I could find the boxing tape (and the 46 remaining rolls, which we stashed in a hidden cabinet so as to not confuse the district manager when she did inspections). This was my first contact with the franchise phenomenon, and coincidentally, my first contact with consultants.

In "The Third Wave", Alvin Toffler states, "Individuals need life structure. A life lacking in comprehensive structure is an aimless wreck. The absence of structure breeds breakdown. Structure provides the relatively fixed points of reference we need." I believe this quote because it rings true. I am a person that needs structure, but more importantly, I believe it improves my ability to do quality work. I know people who do not use structure this way, and most of them have expressed to me the desire to start. Yet, when it comes to certain aspects of life or work, most people I know are extremely resistant to using structure.

(Notice that I say "using" structure, not "being" structured. Structure is something a person does, not who they are. Granted, some people enjoy a structured environment, but for all of us, it is a task.)

Here are some of the reasons I use structure in my work:

• Organizing my tasks keeps my mind free of clutter. Consider a desk covered with PostIt notes; that is what my mind feels like when I have not organized my work. I suppose the person who uses PostIt notes must pat themselves on the back for having tasks written down. That’s a start, I suppose.
• Organizing my workspace keeps my electronic desk free of clutter. My file system is organized by customer and geographic location, project and stage of work. Rather than spending time searching for documents in strange locations with strange names, I choose to spend my time at the beginning of the task organizing my workspace, so that documents are always placed where I expect to find them.
• Prioritizing tasks keeps bosses and customers happy with my turn-around time. I find that doing tasks in order of priority allows me to complete all tasks within a period that is acceptable to each task. Spreading a task out over 3 months is completely acceptable if the task is scheduled for completion in 5 months. I try to ignore what I want to do, and focus on what my customers and boss wish me to complete.
• Using repetitive actions increases accuracy. For example, if I’m writing an important lifecycle document, I immediately save it with the proper document name and designation, and check it into the document repository. This guarantees that if I update the document in the future that the repository software will visually remind me that the document must be updated.
• Using document and code templates increases the readability and consistency across multiple documents and multiple consultants. I find that having consistent document and code structures speeds the development of documents and the maintenance of code.

It may seem that I spend a lot of time organizing; nothing could be further from the truth. I find that having an organized workspace is the simplest thing in the world, and once begun, requires almost no additional time.

At some point in the future I intend to explore how a small consultancy might implement franchising-like paradigms to run the business.