On the First day of CM, my true boss gave to me: a hard drive and a keyboard
In the days of old, most of the team simply stored files on their personal hard drives and these files were shared with others on demand when the need arose. As you might imagine, this lead to many problems with files being lost and other such atrocities.
On the Second day of CM, my true boss gave to me: a shared network drive
In more recent years past, we stored files on shared file systems using simple network drives. This was still inadequate as there was little control over who did what.
On the Third day of CM, my true boss gave to me: Groove
Later, we elected to use peer-to-peer software to improve on issues such as overwriting each other’s files or forgetting to copy our files to the network server. Groove remains to this day one of the most valuable tools our team uses. But Groove is certainly not the ideal tool for maintaining software versions. (wow, a whole paragraph on Groove and not a single snide remark about Microsoft [groove aquisition])
On the Fourth day of CM, my true boss gave to me: Concurrent Versions System
The first quantum leap came in late 2002 when we acquired the first key component in our Source Configuration Management strategy: the version control system CVS. We used CVS exclusively for several years and are still using it today. Although a drastic improvement over shared drives or non-versioning solutions, CVS had some drawbacks. Most annoyingly for our team, CVS did not allow files or folders to be renamed. Further, CVS did a poor job of managing binary files (e.g. Oracle Forms/Reports files, MS Word documents, etc) – every time a binary file is checked in, the entire new version of the file is stored in the repository. It is not uncommon for us to have 50 or more versions of an Oracle Form or Report, each version being 10 megabytes or greater in size. This has lead to high disk usage demands on our repository servers. These and several other downsides led us to search for a solution better than CVS.
On the Seventh day of CM, my true boss gave to me: Subversion
We thought we had seen the star in the East when two team members simultaneously and independently stumbled across a relatively new project that was beginning to gain popularity in the software world. The Subversion project made great effort to proclaim their goal: “to build a version control system that is a compelling replacement for CVS”. We decided to test Subversion out on internal projects before committing to use it for production/customer projects.
Subversion did prove to be an improvement over CVS and addressed many of our gripes with CVS – I even wrote about our anticipated use of Subversion in previous blogs. So we began making plans to convert our customer repositories over to CVS. However, during our investigation phase, we began to realize that while Subversion is great for many types of projects - such as .NET and other web-based projects or Java/C++ projects - it may not be as well suited for Oracle-based, validated LIMS projects.
We went to great lengths to try to squeeze Subversion into the mold of our LIMS projects. But in the end, we could not make it fit. I suppose it is possible that we were stuck in doing something the same way for so long that we could see it no other way. But, I believe we gave our best effort to be open-minded about changing our process to be more closely aligned with the Subversion paradigm, and we still came up short.
The biggest problem we had with Subversion was the way it versions the entire repository instead of individual files. In CVS, we had come to rely on modifying, versioning and delivering single files or groups of files where each file had a precise version. Although Subversion does support the idea of a “version” for each file, the concept is more of a side-effect – versions in Subversion fundamentally apply to the whole repository not to individual files. We could find no reasonable way in Subversion to apply “tags” to individual file versions or to be able to pick and choose precise versions of ad-hoc, individual files when creating builds.
After a long evaluation period we slowly came to the conclusion that Subversion was not the answer for our customer projects. What we really wanted was something that worked like CVS but fixed the things we didn’t like.
On the Ninth day of CM, my true boss gave to me: CVSNT
T’was the night before Christmas and all through the company, not a team member was stirring, not even an intern. Suddenly, Old Saint Jeffrey dropped CVSNT down the chimney. The goal of the CVSNT project, as with Subversion, was to improve upon the flaws of CVS. However, CVSNT differed from Subversion in that it started directly from the open-source CVS codebase and began to add new features that CVS lacked.
CVSNT supports the following features that are not supported by CVS but are important to our LIMS development team:
• File and folder renames
• Efficient storage of binary files
• File level locking and Reserved Edits
• Ability to run the server on Windows
• Simpler setup and configuration of secure, remote access.
We have only been using CVSNT for a few months, but so far it appears to address our concerns with CVS and at the same time is readily compatible with the procedures in place on our LIMS projects. Stated more festively, CVSNT brings all the Holiday cheer of open-source version control without the downsides of CVS which are often associated with the feeling that comes from a night of too much glug.
I hope that this Holiday Season finds you and your LIMS project well.
]]>Benefits
One of the obvious benefits of Java is the code reusability it provides. Generic Java classes can be created and used by many different systems. In a LIMS system, a Java class can be created to perform basic utility functions - for example, a diagnostics system that provides error messaging and outputs the logical flow of your code. This class can then be used by the client applications and GUIs you create and at the same time can be loaded into the database and accessed by all of your database code.
In addition to reusability, Java Stored Procedures can provide significant performance increases over PL/SQL code. PL/SQL is rather slow when performing mathematical computations and string manipulations. So when trying to achieve performance increases, Java Stored Procedures are well suited for numerically or computationally intensive tasks.
Case Study
As an example of the performance benefits of Java Stored Procedures, we will look at a J&R Consulting client whose system was entirely written in PL/SQL. Though the system met all of the User and Functional requirements of the LIMS, the system was unacceptably slow for all but the smallest tasks. Executions of the system on moderate or large datasets would take anywhere from 15 seconds up to 30 minutes or more to complete. And the end-users' client sessions were tied up during the processing, leading to a poor user experience. The consultants at J&R ported the computationally intensive portion of the PL/SQL code to Java, allowing the remaining PL/SQL code to concentrate on what it is best at - database access. This produced a system that worked as follows:
The final system provided dramatic performance increases. Executions that previously took 15 seconds now complete immediately and the 30+ minute executions now complete in about 10 seconds. Needless to say, the users are much happier with the new system.
Deploying Java Stored Procedures
If you are convinced that Java Stored Procedures can benefit your LIMS system, the following provides a brief overview of how to deploy Java Stored Procedures:
The next time you are designing a LIMS system or sub-system, consider how Java Stored Procedures can be used to your advantage or to the benefit of your customers.
So now to the mac. I recently acquired a 17" PowerBook G4. And while I have a lot of great things to say about this fine piece of hardware (which I’ll write about in a future blog), like any good relationship between a consultant and his laptop, I first focus on and overemphasize the negatives (no, not the slow hard drives and stupid pointing devices they put in laptops). I have to say - some things about this machine remind me a lot of that temperature gauge.
First, the reporting of file size: it seems that OS X frequently reports files or groups of files as drastically larger than they really are. For example, my documents folder on my desktop contains 1.7 gigs of files (as reported by `du –h` in linux and confirmed by manually counting the number of bytes in each file:) However, OS X reports my documents folder as 34 gigs. And when I try to copy the folder to the PowerBook (which has 33 gigs of free space), OS X won’t even allow the copy to start because it thinks I don’t have enough free disk space. After freeing a few extra gigs of space, I can copy the directory just fine, and it clearly takes up only 1.7 gigs of space. What’s up with that!? Maybe the geniuses at Apple want to prevent us from using up too much disk space thereby causing system difficulties.
Second, when OS X crashes (yes, I had a crash within the first week of using OS X) the desktop darkens to a shade of grey and you get a warm, fuzzy message indicating that a problem has occurred and you must turn off the computer. I’m pretty sure this crash was caused by a faulty application (one of the apps provided by Apple that many of you probably use even if you are a PC user), but there was no option to ignore the warning and try to save any files or fix the problem without rebooting. While this message is more graphically pleasing than the dreaded Windows BSOD, it is still pretty worthless. I would prefer to know in more detail what caused the mysterious “problem” so that I have the option of dealing with it in the future (again, I like things the hard way).
All the eye candy that comes with a Mac definitely gives the average user lots of warm fuzzies. But for someone who wants to look at what’s happening under the hood and observe precisely the operating temperature of the computer, the buffering from details can be frustrating. It can even cause the user to start distrusting the values they see. Fortunately, since OS X is built on top of an open-source kernel, freaks like me can get their hands dirty and swap out some of those crappy temperature gauges with more accurate tools (that report in Celsius, of course).
If you are interested in additional bizarre analogies, the next time you see Brian, ask him, what “is like a report server”.
]]>For simple problems, it may be possible to hunt-and-peck through inputs and scenarios until we see the results we expect. But for complex issues, trial-and-error will usually be inefficient. And more importantly, when troubleshooting issues in a production LIMS environment it is rarely even possible to perform trial-and-error tests. Unless we are fortunate enough to have the production environment replicated to a testing platform, we are forced to use a more scientific method.
And this is exactly where I found myself recently. While troubleshooting issues in a production LIMS system, I was forced to think through the problems and develop solutions without performing any actions that would modify the state of the system. I walked away from that troubleshooting session with a very clear understanding of the issues and the neccessary solutions. Although I had been through experiences like this many times before, this time it dawned on me just how superior this approach was to the trial-and-error techniques I would have been using had this been a development system.
A scientific troubleshooting technique may take a little extra time up front, but it will usually save time in the long run and will give the troubleshooter a more complete understanding of how a system works. Although not always the easist path forward, troubleshooting issues in a scientific manner really pays off. I, for one, need to become disciplined enough to use the technique all the time, not just when forced into it by circumstance.
At any point in the future, we can examine the properties of the foo package and see that version 27 was unit tested on 28-MAR-2005. And if a subsequent version of the package is tested, a new set of properties can be added to that version to indicate a repeated test with the new unit test document information.
Also note that Subversion properties are versioned the same as the repository files themselves. So, for example, a history of the lifecycle of the unit test document can be tracked - from creation through post-approval of the executed document.
It is easy to see that the flexibility of Subversion properties allows for recording all the information we need in order to answer our version tracking questions. But entering and retrieving the tracking information in a way that is useful to a development team requires some additional thought.
Frequently, a step in the lifecycle (say, an integration test) covers several (maybe hundreds) of files. It would be tedious to enter the relevant information separately for each file. And it would be even more tedious to search through the thousands of versioned files in a large repository for a particular piece of information.
Fortunately, Subversion has the ability to export its version information to XML, which allows for powerful organization of data. And since Subversion is open source, it can be expanded to output properties in the precise layout we desire. Once we have the data, a custom XML-based tool could be created that would allow the user to arbitrarily group files into modules and assign properties to the entire group. Similarly, the tool could be used to retrieve and display the properties of a module in a user-friendly format. If we wanted to find out what version of the foo package was most recently unit tested, we would simply query for its module and see the complete history contained in the Subversion properties.
Another exciting possibility that comes from storing our tracking data right in the repository is the ability to generate complex releases based on the lifecycle history of the files. An interface could be created to automatically select and package the files that meet a set of release criteria. This would help ensure, for example, that only formally tested files are delivered in a release.
Subversion's properties have the potential to turn a basic file repository into a powerful tracking system that meets the needs of complex projects - even validated LIMS projects! I look forward to exploring Subversion's capabilities in more than just theory.
By some strange coincidence, this revelation came at an opportune time. I recently purchased a 10-button Logitech mouse but could only get 9 of the buttons to work in my favorite window manager. Excercising the freedoms of open source software, I checked out the window manager source code from CVS, modified the code, recompiled it and am now enjoying all 10 buttons of my mouse. For years, I had been enjoying the no-cost benefits of open source software, but this was the first time I really took advantage of the fact that the source is "free".
So the next time you are smoking cigars and debating about open source software, remember this important aspect of "free" software. In the open source world, we are living in the land of the free software.
]]>Also, most LIMS projects use the waterfall methodology. In the waterfall methodology, each phase must generally be completed before the next one can begin. While this can be much slower than a more modern or agile methodology, such as XP, the checks and balances that come from one phase flowing into the next are, I assume, what cause the people who make big decisions at pharmaceutical companies to choose waterfalls (or maybe, like the author, they all just have a strange obsession with Hawaii).
A software development project that follows the waterfall methodology may have the following lifecycle phases:
It is worth noting that the testing phases are quite different from other methodologies. For example, while many projects execute unit testing every day (maybe multiple times every day!), unit testing in a validated LIMS environment usually occurs officially only one time, at which point documentation is instantiated to prove the soundness of the code. Once a file has passed unit testing, the test is not repeated unless the code is changed.
With all that said, all you really need to know is that a given file on a typical LIMS project has significant version tracking requirements at each phase of the development lifecycle. Before we start exploring how Subversion can be extended to satisfy those requirements, let's materialize some of the requirements by looking at questions that might be asked regarding the deliverables of a LIMS project:
Someone unfamiliar with the inner workings of a LIMS project may consider this amount of versioning information excessive. But I assure you, the data required to answer these questions is absolutely necessary to maintain control of a validated LIMS project. Next week, we will explore how Subversion can be extended to answer these questions.
CVS Benefits
Three of the primary benefits of using CVS are:
1. Retrieval of historical files
Every time a file is modified, CVS keeps track of the history of a file, including methods to reproduce any version of the file at any time. So if a developer realizes that a recent modification they made has caused adverse affects in the system, a previous version of that code can be retrieved.
2. Software release
CVS can be used to control the release of software. A project can be tagged at a given point in time and delivered to the customer - for example as a prototype of a system being built or as an early phase of a multi-phase project. At any point in the future, all the files that have made up that release can be retrieved.
3. Merging of simultaneous development efforts
Many software development projects involve the efforts of multiple developers. CVS can be used to automatically merge the contributions of more than one developer working on the same file. Our project has used this feature several times, saving time that would have been spent manually merging contributions and decreasing the risk of mistakes.
CVS Limitations
While CVS has served its purpose well, it has drawbacks that have been a source of frustration to members of our team. The biggest complaint by team members has been the inability to rename files and directories. We have encountered several situations where at some point into the development of a system, the folder names and or structures that were defined at the beginning of the project were inadequate. CVS doesn't allow for the renaming of folders or moving files from one location to another. Instead, CVS recommends that you delete the folder or file in question and re-add it to the desired place in the repository. This has the dreadful side effect of losing all of the history of the moved files.
Another problem we've had with CVS is its inability to merge binary files. When a change is made to a plain text file, only the changes are stored on the server. Using diff algorithms, CVS can recreate any version of the file without needing to store every single version. Further, file versions that have gone down diverging paths can be automatically merged back to a main trunk.
However, much of the code our team works with is binary, even in its "source" form. For example, both the source file and compiled file of an Oracle Form or Oracle Report are binary. Because of the limitations in the methods CVS uses to merge files, it can only store full copies of a binary file. Every time we make even the simplest update to a form or report and commit the change to CVS, the entire new file must be uploaded and added to the server. This is obviously a waste of space and bandwidth. But what's worse is the need to manually make changes to a form or report that have already been made on a diverging branch.
The last complaint I will mention with CVS is the potential for selecting the incorrect version of a file for inclusion in a release. Our team has used several documents and tools to record file version numbers for important lifecycle phases (for example to record versions of files that have undergone unit testing). When it comes time to release files, it is a painstaking process to rectify the file versions in the repository with the tracking documents. Because our repositories are disjointed from the rest of our tools, it is possible that an incorrect file version could be tagged and released from the repository.
CVS Replacement
Fortunately, there is a team of developers out there who share these same frustrations and have chosen to take matters into their own hands. Enter Subversion, whose goal is to be a "compelling replacement for CVS". Subversion supports everything [good] that CVS supports and improves on the negative aspects of CVS. From an end-user perspective, Subversion is very similar to CVS, and someone who is already familiar with CVS will feel right at home with Subversion. Best of all, Subversion can be used to address each of the specific complaints that our team has had with CVS.
First, Subversion supports directory and file renaming. And it not only supports the actions, it even versions the changes. Gone are the days when we are forced to live with an awkward organization of our code.
Second, Subversion handles binary files just as easily as it handles text files. It uses a binary diff algorithm, and is therefore much more efficient at transmitting and storing data. We will evaluate whether binary merges can actually be used in practice once we begin using the tool.
Finally, Subversion may be useful in addressing the issues we've had with our version tracking tools being segregated from our repositories. Subversion has great support of meta-data through "properties". Properties can be attached to each file and folder in a repository, providing key-value pairs of information on that object.
The uses of this functionality are far-reaching but could be used, for example, to integrate our version tracking system with our repository. The version tracking data could quite literally be stored right along side the actual files that make up the repository. And the combined versioning and tracking data can be output to formats that allow for extended functionality - for example XML. Someday, I would love to explore the implementation of an XML based tracking and release system that reduces the chances of deploying incorrect versions of a file.
I look forward to the coming days when our team evaluates Subversion and the ways that it strengthens our version control processes.
]]>This software performs a similar function to VNC. Only, instead of streaming the video to the computer you are working on, synergy simply sends mouse and keyboard inputs to the remote computer and displays it on that computer's display. So obviously, Synergy doesn't suffer from the "lag" that makes VNC drag at times.
I am currently using Synergy to add a Windows computer to my Linux environment. I have my Windows laptop LCD sitting next to my Linux monitors, and when I move the mouse off the side of the Linux monitor, the cursor automatically starts moving on the laptop. I can do everything on the laptop as if I were actually interacting with its keyboard and touchpad. Synergy even allows me to cut and paste between these different environments.
For several years, I have been using VMware to run those critical applications that don't run in Linux (SQL*LIMS and Groove). I am considering replacing what I've been using VMware for with this configuration. This spreads my computing tasks over 6 GHz of CPU and 2 gigabytes of RAM, giving me another 1,920,000 pixels of desktop real estate (making my total desktop resolution 6400x1200:)
Note that synergy supports OS X, so those of us using Macs can benefit from a single point of contact to a mixed Mac and Windows environment.
But even for those of us who exclusively use Windows, possible applications for Synergy still exist. We all have more than one computer (i.e. a laptop and desktop/portable server). Synergy can be used to expand the use we get from these computers for the same reasons listed above - namely more computing power and a larger environment.
I could ramble for hours about the benefits of increased desktop space, but for now I will offer only one example: consider the document review tasks we've been doing recently. It is awesome to be able to have the document being reviewed (OQ), a comments document, and the application where document steps are performed (SQL*LIMS) all open and visible simultaneously. There is no need to repetitively switch back and forth between documents and the application.
In the short time I've used Synergy, I have nothing bad to say about it. It is one more small advance in my never-ending search for the ultimate computing environment. Synergy allows the software developer or power computer user to better leverage their computing resources and add more power and real estate to their desktop environment.
]]>