Automation engineers at Pfizer’s biopharma facility at Grange Castle in Dublin have created a one-size-fits-all generic clean-in-place (CIP) phase class for their colleagues at their sister facility in Andover, Massachusetts. The new phase class has the potential to slash current software development times for CIP by up to 80% and to eliminate delays due to automation rework at the commissioning stage. Pfizer combines state-of-the-art research and development (R&D) facilities with flexible, multi-product manufacturing capabilities at its 70-acre site in Andover. The facility is strategically positioned as a fully integrated site, serving as a vital cog in both Pfizer's R&D and Pfizer Global Supply network. To simplify product transfer between sites, Pfizer Global Engineering decided to upgrade its legacy programmable logic controller (PLC) system at the Andover site to an Emerson DeltaV distributed control system (DCS), to match that of the Grange Castle facility. Following this decision, automation engineers in Andover called on their colleagues in Grange Castle to assist with software development for the upgraded control system. As the Grange Castle control system was built on the Emerson DeltaV platform (an Emerson Digital DCS) 15 years previous, automation engineers at the Irish site had a wealth of experience that their US colleagues wished to exploit.

Problem: replacing bespoke CIP phase-logic-modules

[caption id="attachment_39362" align="alignright" width="300"] CLICK TO ENLARGE Fig 1: ISA-S88 standard addressing batch process control from the International Society of Automation[/caption] Having initially assisted with the development of a suite of equipment module (EM) classes on the DeltaV platform, the Grange Castle team was asked to tackle the problem of developing CIP phase classes to clean all of the equipment (see Figure 1 for location of equipment modules and phases in the S88 batch-controls structure). The Andover facility had the full range of equipment one would expect to find in a state-of-the-art biopharma facility, from buffer vessels to bioreactors of various sizes, media vessels, chromatography feed and pool vessels, various depth filtration and virus filtration skids, and a myriad of transfer lines inter-connecting all of this equipment – all cleaned with the help of 600+ phase-logic-modules running on the existing PLC system. To replace this in the traditional way, with one phase per unit class, or per user to be cleaned, would potentially have resulted in a multitude of CIP phase classes. This would have conservatively taken a small army of automation engineers tens of thousands of hours to develop individual functional specifications, then code and test the resulting software on a development system. Faced with this task, the automation team at Grange Castle decided to look at the possibility of creating a single, generic CIP phase class that could be used for cleaning the myriad of different CIP users. An initial analysis of the existing code revealed all of the usual suspects that one would expect to find in CIP phase classes:
  • Liquid cycles such as acid and base chemical wash cycles, reverse osmosis water/ non-potable cold water (ROW/NPCW) cycles and water-for-injection (WFI) rinse cycles;
  • Dry cycles such as air blow and drain;
  • Code to drive the CIP skid (which is home to the CIP supply pump as well as wash and rinse tanks) to various commands and setpoints such as chem-wash and WFI rinse at a specified flow rate;
  • Equipment module commands and sequencing needed to open and close the user loops to be cleaned in the required order and in a manner that prevents dead-heading the CIP supply pump;
  • Various timers: loop timers to measure the duration of flow through each loop, switch timers when switching between loops, and flow stabilisation timers prior to the commencement of pulsing on wash cycles;
  • Loop options such as a local conductivity probe to measure final rinse conductivity on loops that return to a local drain rather than to the main drain at the CIP skid.
Detailed analysis of the liquid and dry cycles revealed significant amounts of very similar code, which were ultimately some variation of:
  • Drive equipment module commands in a set order to open the next loop/close the previous loop;
  • Drive the CIP skid to required command and flow setpoint, wait for flow to stabilise, commence pulsing and start loop timer and monitor flow for the duration of the timer, when the conditions are met and the timer expires – move to the next loop and repeat.
The team at Grange Castle wanted to design a phase class that would cater for all of the above, but would allow for maximum flexibility when it came to equipment specific data and be portable across unit classes of process equipment.

Solution: better separation of generic CIP process tasks

The result of the team’s efforts is a two-part solution:
  • A generic CIP phase class with composites for each of the CIP cycles (air blow, rinse, chem-wash, etc), each of which have all of the steps required such as open loop, close loop and loop timer. However, all of the data that the steps require to execute – such as EM commands, EM order, loop flowrate, loop duration etc – is abstracted to a CIP data function module (FM – control module without any input/output);
  • A CIP data FM (and accompanying detail faceplates), which acts as a repository for all of the user configurable and loop sequencing data. The data FM stores data such as the collection of CIP cycles which go together to form the cleaning regime, individual loop names, equipment module names and durations for each loop to be cleaned. The FM stores the enormous amount of EM-related data (command, order, path, pulse data) in a series of floating point arrays arranged on a cycle and loop basis. This facilitates the storage of up to 90,000 pieces of data in a highly efficient manner.
[caption id="attachment_39363" align="alignright" width="300"] CLICK TO ENLARGE Fig 2: Generic CIP component interaction[/caption] The fundamentals of the design for this phase were a further development of the existing in-house Pfizer software library, which was initially created for the active pharmaceutical ingredient industry, and was then broadened to suit the rapidly expanding biotech sector. The detail of how the various components of the generic CIP interact is shown in Figure 2.

Built-in flexibility

The phase class is designed to be flexible. The cycles can be run in any preconfigured order, as can the loops. Each step contains code to drive up to twenty EMs; however, only those EMs that have a command configured for the running loop/cycle are driven. Likewise, if a switch timer setpoint for a cycle is left blank, the timer step is skipped. [caption id="attachment_39365" align="alignright" width="300"] CLICK TO ENLARGE Fig 3: Raw data is stored in floating point arrays[/caption] For speed of operation at runtime, the phase class copies all of the data from the CIP data FM when it first loads. This avoids delays in reading from a separate module when loop cleaning is in progress and time is at a premium. One CIP data FM is required for each CIP circuit and has enough storage to cater for various CIP types e.g. full and abbreviated (an abbreviated CIP is typically one without any chemical wash cycles). [caption id="attachment_39367" align="alignright" width="300"]pfizer-grange-castle CLICK TO ENLARGE Fig 4: Detail faceplates add context to the data (main detail)[/caption] The phase class was developed to maximise portability by avoiding entirely the use of aliases when referencing user equipment modules (an alias is a parameter at the unit class which is then resolved to the relevant EM at unit level). It is generally this list of aliases which ties a phase class to a particular unit class and hinders portability, since not every vessel has an agitator EM, or a pH EM for example. While DeltaV has the ability to ignore aliases, it still requires the alias to exist on the unit class before the phase can be assigned. This method can result in numerous aliases on the unit class which are ignored and/or numerous aliases in the phase class code which are only used conditionally – this just adds to the morass of code and makes troubleshooting more cumbersome. Instead, the phase class was developed using a composite which is populated with the equipment module name at runtime (supplied from the CIP data FM) and which then dynamically binds to all of the required EM parameters such as command setpoint and process variable etc. This approach means that the phase class can be assigned to a new unit without any prerequisites. [caption id="attachment_39369" align="alignright" width="300"]pfizer-grange-castle CLICK TO ENLARGE Fig 5: Detail faceplates add context to the data (EM sequence)[/caption] To simplify interpretation of the parameter values in the CIP data FM, the Grange Castle team created several detail faceplates that contextualise the data into a readily understandable format. These allow the user to see the data in a pseudo-English format, instead of as a jumble of numbers, thus enabling them to interpret what goes on at the equipment level on a loop-by-loop basis. The main detail faceplate (Figure 4) allows for viewing and configuring equipment setup such as cycles, loop names etc as well as loop durations and CIP system options, while the equipment module sequence detail faceplate (Figure 5) allows for viewing and configuring the EM commands and sequences that the equipment will execute to open and close each loop as the CIP cycles are performed.

Generic model slashes development time

This approach reduces software development and testing time for automation engineers, and empowers process commissioning engineers on the production floor. According to the team at Grange Castle, this approach slashes development time by up to 80%. Using the traditional approach for a new vessel or line CIP phase class, estimates of 200-300 hours are typical depending on complexity. This estimate allows for the following:
  1. Gathering of process requirements, which might include a set of piping and instrumentation diagrams (P&IDs) with highlighted cycles/loops;
  2. Development of an individual functional specification – hand crafted to the specifics of the new circuit;
  3. Development of phase class code on a test system;
  4. Development of an individual test specification for a software factory acceptance test (FAT); and
  5. Software FAT execution.
With a generic CIP phase class Step 1 remains, as one would expect, however Steps 2, 3, 4 and 5 are eliminated as the code is fixed and is previously tested for all scenarios. These steps are instead replaced with a parameter configuration step for the CIP data FM. This is a relatively straightforward exercise of translating the information from the highlighted P&IDs to a series of loops and paths to be inputted to the CIP data FM, along with loop durations and switch timers etc, an exercise which is carried out either via the detail faceplates or via the customised Excel configuration tool. This simplified process is conservatively anticipated to require no more than 40 hours to configure a CIP data FM for a new circuit. In addition to simplifying the task of CIP configuration, the detail faceplates also minimise the possibility of configuration errors. When the engineer performing the configuration selects an EM on which to specify a command, an intelligent script in the faceplate first interrogates the EM to read the list of available commands and presents them in a drop-down list on a pop-up, thereby ensuring that an invalid command cannot be specified. Similar scripts in the detail faceplates also prevent the selection of sub-loop paths in loops for which sub-loops have not been configured or the addition of a local conductivity probe which does not match the name of a probe in the configuration database. These scripts help prevent a myriad of errors which would otherwise not get caught until testing were the code configured directly via DeltaV Control Studio (Emerson’s DCS coding application).

Detail faceplates streamline commissioning

Not alone do the detail faceplates ease configuration by giving access to all of the equipment and sequencing data, they allow for changes at runtime too – empowering the process engineers involved in commissioning. Using the traditional approach where the phase class is hard-coded, if a loop fails to open because of an incorrect configuration, commissioning generally grinds to a halt and the assistance of an automation engineer is required. The automation engineer then has to initiate a change control process: document the requirements, seek approval signatures, modify and test the software and then download the tested code before the process commissioning engineer can rerun the CIP. This process can be time consuming, losing valuable commissioning time or perhaps losing a slot in the production schedule. Now, instead the process commissioning engineer simply modifies the configuration, via the detail faceplates, to correct the valve sequencing, or modify the parameter value that caused the disruption and restarts the CIP. This process takes only minutes rather than the hours, or indeed days, required for a standard change control process. On completion of commissioning, the modified CIP configuration can be uploaded to the offline database and exported to capture the modifications made during commissioning. Automation engineers at Grange Castle have also created a configuration tool, using Visual Basic to create an Add-In for MS Excel, which allows the CIP data FMs to be created in an offline environment on a standard PC without any special licensing. The tool also facilitates the import and export of the CIP data FMs to the DeltaV database and creates a software design specification document in the form of an Excel worksheet.

The future is bright, the future is generic

Given the potential savings to projects – through shorter and simpler code configuration, in addition to streamlined commissioning and better utilisation of process commissioning engineers – the team at Grange Castle see a bright future for the application of generic phases. The principles of this generic approach to CIP are equally relevant to any facility wishing to reduce the commissioning overhead inherent in the traditional bespoke CIP process and automation engineering activities, and move towards a leaner, more process-driven, approach. Having delivered a generic CIP solution, the automation team are currently working on the development of a combined generic pressure test and steam-in-place solution, with complementary data FM and detail faceplates. They anticipate that this will deliver similar benefits for their US colleagues, and indeed future Irish use cases, in terms of flexibility and time savings. [caption id="attachment_39378" align="alignright" width="240"]conor-boyle-rockwell-automation-grange-castle-pfizer Conor Boyle[/caption] Author: Conor Boyle CEng, MIEI is an automation engineer for Rockwell Automation with over 15 years’ experience in biopharma and is embedded at Pfizer, Grange Castle as a subject-matter-expert in Delta-V Acknowledgements Kieran Sheridan, automation manager, Pfizer Grange Castle Amod Gokhale, automation engineer, Pfizer Grange Castle Sergiu Pasca, automation engineer, Pfizer Grange Castle James Galloway, process engineer, Pfizer Global Engineering Iain Logie, MES consultant, Emerson Process Management