WO1992015964A1

WO1992015964A1 - On-line image processing for real-time control of a process

Info

Publication number: WO1992015964A1
Application number: PCT/US1992/001514
Authority: WO
Inventors: John F. Reid; J. Bruce Litchfield
Original assignee: The Board Of Trustees Of The University Of Illinois; Biotechnology Research And Development Corp.
Priority date: 1991-02-28
Filing date: 1992-02-28
Publication date: 1992-09-17
Also published as: AU1577392A

Abstract

A system suitable for real-time, on-line control of a process is disclosed. The system includes a sampler (12), an image sensor (13), an image analyzer (14); a signal generator and a controller (16) of at least one setpoint of the process in response to the appropriate signal. A process for real-time, on-line control of a process is also disclosed.

Description

ON-LINE IMAGE PROCESSING FOR REAL-TIME CONTROL OF A PROCESS Technical Field

The present invention relates to the control of a process using image processing for feedback and in particular relates to on-line image processing that enables real-time control of the process that includes a process fluid. The present invention also relates to a method for carrying out real-time control of the process utilizing on- line image processing.

Background of the Invention

Processes are performed on materials to change the starting materials to an intermediate or finished material. Often, the process is performed on a fluid material within a reactor. The fluid can contain liquids, gases and particulate matter and can be referred to as the process fluid. Representative processes include bioprocesses, synthesizing processes, refining processes, extraction processes, separation processes, purification processes, and the like. A bioprocess will be used herein as illustrative of the various processes.

For example, a bioreactor provides a controlled environment for a bioprocess that is utilized to grow cells. A representative bioprocess is fermentation. The bioreactor includes therein the cells, media, liquid and gaseous by-products of cell growth, and liquid and gaseous products of cell growth. Collectively, the contents of the bioreactor are referred to as the bioprocess fluids or by their specific bioprocess name, e.g., fermentation fluids. Setpoints, which are desired values of process variables of the bioprocess, often must be varied over time as the cell growth progresses. These variables include the temperature, pH, flow rates, agitation speed, gas content, nutrient content and the like. Often, a current state of these variables of the bioprocess must be determined to ascertain if it is time to change a particular setpoint.

Unfortunately, many conventional methods of determining the current state often require a relatively /15964 _ , .

long time period during which the process variables of the bioprocess can change. This elapse of time can result in the process variables of the bioprocess being maintained for an undesirably long period of time at the incorrect setpoint. The delay in adjusting the setpoint can adversely affect the resultant product or can result in the bioprocess taking an unnecessarily long period of time to be completed.

For example, a sample of the bioprocess fluid can be removed and subjected to analysis to determine the state of the bioprocess. Representative methods of analyzing the bioprocess fluid include bioassay which takes about 4 to 7 days to complete, immunoelectrophoretic assay which takes about 2 days and enzyme-linked immunosorbent assay (ELISA) which is disclosed in "Enzyme-Linked Immunosorbent Assay for Quantative Detection of Bacillus thurin iensis Crystal Protein" Applied and Environmental Microbiology, Feb. 1983, pp. 586-590 which takes about 4 hours. An alterna¬ tive method is described in "A simple haemolytic method for quantification of 5-endotoxin of Bacillus thurincriensis from crude samples" M. K. Majumdar et al., J. Appld. Bacteriology 69, 241 (1990). The elimination of these long time delays is highly desirable.

Many methods of detecting the current state of the bioprocess measure secondary metabolites such as temperature, carbon dioxide and the like. However, these measures provide indirect information on the current state of the bioprocess rather than the preferred direct informa¬ tion on the state. The processing industry desires a system that provides improved control by at least minimizing the time period between sampling and the decision of whether or not to adjust the setpoint and that measures the current state of the process directly. Summary of the Invention

The present invention is directed to a control system that provides real-time control of a process having a process fluid and is especially useful for real-time control of a bioprocess. The control system includes a sampler that isolates a sample of the process fluid; an image sensor that senses an image of the sample; an image analyzer; a signal generator that signals if a setpoint change is necessary; and a controller having at least one setpoint of the process that is updated in response to the appropriate signal.

The present invention is also directed to a method for real-time control of a process including the steps of providing a reactor containing process fluid; sampling the process fluid; sensing an image of the sample; analyzing the image; signaling if a setpoint is to be changed; and changing the setpoint in response to the appropriate signal.

Utilization of the present invention permits real-time control of at least one process variable by the use of a vision-based control. That is, the time period between sampling and the decision to either exercise control by changing a setpoint or not to exercise control is extremely short and, depending upon the components selected for the control system, can be almost instanta¬ neous. This shortened time period is achieved by analyzing the sample to make a visual determination as to at least one morphometric or photometric feature of the sample that preferably provides direct information on the current state of the process. Representative process variables can include temperature, pH, flow rates, agitation speed, gas content, liquid content, particulate content and the like. The morphometric features include size and shape of objects in the process fluid. The photometric features include grey level intensity to determine which portions of the image correspond to particular image components within the sample. The sampler presents process fluid to the image sensor that senses at least one morphometric or photometric feature of the process fluid. The sample of the bioprocess fluid can be isolated by removal from the bioreactor or by isolation therein.

The image sensor senses a one-dimensional (ID) , two-dimensional (2D) , or three-dimensional (3D) image of the sample that can be stored as an array and manipulated by computer software as a computer-based representation of the sample image.

The image analyzer processes the image of the computer-based representation of the array to determine at least one morphometric or photometric feature of the contents of the image. The image analyzer then compares the morphometric and photometric features to the known morphometric and photometric features at which a setpoint should be changed to determine if the current state is one at which requires regulation as by updating.

The signal generator can then actuate a process control setpoint change.

The controller is capable of altering at least one setpoint of the bioprocess in response to the appropri¬ ate signal from the signal generator.

The present invention overcomes at least some of the aforementioned shortcomings of devices utilized to control bioprocesses.

Brief Description of the Drawings

FIGURE 1 is a representation of the real-time control system of the present invention; FIGURE 2 is a flow chart representation of the communication link between components of the control system;

FIGURE 3 is a flow chart representation of a supervisory control software routine; FIGURE 4 is a flow chart representation of a sample control subroutine; FIGURE 5 is a flow chart representation of a vision sensing subroutine;

FIGURE 6 is a flow chart representation of a bioreactor controller subroutine; FIGURE 7 is a graphic representation of a history of a typical batch fermentation of Bacillus thuringiensis (BT) ;

FIGURES 8A, 8B and 8C are images of a sequence of images taken during fermentation of BT; FIGURE 9 is an image taken during the stationary phase of fermentation of BT;

FIGURE 10 is an image taken during the lyse phase of fermentation of BT;

FIGURE 11 is a graph of total cell area in an image versus time; and

FIGURE 12 is a schematic representation of a cell.

Detailed Description of the Preferred Embodiments

The control system of the present invention provides real-time, on-line control of a process. A bioprocess will be used as a representative process. The control system has a sampler that isolates a sample of a bioprocess fluid of the bioprocess, an image sensor that senses an image of the isolated bioprocess fluid, an analyzer that analyzes the image to determine the current state of the sample and hence the current state of the bioprocess; a signal generator that signals if a setpoint is to be changed, and a controller having at least one set¬ point of the bioprocess that is updated in response to the appropriate signal.

The term "real-time", as used in its various grammatical forms, indicates that the time period between the sampling of the bioprocess fluid and the determination of the current state of the sample is relatively short and can be instantaneous upon the proper selection of compo¬ nents in the control system. — o —

Real-time computer tracking is discussed by Poole et al. "Real-Time Computer Tracking of Free-Swimming and Tethered Rotating Cells" Analytical Biochemistry 175. pp 52-58 (1988) . The term "on-line", as used in its various grammatical forms, indicates that the bioprocess is being automatically controlled by sensing and analysis of the bioprocess without requiring human intervention.

The term "fluid", as used in its various gram at- ical forms, includes liquids, gases and particulate matter of the process.

The term "setpoint", as used in its various grammatical forms, identifies a desired value for a process variable of the bioprocess. The setpoint of a particular process variable can change as the bioprocess proceeds.

Representative process variables include pH, temperature, flow rates, nutritional levels, agitation speed, gas content, liquid content, particulate content, process decisions and the like. Representative process decisions include when to change control strategies, determination of process termination due to harvesting or contamination, and the like.

The desired sequence of setpoints can be deter¬ mined empirically from a previous running of the bioprocess or theoretically from a computer model to obtain informa¬ tion on morphometric or photometric features at various times in the bioprocess. This information can be referred to as calibration data. The calibration data of each of the setpoints in the sequence are stored for future refer- ence as a process model. The process model utilized can be changed, or a new process model selected, during processing based upon changes in process characteristics or the current state of the bioprocess. Also, the morphometric and photometric features of the bioprocess being controlled can be used to update the process model.

The image can be analyzed to determine morpho¬ metric features such as cell density, cell size and shape, _ _η _

cellular properties, cell development events, cell counts, sporulation stages, spore and crystal protein measurement, inclusion bodies, timing of cell development, contamination and the like. Morphological measurements using automatic image analysis are discussed in Packer et al. "Morphological Measurements on Filamentous Microorganisms By Fully Auto¬ mated Image Analysis" Biotechnology and Bioengineering 3_5 pp 870-881 (1990) and U.S. Patent No. 4,918,739 to Losente et al.

The image can also be analyzed to determine photometric features such as the gray level intensity, color, fluorescence and like electro-magnetic responses. Gray level intensity is the selection of excitation energy applied to the sample so that the features of, e.g., the cell and inclusion bodies are contrasted by the lighting conditions and can be distinguished from each other. The gray level intensity has typically ranges of 0 to about 255. The present invention is capable of imaging at least one of bacteria, animal cells, vegetative cells, spores, crystals, inclusion bodies, secondary metabolites, particulates, emulsions, reporting structures and the like. Other contents of the fluid can also be imaged. The present control system uses a vision-based control system that analyzes a computer image of a sample to provide direct information on the state of the sample. It is the image generated by the vision sensor that ϊs used to determine the state. The term "direct information", as used in its various grammatical forms, indicates that the information is derived from measuring the morphometric orphotometric features of the process fluid as opposed to measuring the temperature, carbon dioxide level or the like. The direct information can be used to provide information about and control temperature, carbon dioxide level and the like. -■ ϋ *■

FIGURE 1 schematically illustrates a control system 10 that includes a sampler 12, an image sensor 13, a supervisory computer 14 and a bioprocess controller 16. The control system 10 is operable associated with a conven- tional batch-type or continuous-type bioreactor 18 that has a bioreactor vessel 19 and a digital control unit (DCU) 16. The image being analyzed by the computer 14 can be viewed on an optional monitor 20.

The sampler 12 isolates a sample of a bioprocess fluid from the bioreactor vessel 19. Isolation can occur either within the bioreactor vessel 19 or, as shown in

FIGURE 1, outside of the bioreactor vessel 19. At the present time, removal of the sample from the bioreactor vessel 19 is preferred. The sampler 12 has a high preci- sion pump 15 and a series of computer controlled valves 17 and can admix the bioprocess fluid with a diluent or an additive that is an optically differentiable chemical entity, e.g., a stain, antibody or probe, to control the characteristics of the admixture entering the image sensor 13 to enhance the image.

The image sensor 13 projects a view of the bio¬ process fluid onto a camera 24 which forms an image sensor array that can be stored and manipulated by the computer 14 or by an computer (not shown) that is dedicated to the image sensor 13. The image sensor 13 can include vision devices such as a microscope 22 and the video camera 24 that can have conventional controls, e.g., focus, resolu¬ tion and the like that are controlled by the computer 14. A video imaging board (not shown) can be associated with the camera 24, the computer 14 or the image sensor comput¬ er.

The computer 14 contains software that enables conversion of the computer representation of the array into a set of morphometric or photometric features regarding the content of the image. The data on the morphometric or photometric features can be collected and stored in the _ . _„

computer 14 or in an external, dedicated computer (not shown) .

The computer 14 has software that analyzes and interprets the detected image of the morphometric or photometric features as by comparison with the calibration data stored therein. Using this comparison, it can be determined whether or not to signal to the bioprocess controller 16 to change a setpoint thereof.

The computer 14 can also contain software that signals to adjust a setpoint of the bioprocess controller 16.

The bioprocess controller 16 maintains the setpoint of the bioprocess in the bioreactor vessel 19. The controller 16 can be a closed loop control that regu- lates the bioprocess by adjusting a controlled process variable in response to feedback from the bioreactor vessel 19 to change the value of the process variable unless it is at the desired setpoint. This aspect of the controller 16 can help maintain the process variable at a setpoint. The optional monitor 20 can be a cathode ray tube

(CRT) that is operably associated with the computer 14 to permit observation of the image being analyzed by the computer 14.

FIGURES 2 to 6 are flow chart representations of software programs that can be utilized in the present invention. Preferably, the supervisory computer 14 con¬ tains all of the software necessary for the control system 10.

FIGURE 2 is a flow chart representation of the communication pathway of a software routine for the control system. A supervisory control subroutine 26 supervises the control system and regulates the operations of the other elements, e.g., the subroutines, that control the compo¬ nents of the control system. The supervisory control subroutine 26 activates a sampling control subroutine 28 and receives information on the status of the sampling, activates a vision sensing system subroutine 32 data for /15964 _ _1Q _

sensing the current bioprocess state and receives from the vision sensing system 32 data on the morphometric and photometric features detected in the images, activates a bioprocess control subroutine 30 when appropriate to modify a setpoint of the bioprocess controller and reads the status of the bioprocess controller, and also makes deci¬ sions about the course of action to take to sense the current bioprocess state.

FIGURE 3 is a flow chart representation of the supervisory software routine 26. Start 36 of the program corresponds to when the bioprocess is started. The initialization of the supervisory control occurs at a block 38. An inquiry is then made at a decision block 40 to determine whether or not it is time to take a sample. If the answer to the inquiry is "no", the inquiry is repeated. If the answer to the inquiry is "yes", activation of sample collection occurs at a block 42. Then, an inquiry is made at a decision block 44 to determine if sampling is com¬ plete. If the answer to the inquiry is "no", the inquiry is repeated. If the answer to the inquiry is "yes", activation of vision sensing system occurs at a block 46. An inquiry is then made at a decision block 48 to determine whether or not the vision results are ready. If the answer to the inquiry is "no", the inquiry is repeated. If the answer to the inquiry is "yes", the following steps are sequentially performed: activation of sampler evacuation and cleaning occurs at a block 50; retrieval of vision sensor and bioprocess data (calibration data) occurs at a block 52; performance of bioprocess control analysis occurs at a block 54; and analysis and updating of bioprocess setpoints occurs at a block 56 (if necessary) . An inquiry is then made at a decision block 58 to determine if it is time to end the bioprocess. If the answer to the inquiry is "no", then the routine returns to the decision block 40 where an inquiry is made as to whether or not it is time for a sample and the above blocks 40 to 58 are repeated. _ _χι .

If the answer to the inquiry as to whether or not to end the bioprocess is "yes", then end 60 is the next step.

The sampling control subroutine 28 controls the operation of pumps and valves, the admixing of additives, and the cleaning of the sampler and other aspects of the sampling process. FIGURE 4 is a more detailed flow chart representation of the sampling control subroutine 28. Start 64 is the first step of the subroutine 28 which occurs when the process is initiated. Initialization of the sampling devices occurs at block 66 to prepare the system for sampling and test functionality. Then, an inquiry is made at a decision block 68 as to whether or not a command from the supervisory control has been received to perform sampling. If the answer to the inquiry is "no", then the inquiry is repeated. If the answer to the inquiry is "yes", the sampling process occurs at a block 69. The sampling process can include isolating a sample from the bioreactor, admixing the sample with a specified diluent or a chemical to enhance the visual properties of the sample, pumping the prepared sample to the viewing station and purging the sampler. An inquiry is made at a decision block 70 as to whether or not the sampling process is complete. If the answer to the inquiry is "no", then the inquiry is repeated. If the answer to the inquiry is "yes", then, at 71 a signal is returned to the supervisory software routine and the sampling control subroutine 28 returns to block 68 to wait for the next command from the supervisory computer. The signal can include data.

The vision sensing system subroutine 32 receives commands from the supervisory control 26 and executes a series of operations to extract meaningful information from the images in response to the command. FIGURE 5 is a more detailed flow chart of one representation of the vision sensing system subroutine 32. The first step of the vision sensing subroutine 32 is start 72 which occurs when the process is initiated. Initiation of the vision devices occurs at block 74 to test the systems functionality. An /15964 _ . . _

inquiry is made at a decision block 76 as to whether or not a command from the supervisory control has been received to activate the vision sensing system. If the answer to the inquiry is "no", then the inquiry is repeated. If the answer to the inquiry is "yes", then the next steps are: acquisition of the image which occurs at a block 78; enhancement of the image features which occurs at a block 80 (if necessary) ; determination of feature representation which occurs at a block 82; extraction of features and classification which occurs at a block 84; sending the image properties to the supervisory control which occurs at a block 86; and then, at 87 a signal is returned to the supervising software routine and the vision sensing system subroutine 32 returns to block 76 and waits for the next command from the supervisory routine.

The bioprocess control subroutine 30 is conven¬ tional and can be part of a digitally-controlled bioreactor that has provisions for programming the bioreactor to follow a particular schedule of setpoints that are defined by the bioreactor operator. In the conventional system the desired schedule is keyed into the controller. The biopro¬ cess control subroutine 30 can have provisions for controlling the setpoints from a remote computer system. FIGURE 6 is a more detailed flow chart of a representation of the bioprocess control subroutine 30. The first step of the bioprocess control subroutine 30 is start 88 which occurs upon initialization of the process. Initialization of the bioreactor control occurs at a block 90. An inquiry is made at a decision block 92 as to whether or not a command from the supervisory control has been received. If the answer to the inquiry is "yes", then updating the setpoint or process understanding (model) occurs at a block 94. At block 94, the characteristics of the model upon which control of the system is based can be adjusted or a new model can be selected based on the determined change in the process state. If the answer to the inquiry at deci¬ sion block 92 is "no", or after implementing block 94, implementation of the control algorithms for the bioreactor occurs at a block 96. An inquiry occurs at a decision block 97 as to whether or not implementation of the control algorithms is complete. If the answer to the inquiry is "no", implementation at the block 97 is repeated. If the answer to the inquiry is "yes", a signal is returned to the supervisory return and signal at 98 is the next step wherein software subroutine and the bioprocess control subroutine 30 returns to block 92 to await the next control signal.

A representative application will be presented for the image processing functions that can be implemented using a fermentation process for Bacillus thurinqiensis (BT) with a camera that provides monochrome imaging and with a phase-contrast microscope. The processing opera¬ tions of the bioprocess are time dependent with the state of the fermentation. FIGURE 7 illustrates a history of a typical batch fermentation of BT. After the initiation of the fermentation (at time ■ 0) there is an idle period of growth called the lag phase. After the lag phase, the cell population grows continually by utilizing glucose (•) nutrient as the chief energy source for growth until the nutrient is exhausted as indicated by the logarithmic increase (log phase) in optical density (♦) and dry weight (■) up to about hour 9. The optical density was measured using light having a wavelength of 600 nanometers (nm) . When the nutrients are exhausted the bioprocess enters a state of sporulation and crystal production called the stationary phase when further growth ceases and cells start to sporulate. Toxin crystal proteins start to occur at the end of the log phase and almost become full-sized when the spore matures in the stationary phase. After the station¬ ary phase there will be a decrease in the cell population, i.e., the lyse phase, and an increase in the toxin protein content resulting from spore and crystal production.

The viable count (A ) was conventionally deter¬ mined by placing an aliquot of the cell-containing fermen- tation fluid after proper dilution in a petri dish having growth media that is incubated at 30^βC for two days and permitting the viable cells to grow. The number of growing cells are counted to determine the number of viable cells in the aliquot and hence in the fermentor. A comparison of the viable count and the optical density indicates that there is a relationship between the actual number of viable cells as indicated by the viable count and the estimated number of cells as indicated by using machine vision. The image processing functions are preceded by sampling and image acquisition operations directed by the supervisory controller. Protocol between the supervisory controller and the image processing module will dictate the type of analysis to perform and necessary data to pass to the vision module. The complexity of the image processing for an application will match the quality of information that can be derived from the images with the time require¬ ments of bioprocess control.

This description will be limited to two aspects of the bioprocess control; detection of cell-growth during the log phase and detection of spore and crystal production during the stationary and lyse phases. FIGURES 8A, 8B and 8C illustrate a sequence of images during the log phase taken at 5, 7 and 9 hours, respectively. The^* BT cells 100 are distinct, rod shaped dark cellular objects on a background of average intensity.

FIGURES 9 and 10 are images taken during the stationary phase and lyse phase, respectively. 'in the ea_ςly stationary phase (FIGURE 9), the dark cellular objects (cells) 102 being forming inclusion bodies that are the spores 104 and crystal proteins 106. Later in the process (FIGURE 10) , many of the cells 102 have lysed and the image contains bright free-floating spores and crystal proteins 108. The overall structure of the image processing functions for this application are as follows: (1) Reduce the gray-level image to binary images representing the features of interest.

(2) Analyze the objects detected in the binary images to extract morphometric and photomet- ric features.

(3) Make some decisions about the current state of the bioprocess based on classifications performed on the morphometric and photomet¬ ric functions. These image processing operations will vary in complexity based on the time course of the fermentation.

The first step of the image processing functions is to reduce the gray level image to binary images for feature extraction. For BT analysis, the image processing step is simply to threshold the image at multiple levels of intensity to create binary images of cell, spore and crystal features. The vegetative cells can be separated from the rest of the image by selecting a gray level low in value, since these cells are the darkest objects in the image. A second image containing only spores and crystals can be formed by selecting a high gray level since these objects are the brightest objects in the image. Under constant lighting conditions, it can be possible to use the same threshold for all images. If the lighting conditions change over time, automatic techniques for selecting the threshold are intimated.

The term "binary image", as used in its various grammatical forms, indicates the result of converting a multi-level image by image gray level thresholding. Commonly, the threshold value is selected at an intensity boundary between desirable features, e.g., objects to be counted, and other, non-desirable features, e.g., back¬ ground and uninteresting objects that are not to be counted. Thresholding is successful if the intensity properties of the image are related to the features in the image. A binary image can also be referred to as a two level image. — lo —

The second step of the image processing functions is to analyze the objects detected in the image. The feature representation of block 82 (as shown in FIGURE 5) of the image analysis performs one of several types of analysis based on the needs of the particular bioprocess. Some common examples include connectivity analysis and chain coding of boundaries. Morphological image processing techniques represent an alternative example. In the case of BT fermentation, the binary images from thresholding go through a connectivity process conventionally referred to as Run-Length Encoding (RLE) which represents the pixels in the image as a series of connected segments. The connected pixels would identify the objects that might represent vegetative cells, spores and crystals in the image. The term "Run-Length Encoding", as used in its various grammatical forms, identifies a module used to represent the image as a series of connected segments rather than individual pixels. The segments are connected based on similar intensities. An algorithm of the module computes features of the connected segments such as area, perimeter, centroid, bounding rectangle, shape, parameters and the like. RLE can be used to establish "family rela¬ tionships" between objects on the screen, e.g., RLE can be utilized to detect "holes" inside of objects wherein the "hole" represent objects inside of the object. RLE can also be referred to as connectivity analysis.

The feature extraction and classification of block 84 (as shown in FIGURE 5) performs an analysis of the representative features of the BT images. The RLE repre- sentation of the image is used to locate features of the objects that are connected together. The connected pixels have common properties of intensity and common properties of shape. Features that identify the parts of the image based on these properties can be computed for each individual object. Only those features of BT for a parti¬ cular control decision need be computed. The shape charac¬ teristics of the vegetative cells might be computed thus classifying the objects in the image. The objects that are identified as vegetative cells are counted to provide a number that can be related to the population of the biopro¬ cess. Objects that appear as spores and crystals can be related to the toxin protein yield of the bioprocess.

The third step of the image processing functions is to make a decision about the current state of the bioprocess. The visually detected relationships have to be calibrated for the bioprocess. Calibration data can be defined in the supervisory control at initialization and the appropriate relationships could be available to the vision control or the data from the vision sensor could be sent to the supervisory control for evaluation of the relationships. Referring back to the batch fermentation of BT whose history is illustrated in FIGURE 7, in the log phase, the basic goal is to track the increase in cell population over time. In a larger scale fermentation such as a continuous fermentation, this information could be used to control, for example, the nutrient (glucose) feedrate. The following determination of feature representation occurs at the block 82 of FIGURE 5. The image processing functions during this phase involve image thresholding to segment the gray level image into a binary image of the cells (black) on a background (white) . The following extraction of features and classification occurs at the block 84 of FIGURE 5. Connectivity analysis is performed to determine the properties of the objects in the binary image. Morphometric features of the binary images are computed. Photometric (gray level) features of these binary image locations in the image are retrieved from the original image. Some images may contain features that-are not the actual cells. A range of typical values for the features is used to classify the objects. Cumulative properties of these objects are used to indicate the state of the fermen¬ tation process. The morphometric and photometric features can be directly correlated to the number of cells (like object counts) or may be indirectly related to the number of cells (total object areas) .

The following updating of a setpoint and imple¬ mentation of the controller of the bioprocess occurs at blocks 94 and 96, respectively, of FIGURE 6 based upon the determination of the number of cells wherein the number of cells and nutrient feedrate are the process variables. In the fermentation of some, for example, bacteria, the nutrient must be added at various times or the feedrate thereof changed to achieve optimum cell growth as indicated by the number of cells and other indicators. The change in growth rate also provides an indication of the amount of nutrient available for cell growth. The setpoints for the number of cells and the feedrate are set at the beginning of the fermentation process. When the number of cells reaches the setpoint, the setpoint can be updated to a value that represents a greater number of cells. Also, the feedrate setpoint can be updated to change the feedrate into the bioreactor to achieve the desired greater number of cells and to obtain optimum cell growth. For example, when the number of cells have increased so that the images is that of FIGURE 8C, the setpoints can be updated. The updated setpoints are then sent to, and implemented by, the controller. FIGURE 11 illustrates the use of the total object area to indicate the increase in cell population of the fermentor. FIGURE 11 was produced using different gray level intensities of 15 (■ ) , 31 (+) , 47 (*) and 63 (D ) . As the gray level intensity increases, the total pixel area representing cells is increased because more objects in the image are determined to be cells. Log phase analysis will provide information on the beginning of the log phase (end of lag phase) based on the minimum detectable growth, the population of the fermentor based on the increase of object features related to cells, and the end of the log phase based on reaching a desired population state. During the growth process, images represent a fermentation sample at a particular dilution. The lowest detectable state of growth will be an undiluted sample. As the cell population increases, dilutions will control the number of objects appearing in the image. Generally, the principles used for counting plate counts can be applied here. The statistical significance of the counts will be optimal for cell counts between 30 and 300 in a single field of view. The consumption of dissolved oxygen (p0₂) dips in the log phase and returns to peak levels in the stationary phase as the oxygen demand of the cells decrease. Similar¬ ly, if motor speed control of the agitation in the fermentor is based on p0₂, motor speed will track the logarithmic growth and slow down in the stationary and lyse phase. The stationary phase is indicated by the fermen¬ tation reaching a maximum cell population. The lyse phase is indicated by cell death and free-floating spores and crystals which indicates the end of the stationary phase. In FIGURE 7, the beginning of the lyse phase occurs at about hour 20. These standard measurements provide valu¬ able information in the state of growth related to oxygen demand. Machine vision can be used to replace or augment these measurements with sporulation, lysing and the visual appearance of crystal protein.

The same image processing to create a binary image of the cellular objects in the image will be per¬ formed in the stationary and lyse phase. However, at this point morphometric and photometric features related to the properties of objects detected within the cell will be used to indicate the appearance of spores and crystals, and to indicate the spore and crystal population as illustrated in FIGURES 9 and 10.

FIGURE 10 also illustrates that the crystal proteins 108 can appear as free floating bright objects in the image. These bright objects would be detected at a different binary threshold, but would undergo the same type of morphometric and photometric feature analysis used for the cell analysis. Decisions made on the images would provide the quantification of the protein production relative to the results of the feature analysis from the original image processed at the different thresholds.

A single cell is labeled in FIGURE 12. In the classification algorithm of the block 84 of FIGURE 5, the cell is labeled as a "parent" object. The spore and crystal features are labeled as "child" objects related to the parent object. Morphometric and photometric features of the "sibling" objects are available for discriminating between objects.

As indicated hereinabove, the present invention has been described in conjunction with a bioprocess. Other representative processes and uses for the present invention include separation processes, e.g., osmosis and ultrafil- tration, detection of micro-organisms in the process fluids, detection of suspended particulates, detection of characteristics of emulsions or powders and any other process where microscope images provide control informa¬ tion.

The bioreactor can be a fermentor, the bioprocess can be a fermentation process and the bioprocess fluid can be fermentation fluid. The fermentation process can be utilized to produce pharmaceuticals, hormones and the like. A representative commercially available fermentor is a Braun BIOSTAT MD that has a DCU.

Morphometric and photometric features of the other representative processes are analyzed. Representa- tive samplers utilize a sampling loop of continuous piping such as flow through micro-channels from which images can be viewed. Alternatively, a high speed sampling device such as those utilized on high-performance liquid chromatographs (HPLC) can be utilized. A representative sampler is available from Webster Associates, MA.

The sampler can admix the process fluid with a diluent or an additive, e.g., a stain, antibody, probe or the like. The diluent or additive can function to mark a region of interest in the image or to label a region to report a morphometric or photometric feature. Representa¬ tive morphometric and photometric features include the presence or absence of cell characteristics, desirable or undesirable components or properties of the process fluid, antigenic determinants, RNA, DNA, nucleic acids, particular metabolic states, secondary metabolites, reporting struc¬ tures and the like. Representative reporting structures include immiscible droplets containing an optically differ- entiable chemical entity, chemically modified spheres, e.g., latex microspheres that contain fluorophore or absorbance probes and the like.

Vision systems are described in Grand d'Esnon et al. "On Line Evaluation By Vision Systems In Biotechnologies" Paper No. 89 7057 presented at the June 25-28, 1989 International Summer Meeting jointly sponsored by the AMERICAN SOCIETY OF AGRICULTURAL ENGINEERS and the CANADIAN SOCIETY OF AGRICULTURAL ENGINEERING. Preferably the microscope of the image sensor utilizes microscopy coupled with a solid-state video camera and a PC-based vision board. Many types of conventional microscopes can be utilized. A suitable imaging microscope work station is commercially available from Perceptecs Corp., Knoxville, Tennessee. A representative commercially available microscope is an Olympus BH-2. The video imaging board should utilize a transmission protocol compatible with the camera. Optical fibers (not shown) can be used with the camera to include or omit certain spectral content of the images. A representative commercially available computer vision system is available from DataCube. A representative commercially available computer vision software program is available from Inovision.

A representative commercially available supervi- sory computer is a Sun SPARCstation 1+.

Representative processes available to perform the conversion of the computer representation of the 2D array into a set of morphometric or photometric features include image processing, feature extraction, pattern recognition and artificial intelligence fields. In feature extraction the software can analyze the image by taking the raw image which is often multi-level, i.e., a gray level image, and segmenting the image into key features based on some morphometric or photometric feature of the image. The software can be selected to analyze color images where the colors are represented by responses on elements sensitive to visible light components in the red (R) , green (G) , and blue (B) portions of the electro-magnetic spectrum. Each pixel is a vector having an RGB component. Next, the enhanced monitored features of the image can go through a procedure to uniquely identify the features from their image attributes. Each like morphometric or photometric features in the image can be classified together.

An image analysis method is described in Costello et al. "Image Analysis Method For the Rapid Counting of Saccharomvces cerevisiae Cells" Applied and Environmental Microbiology, 49., 4, April 1985, pp 836-866.

A commercial image analysis system is a bio-Foss Automated Microbiology System, available from Foss Food Technology Corp., Eden Prairie, Minn., which is based on the direct epifluorescence filter technique (DEFT) prin- ciple. The bio-Foss system, suitable for testing samples containing 5xl0³ to 5xl0⁸ bacteria/ml, is a modular unit, consisting of a filtration manifold and a reagent system, a macro-stand (to provide incident and transmitted light for viewing petri dishes) , and an epifluorescence icro- scope system with stepping stage, and a microprocessor- controlled image analyzer. With the bio-Foss system, a membrane containing stained cells is placed on a microscope slide and positioned onto the epifluorescence microscope. The image analyzer counts the bacteria in a series of randomly selected fields and prints the count out in both standard and log formats. Results are available in less than 10 minutes depending upon the organism. An alternative image analysis system is the SAMBA 4000 Cell Image Analysis System available from Dynatech Laboratories, Inc., Chantilly, VA. The SAMBA system inte¬ grates cell image acquisition, analysis and data processing to transform images into accurate, repeatable, quantitative measurements of shape, color, texture, optical density, and fluorescence.

The vision system executes the appropriate machine vision algorithms to perform image processing for feature enhancement, represents the image features for feature extraction and performs feature extraction and classification. Different applications can require specific software for each image processing function performed by the vision system. A general software frame- work can implement these operations.

The control system of the present invention enables real-time control of a process by using a vision- based control that images a sensed morphometric or photo¬ metric feature.

Claims

WE CLAIM:

1. A control system suitable for real-time control of a process wherein at least one setpoint, which is a value of a process variable, is maintained by a controller and an analyzed image of a sample provides data on the current state of the process, the control system comprising a means for signalling the controller to update the setpoint based upon the data on the current state as provided by the analyzed image.

2. The control system in accordance with claim 1 wherein the data is of a morphometric or photometric feature.

3. The control system in accordance with claim 1 wherein the control system is computer based.

4. The control system in accordance with claim 1 wherein the data is of at least one of bacteria, animal cells, vegetative cells, spores, crystals, inclusion bodies, secondary metabolites, particulates, emulsions and reporting structures.

5. A control system suitable for real-time control of a process having a process fluid wherein at least one setpoint, which is a value of a process variable, is changed and the change is dependent upon the state of the process, the control system comprising; means for sampling the process; means for sensing an image of the sample to provide data on the current state of the process; means for analyzing the data to determine if the setpoint is to be changed; means for signaling that the setpoint is to be changed based upon the analyzed data; and means responsive to the signal from the signal means to change the setpoint when a setpoint is to be changed.

6. The control system in accordance with claim 5 wherein the image sensing means senses an image that shows at least one morphometric or photometric feature and the data is on at least one morphometric or photometric feature.

7. The control system in accordance with claim 6 wherein at least one morphometric or photometric feature is of a bacteria, animal cells, vegetative cells, spores, crystals, inclusion bodies, secondary metabolites, particulates, emulsions and reporting structures.

8. The control system in accordance with claim 5 wherein the analyzing means compares the data to calibra¬ tion data of a state when a setpoint is to be changed and the setpoint is changed when the current state is substan¬ tially similar to the state indicated by the calibration data.

9. The control system in accordance with claim 5 further comprising a means for maintaining the process at the setpoint.

10. The control system in accordance with claim 1 wherein the sampling means includes a means to admix a diluent or additive with the processed fluid.

11. A method of real-time control of a process having a process fluid wherein at least one setpoint, which is a value of a process variable, is changed and the change is dependent upon the state of the process, the method comprising: providing a process reactor containing process fluid; sampling the process fluid; sensing an image of the sampled process fluid to provide data on the current state of the process; analyzing the data to determine if the setpoint is to be changed; signaling to change the setpoint based upon the analyzed data; and changing the setpoint in response to the signal.

12. The method in accordance with claim 11 wherein the sensed image is of at least one morphometric or photometric feature and the data is on the at least one morphometric or photometric feature.

13. The method in accordance with claim 12 wherein the at least one morphometric or photometric feature is of a bacteria, animal cells, vegetative cells, spores, crystals, inclusion bodies, secondary metabolites, particulates, emulsions and reporting structures.

14. The method in accordance with claim 11 wherein the step of analyzing compares the data to calibra¬ tion data of a state where a setpoint is to be changed and the setpoint is changed when the current state is substan¬ tially similar to the state indicated by the calibration data.

15. The method in accordance with claim 11 further comprising the step of admixing a diluent or an additive with the processed fluid to enhance the image wherein the admixing step occurs before the sensing step.