JP2001503897A - Pipeline processor for medical and biological image analysis - Google Patents

Abstract

  (57) [Summary] Pipeline processor for medical and biological image analysis. The pipeline processor includes a plurality of fine processing pipelines in which computational work is subdivided into a plurality of basic logical operations. The pipeline processor includes an input stage, a split pipeline stage, a feature extraction pipeline stage, and an output stage. The input stage includes means for receiving a digitized image of the biological sample and conditioning the image prior to the dividing stage. The split pipeline receives the conditioned image from the input stage and performs a split operation on the image. The output from the split stage is provided to a feature extraction pipeline stage, which performs the extraction operation and associates a predetermined feature with the split result. The extracted features are stored in an output stage and sent to another computer that performs a classification task to classify the features according to biological meaning.

Description

DETAILED DESCRIPTION OF THE INVENTION A pipeline processor for medical and biological image analysis Field of the invention The present invention relates to processor construction techniques, and more particularly to pipeline processors for medical and biological image analysis. Background of the Invention In the fields of medicine and biology, various images are visually inspected to assist evaluation and diagnosis. For example, radiographs can be used to examine bone and soft tissue. Microscopic images of cells can also be used to identify disease and determine the course of treatment. When these images are further converted to digital representation, the computer can be used to improve image clarity, identify key components, or even perform automatic inspection. These images are very complex, tend to contain a lot of information, and the need for a computer is high. To enable the development of commercially available medical or biological image analysis systems, designers are addressing the problem of performing many computational tasks in the shortest possible time. This requirement led to the use of computer pipeline processing. FIG. 1 shows basic processing steps based on conventional classical image analysis. In conventional image analysis, the field of view 10 is digitized in a first step 1 into a series of spatially distinct elements or pixels 12 (shown individually in FIG. 1 as 12a and 12b). Each pixel 12 contains a digital representation of the integrated light intensity in a spatially distinct area of the field of view 10. After the digitizing step, the image 12 is segmented in processing step 2 in order to separate important areas of the image, ie the objects of interest 14a, 14b, from the background. This division is often a difficult and computationally intensive operation. Next, in step 3, each distinct object 14 in the split phase is uniquely labeled as object "y" (16a) and object "x" (16b). Thereby, the identity of the object 16 can be recovered. In the next step 4, a series of mathematical measurements are made for each subject 16. Each object contains its visual appearance as a series of numerical quantities. This step is known in the art as feature extraction. The final task performed in step 5 is to classify the extracted features 18 using any one of a variety of known hierarchy classification algorithms to classify or identify each segment. As will be apparent to those of ordinary skill in the art, in prior art image analysis, the task of converting a field image into a number representation as a plurality of objects (18a and 18b in FIG. 1) having respective characteristics. Usually requires a great deal of computational effort. Fortunately, the use of high-speed pipeline-building techniques for the entire analysis engine, thanks to the discrete separation of the digital images produced by the image analysis method and the wise design of the sequence of operations The way led. In pipeline processing, computations are divided into a number of computational tasks, similar to an assembly line in computer work. In image processing, if a series of complex image processing operations is subdivided into a smaller series of multiple operations that can be performed serially, and also the material data for processing can be subdivided, pipes Line techniques can be used. In the area of image analysis, the material data consists of the digitized image 12 (FIG. 1). For pipelining, each operation must be performed in series, one small portion of the image at a time, and the image portion must be available in addition to the information available prior to reaching the operation. Cannot request any additional information. In a practical system, it is possible to use any number of sets of delay elements to allow any one pipeline operation to access further information. Generally speaking, pipeline processors can provide speed-ups that are proportional to the length of the pipeline. In pipeline processing, two approaches have been proposed. That is, there are two methods, the coarse particle pipeline processing 20 and the fine particle pipeline processing 30 shown in FIG. In the case of the coarse particle pipeline processing, the image processing operation is divided into relatively large operation blocks 22a and 22b. Each block 22 includes a complex series of operations. Coarse particle methods use relatively high-level computational mechanisms to successfully perform computational tasks. Each high-level computing mechanism 22 can pipeline itself if the work at that level is suitable for pipeline processing. In the case of fine particle pipeline processing, the computational work is divided into a basic series of logical work blocks: AND 32a, OR 32b, NAND 32c, NOR 32d, NOT 32e, XOR 32f. Fine particle pipeline processing techniques are generally the most difficult to design, but give theoretically maximum working speeds. There are attempts in the art to perform calculations based on the speed and power of computer pipeline operations as applied to cytological image analysis. In published PCT Patent Application No. WO 93/16438, Johnston et al. Discloses an apparatus and method for processing data sequences at high speed. The Johnston system includes a massively parallel configuration with distributed memory. The result of each processing step is transferred to the next processing step by a complex memory distribution scheme. In addition, each individual, or atomic operation that is not further divided, must process the entire frame before results for the next operation are available. The overall processing time is of the order of the processing time of one image frame multiplied by the number of each atomic operation. That is, the total processing time is obtained by dividing the number of image storage elements and processing elements by the image scanning time. To reduce hardware requirements, it is necessary to use fast logic and memory. Another problem with this technique is the interpretation of the boundaries of digital images. Boundaries have special problems because objects are deleted or deformed. In fact, overlapping digitization methods eliminate this problem on a global level. Nevertheless, it is necessary for the computer to achieve the correct boundaries of the image in order to limit the work in such areas. Looking at Johnston's system, the system software will need to be edited with new constants. Otherwise, it would be necessary to supply an image size variable to each processor. In this case, those variables must be checked after each pixel operation. Pipeline processors are generally required in applications where data must be processed at a rate of about 200 million bits of digital information per second. If such information were to be processed for image analysis, the number of operations required would be well over 50 billion (50 billion) per second. Therefore, a practical pipeline processor must be able to handle such a huge amount of data. Therefore, there is a need in the medical and biological imaging arts for suitable pipeline construction techniques for image analysis. Summary of the Invention The present invention provides a technique for constructing a pipeline processor suitable for medical and biological image analysis. The pipeline processor includes a fine particle processing pipeline in which computational work is subdivided into a plurality of basic logical tasks. One feature of the pipeline processor according to the present invention is that the output of each processing step is directly connected to the input of the next processing step. Furthermore, each atomic operation requires only one column and three clock cycles from input to output. That is, the total storage of intermediate results is greatly reduced. In addition, most of the elements for storage can be in the form of data shift registers, eliminating the need for memory addressing and reducing the need for memory control. Furthermore, the required processing time is reduced to the quotient of the image scanning time divided by the number of pixels. In practice, this speed is about two-thirds of the pixel scanning speed, which allows the use of much slower, more reliable and cheaper hardware, and also allows the use of fewer elements. According to another embodiment of the present invention, the pipeline processor system uses a single frame, line synchronization scheme where the boundaries of each image can be detected by simple logical operations. These synchronization signals are pipelined to each processing step. This allows image dimensions to be determined within the limits of available storage during a hardware setup operation. The pipeline processor further features a rework buffer. This rework buffer provides a feedback path for examining the results of the pipeline work without changing the normal operation of the pipeline. This rework buffer serves for self-checking of the system and for performance monitoring. In one aspect, the invention provides a pipeline processor for image processing, the pipeline processor comprising: an input stage for receiving an image, connected to an output of the input stage. A split pipeline stage, including means for splitting an image into selected portions, a feature extraction pipeline stage connected to an output of the split pipeline stage, wherein the features are split into selected portions. Means for associating; and an output stage for outputting information related to the processing of the image and a controller for controlling the operation of these pipeline stages. In another aspect, the invention relates to an input stage for receiving an image of a biological sample, a split pipeline connected to an output of the input stage, for splitting the image into selected portions. A pipeline processor for image processing comprising a stage, a feature extraction pipeline for associating features with the selected portion, and an output stage for outputting information related to image processing, including: Hardware mechanism to perform Here, the hardware mechanism has a back surface having a plurality of slots adapted to receive cards with electronic circuits, the cards being a processor card for holding a control processor, processed by the pipeline stage. An output card for holding a memory circuit for storing the obtained information and a communication interface for transferring the information to another computer, one or more module cards, each of which has a plurality of module cards. Including means for receiving pipeline cards, each of the pipeline cards includes a module for a split stage and a feature extraction stage; said back surface for transmitting information and control signals between cards inserted into slots on its back surface. Bus means. BRIEF DESCRIPTION OF THE FIGURES Hereinafter, the present invention will be further described with reference to the accompanying drawings showing preferred embodiments of the present invention by way of example. In the accompanying drawings, FIG. 1 shows the processing steps according to the conventional classical image analysis technique; FIG. 2 shows in a block diagram two approaches for the pipeline processor construction technique; FIG. 3 shows the pipeline processor according to the invention. FIG. 4 shows a pipeline processor according to the invention as a high-speed image analysis system; FIG. 5 shows the acquisition of a spectral image by a camera subsystem for the image analysis system of FIG. 6 shows the hardware arrangement of the pipeline processor according to the present invention; FIG. 7 shows the data distribution on the back of the custom of FIG. 6; FIG. 8 shows the quad card for the hardware of FIG. 6; FIG. 10 shows the data transmission timing on the bus for the back; FIG. 10 shows the pipeline processor FIG. 11 shows a general filter stage for a split pipeline; FIG. 12 shows a general process stage of a feature extraction pipeline in a pipeline processor. Detailed Description of the Preferred Embodiment Referring to the drawings, and in particular to FIG. 3, a pipeline processor 100 according to the present invention is shown in block diagram form. The main subsystems of the pipeline processor 100 are a control processor 110, a split subsystem pipeline 120, a feature extraction pipeline subsystem 130, and an uplink transfer module 140. As described below with reference to FIG. 6, these subsystems of the pipeline processor 100 are carried by a custom back 202 and by a hardware arrangement 200 according to one embodiment of the present invention. The pipeline processor 100 forms the "front end" of the high-speed image processing system 50 as shown in FIG. The pipeline processor 100 performs image pre-processing for final classification or analysis. Processing steps include image segmentation for image conditioning and feature extraction. When these operations are completed, the conversion of the digital image into the characteristic division target is completed, and the next pattern classification and analysis can be easily performed. As described below, the pipeline processor features a general-purpose design that allows the computing elements to be quickly reorganized or changed. The high-speed image processing system 50 shown in FIG. 4 is for automatic evaluation and analysis of a Pap monolayer sample. The image processing system 50 includes a camera subsystem 52, a control computer 54, a host computer 56, and a series of peripheral devices 58. The control computer 54 performs overall control of the system 50 and includes a peripheral control interface 55 for controlling a peripheral device 58. Peripheral devices include a barcode reader 58a, a focusing system 58b, a scanner 58c, and a slide loader 58d. These peripheral devices 58 do not form a part of the present invention. It is merely described in order to see the entire image processing system 50. The focusing system and slide loader are the subject of the inventions of the currently pending applications CA 96/00476 (filed July 18, 1996) and CA 96/00475 (filed July 18, 1996), respectively. Host computer 56 includes a communication interface 57 for receiving processed data from uplink transfer module 140 in pipeline processor 100. The main function of the host computer 56 is to classify the processed data according to a classification algorithm. As shown, the control computer 54 and the host computer 56 are connected by a serial RS232 communication link 59. Control computer 54 is responsible for providing general instructions to pipeline processor 100 and image analysis system 50 (eg, cytological equipment). Here, the pipeline processor 100 is described as an image analysis system 50 for detecting precursors of cervical cancer in cytological samples prepared by standard monolayer techniques and stained according to normal laboratory procedures. However, the pipeline processor 100 according to the present invention provides a construction technique that can be conveniently and quickly reconfigured for other related medical and biological image analysis. Camera subsystem 52 includes light source 61 and charge coupled device arrays (CCDs) 62. As shown in FIG. 5, the camera subsystem 52 creates a series of three digital images I1, I2, I3 from the slide containing the Pap monolayer sample S. This monolayer sample consists of cervical cells and related cytological components prepared according to the well-known Pap protocol. For observation in the visible spectral range, these cells have been stained according to the Papanicolaou protocol, and the digital images I1, I2, I3 each correspond to one narrow bandwidth spectral band. The three narrow bandwidth spectral bands for images I1, I2 and I3 are stained according to the Papanicolaou protocol and selected to maximize contrast between the various key elements of cervical cells. For the purposes of this application, pipeline processor 100 includes three parallel pipelines, each corresponding to one channel or spectral band. Each of these pipeline channels can operate independently of one another or, in certain situations, provide data to an adjacent pipeline. Referring again to FIG. 4, the pipeline processor 110 includes an input conditioning module 150, a high-speed receiver module 152, an analog control module, in addition to the control CPU 110, the split pipeline 120, the feature extraction pipeline 130, and the uplink transfer module 140. 154, a rework buffer module 155. A high-speed receiver module 152 and an analog control module 154 interface the pipeline processor 100 to the camera subsystem 52. The rework buffer module 155 is connected to the control bus for the control processor 110. The rework buffer 155 provides a feedback path for examining the results of the pipeline operation in real time without changing the normal operation of the pipeline. The information is useful for automatic detection and diagnosis of hardware failures. The pipeline processor 110 further includes a two-way communication interface 156 for communicating with the control computer 54. The pipeline processor 110 also includes a general-purpose serial RS232 port 158 and a general-purpose parallel (ie, printer) port 160. As shown in FIG. 4, uplink transfer module 140 includes a communication interface 142 for sending processed data from pipeline 100 to host computer 56. Receiver module 152 and communication modules 142, 156 are preferably fiber-optical based links to provide a high speed communication link with very wide bandwidth. Receiver interface module 152 is used to receive output images from camera subsystem 52. The two-way communication interface 156 is used to receive control commands and status requests from the control computer 54. The uplink communication interface 142 is used to send the results of the segmentation and feature extraction generated by the pipeline processor 100 to the classification module of the host computer 56. Referring to FIG. 6, a hardware mechanism 200 for a pipeline processor 100 according to the present invention is shown. This hardware mechanism 200 allows the computing elements to be quickly reconfigured or modified for use in another type of image processing. As shown in FIG. 6, this hardware arrangement 200 for the pipeline processor 100 includes a back surface 202 and a set of printed circuit cards that plug into the back surface 202. These printed circuit cards include a processor and input card 204, an uplink communication card 206, and four pipeline module cards 208, shown as 208a, 208b, 208c, 208d, respectively. The flow of data between the cards inserted into the back 202 passes through four basic parallel data buses 203, namely a B-bus 203a, an S-bus 203b, an L-bus 203c, and an F-bus 203d. The back 202 further has two serially connected data buses, a video-out bus 205a and a video-in bus 205b. The layout of these parallel buses 203 and serial buses 205 on the back 202 is shown in more detail in FIG. The pipeline module 208 comprises a "quad" module card that can accommodate up to four smaller pipeline modules 209. Advantageously, this configuration allows for rapid prototyping, reconfiguration and modification of the pipeline processor 100. As shown in FIG. 8, the quad module card 208 is a circuit for controlling the direction of data flow between the back 202 of the card 208 and the plug-in module 209 without the need for external jumpers. The device 300 is included. The circuit 300 includes a field programmable gate array (FPGA) 301 and a set of transceivers 302, individually illustrated as 302a, 302b, 302c. The first transceiver 302 connects the quad module card 308 to the F-bus 203d. The second transceiver 302b connects the quad module card 208 to the S-bus 203b. The third transceiver 303c connects the card 208 to the L-bus 203c. In response to the control signals, the FPGA 301 appropriately sets the direction of the data flow to enter or exit the back 202 and the plug-in module 209. Referring to FIG. 7, six data buses 203a-203d and 205a-205b on the back 202 provide a means for distributing information to or from the control processor card 204, uplink communication card 206, quad module card 208. ing. On the back 202, the six buses are arranged in a pair of 96-pin DIN connectors as a 26-bit signal bus. In addition, the back side 202 distributes power and a small set of global signals to all of the cards 204-208. The back 202 also includes a mechanism for identifying each card slot through the use of 4-bit slot identification during reset. Video out bus 205a and video in bus 205b each comprise a 26-bit signal bus. These video buses 205a and 205b are connected in series via a card inserted into the rear surface 202, for example, a control processor card 204 and a quad module card 208a as shown in FIG. This serial connection scheme means that the control processor 204 and the quad module card 208 should be placed next to each other in the back 202 to avoid breaking the serial link. Referring to FIG. 7, a B-bus 203a, an S-bus 203b, an L-bus 203c and an F-bus 203d provide a data bus for the back surface 202. These four data buses 203a to 203d are connected in parallel so that any one of the cards 204 to 208 inserted into the back surface 202 is connected to the bus for data transmission to or from the card. The leveling image bus or L-bus 203c is driven by the input leveling or conditioning circuit 150 (FIG. 4). The L-bus 203c provides normalized image data for each module in the back 202. The L-bus 203c is synchronized with the divided data bus, that is, the S-bus 203b. The split bus 203b is driven by a split output stage 121 in a split pipeline 120 (FIG. 4) residing on the quad card 209. The split output module 121 provides a split map generated in the split pipeline 120 as well as a set of binary images of two label maps (one cytoplasmic and the other nuclear). The feature bus or F-bus 203d carries the smoothed image information from the input leveling module during each image frame. During this time, the frame synchronization and line synchronization bus lines are synchronized with the split bus 203b. At the end of each image frame, feature bus 203 d is used by each feature module in feature extraction pipeline 130 and feature extraction pipeline 130 sends feature information to uplink communication module 140. The operation of the data buses 203a to 203d will be further described with reference to the timing diagram of FIG. The timing for transmitting an image from the CCDs 62 is a timing signal T. _CCD Indicated by The time to transmit the image, ie I1, I2 or I3 (FIG. 5), is t1. After the first image is sent to the image processing section (ie, the high-speed receiver module 152), the input leveling module 150 performs calculations to level the image I1 (I2, I3) and the video for the split pipeline 120 The leveled image is output to the out bus 205a. The time required for the input leveling module to complete its operation is t2, which is shorter than t1. This difference provides the data buses 203a-203d with a gap in the data stream that does not require any transmission, ie, the window shown at time t3. (This interval t3 is approximately 15% of the time t2). Split bus 203b transmits the split result and feature bus 203d transmits the smoothed image result. When the operation of the division pipeline 120 is completed, the division results are provided to the S-bus 203b and the L-bus 203c together with the leveled and smoothed images, respectively. Since no data is transmitted during the timing window t3, the result from the feature extraction pipeline 130 is sent to the next operation without interfering with the normal data flow. Its amount reaches a multiple of the feature on the F-bus 203d. Each of the 26-bit buses 203a-203d and 205a-205b has 24 data bits and two synchronization bits. One of the two synchronization bits is for an image frame and the other is for an image line. In the case of the back control bus B-bus 203a, the frame synchronization signal is used as a data strobe signal and the line synchronization signal is used as a data direction signal, that is, a read / unwrite signal. In the video feedback mode, the back control bus B-bus 203a monitors the output of any stage of the split pipeline 120, and can be used without interrupting the operation of the split pipeline. The pipeline control CPU 110 and the input conditioning / leveling module 150 are conveniently stored on the same printed circuit card 204. The control CPU 110 controls the camera subsystem 52, the divided pipeline 120, the feature extraction pipeline 130, and the uplink transfer module 140. The control CPU 110 includes a processor, port memory, instruction and data memory, control ports, rear interface, and monitor reset (not shown). Control CPU 110 receives instructions (eg, commands and initialization data) from control computer 54 via bidirectional interface 156 (FIG. 4) and returns status information. Following start-up, the control CPU 110 scans, initializes, tests, and returns status information to the control computer 54 for various elements of the pipeline processor 100. Only after a successful start-up is the control computer 54 instructing the pipeline processor 100 to begin capturing images with the camera 52. Control CPU 110 preferably uses a highly integrated RISC-based microcontroller and includes various on-board support functions, such as functions for ROM and DRAM, serial ports, parallel printer ports, and a set of peripheral strobes. A suitable device for this control CPU is Advanced Micro Designs, Inc. AMD 29200 RISC microcontroller manufactured by Sunnyvale, California. The fiber optic bidirectional interface 156 to the control computer 54, the analog control interface 154 for the camera subsystem 52, the control bus interface for the image capture (CCD) control and leveling circuitry and the back 202 are preferably one field programmable. -Configured as a gate array (Field Programmable Gate Array) (FPGA). This FPGA further controls another FPGA in the system 50. The monitoring circuit is embodied as a single chip. The configuration of the FPGA and monitoring circuitry is within the knowledge of a person skilled in the art. The control CPU 110 preferably integrates a controller (not shown) for the boat memory, a serial port 158 and a parallel port 160. In addition, control CPU 110 decodes the six strobe signals for off-chip peripherals. In the preferred embodiment, the control CPU 110 uses an 8-bit wide ROM (Read Only Memory) as the port memory and is configured to be able to directly control this memory. A single SIMM (single in-line memory module) or a dual-bank SIMM is used for command and data memory. Serial port 158 is a pin compatible with a 9-pin PC serial port (an additional serial control line is connected to the on-chip I / O line). The control line for parallel port 160 is connected to a bidirectional latch (not shown) to drive a PC compatible 25-pin parallel port. Some additional control lines can be handled by on-chip I / O lines. This control port drives two dual control channels. Each of these channels can be used in series via interface 154 (FIG. 4) to send data to an analog device such as camera subsystem 52. Referring now to FIG. 10, the input conditioning module 150 is shown in further detail. The input conditioning module 150 is connected between the high-speed receiver module 152 and the split pipeline 120. Input conditioning circuit 150 first receives a set of three material images I1, I2, I3 from camera subsystem 52. To ensure high-speed operation, camera data is transferred over the fiber-optic data link in the receiver module 152 (FIG. 4). This has the added advantage of being relatively immune from electronic noise generated by the surroundings of the automation system 50. The primary function of the input conditioning module 150 is to condition the set of three images I1, I2, I3 before they are sent to the split pipeline 120. Conditioning refers to correcting the images I1, I2, I3 for local variations in illumination level across the field of view and variations in illumination density. The input module 152 includes an input buffer 170, a leveling pipeline function 171, a background level buffer 172, a background level detector 173, and a focus calculator 174. The output of the leveling function pipeline 171 is connected to the input of the split pipeline 170. The output of the leveling pipeline 171 is connected to the F-bus 203d via the delay buffer 175 and the smoothing function filter 176. Furthermore, the output of the leveling pipeline 171 is also connected to the leveling bus 203c via two delay buffers 175 and 177. Each of the set of three images I1, I2, I3 is stored in the input data buffer 170 and is ready for processing by the leveling pipeline function 171 and other computational tasks required by the system 50. The second set of fiber-optic links is used to receive image information from camera subsystem 52 via analog interface 154. While each image I1, I2, I3 is received and stored, a histogram calculator 173 calculates background information and a focus calculator 174 calculates level information. A background leveling buffer 172 stores the background correction values for each of the three images I1, I2, I3. Once calculated, the background correction data remains unchanged and is called repeatedly for each new image entering the input conditioning module 150. The control CPU 110 transmits the focus information to the control computer 54. After each image I 1, I 2, I 3 has been stored in the input buffer 170, they are sent to a leveling (ie, normalization) pipeline function 171 with the background level values stored in the background buffer 172. This leveling pipeline function 171 levels images I1, I2, I3 to derive a cytoplasmic binary map and a nuclear binary map. The background illumination (flash) level detector 173 uses a histogram technique to locate and measure the intensity peak of each image I1, I2, I3. While the intrinsic background can be easily corrected in the pixel pipeline, the variation in illumination levels of the strobe flash in the camera subsystem 52 needs to be examined and corrected to derive the maximum dynamic range from the leveled image. The use of background level information allows the image to be optimally corrected for variations in strobe flash level intensity. The histogram peak detection interface catches the most frequent pixel input values for each image frame and each image channel. This information is used to level (normalize) the input image. Further, this information is used in a feedback path via an analog control line to control and stabilize the flash intensity of the strobe flash lamp. Focus calculator 174 is used to calculate the optimal focus position. The optimal focus position is not generally required in image processing routines, but if the focus position is not yet known, it will be useful at the beginning of the sample analysis. That is, during this early stage, the input conditioning module 150 receives the material images I1, I2, I3, levels them, and then calculates the so-called focus number (based on the Laplacian measurement of image sharpness). Execute The result of the focus accuracy measurement is returned to the control computer 54, whereby the optimum focus position is found in the normal algorithm of movement and measurement. Leveling pipeline function 171 includes a pipelined computing system that receives a single pixel for each of the three image channels and performs leveling operations on them. In the first stage, the leveling pipeline 171 uses the material image and background correction data to correct the image for inherent heterogeneity associated with the imaging system 50. This is done by dividing the source image pixels by the appropriate background image pixels and can therefore be implemented with single pixel pipeline construction techniques. It is implemented using an FPGA (in combination with a look-up table for division) at the logic or gate level and includes the first of the multiple fine pipelines of processor 100. The leveled image from the leveling pipeline 171, the binary map of cytoplasm and nucleus, is then sent to the segmentation pipeline 120. Further, the frame synchronization signal and the line synchronization signal are sent to the divided pipeline 120. These synchronization signals simplify the detection of the edges of the images I1, I2, I3 for special handling. The first stage of the split pipeline 120 is a nucleus detection (NetCalc) function. This stage 122 (FIG. 4) uses a neutral network based approach to determine whether each pixel is nuclear or cytoplasmic. The neutral network is implemented as a look-up table stored in memory and accessed by decoding addresses consisting of pixel density values. This configuration allows the neutral network (or scheme for this type of determination) to be quickly updated and changed when needed. It also allows real-time adjustment of the nucleus detection function based on preliminary measurements of image quality and properties. This configuration of a neutral network is described in co-pending application Ser. No. CA96 / 00619, filed Sep. 18, 1996 under the name of the same applicant. The next stage in the split pipeline 120 includes the Sobel function and the cytoplasm threshold function. The Sobel function includes known algorithmic techniques for detecting the edges of a grayscale image. The Sobel function is required by the split pipeline 120 to guide the purification following the split. For efficiency purposes, this Sobel function is implemented by processing 3x3 blocks of pixels. The cytoplasm detection function uses, in a preliminary step, a threshold routine to identify cytoplasmic regions from background cell debris based on the integrated optical density. The leveled image from leveling pipeline 171 also passes through delay buffer 175. This delay buffer 175 delays the leveled image to hold until the feature extraction pipeline 130 begins processing and all of the images generated by the various pipeline operations are simultaneously present. The smoothing function filter 176 smoothes the leveled images before they are output to the feature bus 203d. The smoothing function uses a standard image smoothing operation that requires 3x3 pixel blocks in a wide pipeline. The smoothing operation is based on averaging neighboring pixels with two different weights. As shown in FIG. 10, another delay 177 is applied to the leveled images before they are output to the leveling bus 203c. The total delay along this path is set so that the images appearing on the L-bus 203c and F-bus 203d are synchronized with the output of the split pipeline 120. The result of the operation of the input conditioning module 150 is the output of a binary image of the preliminary nuclear position and the preliminary cytoplasmic position output to the video-out bus 205a, and the smoothing result of the Sobel operation is output to the feature data bus 203d. Is done. These three data streams are received in the next step of the split pipeline implemented in the modules on quad module card 208. Referring again to FIG. 6, the quad module card 208 is designed to hold up to four pipeline boards 209 to perform the partitioning and feature extraction tasks. Quad module card 208 is designed to provide line drive functionality, time doubling of feature bus and control data bus, and power distribution. It consists of four parallel buses 203a-203d and two serial buses 205a-205b, as described above for the custom back. These buses are driven by bus transceivers and the necessary logic is implemented in a small FPGA. These configurations are within the ordinary skill in the art. Quad module card 208 is central to the general design of the image processing system. The quad module card 208 makes it possible to arrange various modules for performing the dividing operation and the feature extracting operation with appropriate hardware (ie, FPGA). This provides flexibility in the operation of the components, which can improve the accuracy of the segmentation results or add additional features as required by subsequent classification algorithms. Each of the filters 122 generally processes a 3x3 pixel block (8-bit or 1-bit) that is fed to the filters after leveling and input conditioning preparation has been performed as described above. The sequence of operations in the split pipeline 120 includes a global nozzle reduction operation, followed by labeling of the resulting cytoplasmic region, followed by another global noise reduction operation followed by labeling of the nuclear region, and After the final noise reduction is performed, the result is supplied to the S-bus 203b. Referring to FIG. 11, the filter stage 122 for the split pipeline 120 is shown in more detail. As will be appreciated, the filter stage 122 does not include a microprocessor incorporating some kind of software or a set of general purpose adders. The configuration that includes such a processor is a coarse-grained pipeline approach and cannot adequately exploit the power of this type of computational construction technology. Instead, the pipeline processor 100 according to the present invention employs a particulate pipeline approach. Thus, each filter unit 122 includes a number of logical elements, which are arranged in a serial pipeline and perform their functions quickly, thus waiting for the next line in the line. Pass data blocks at high speed. As shown in FIG. 11, the filter stage 122 includes a filter pipeline function module 180. The filter pipeline module 180 has an input 181 and an output 182 for the pixel stream. Also, the pixel stream is connected to input 183 via column delay buffer 184 and to another input 185 via another column delay buffer 186. Filter stage 122 includes a mask stream input 188, a frame synchronization input 189, and a line synchronization input 190. The frame synchronization input is provided via delay buffer 192 to another input 191 and the line synchronization input 190 is provided via delay buffer 194 to another input 193. In operation, the input pixel stream 181 is provided directly to the filter pipeline function 180. Before the pipeline 180 is activated (corresponding to a 3x3 processing element), there is a latency of two full image columns plus three pixels of the third column. The pixel stream 181 is delayed by one column buffer 184 until the second pixel block column is full and the pixel stream is further delayed by the next column buffer 186 until the final 3x3 block column is full. However, it will be appreciated that this total in-out delay time for the pipeline (as opposed to latency) is only one row and three clocks. A mask stream 188 stored in memory is available for logic functions. The frame sync signal 189 and the line sync signal 190 together with the delayed inputs 191 and 193 form the input to the filter pipeline function 180. Noise reduction elements in the pipeline include a combination of specialized erosion and expansion operations. The basic result of these tasks is that the state of the central pixel in the 3x3 block is changed based on the state of one or more neighboring pixels. In the case of erosion, the center pixel is turned "off" provided that the neighbor is correct. In the case of expansion, the pixel is turned "on". Extension functions work on binary pixel maps to correct irregularities. The 3x3 matrix is examined to determine if the center pixel should be turned on based on the number of neighboring pixels that are on ("order"). If the pixel is already on, leave it on. The erosion function is the opposite of the function of the expansion function by returning the boundaries of a block of pixels to their original dimensions. That is, the 3x3 matrix is examined to determine if the center pixel should be turned off based on the number of neighboring pixels that are on ("order"). If the pixel is already off, leave it off. Extended special functions work on source binary maps and edge binary maps to correct irregularities. The 3x3 matrix is examined to determine if the center pixel should be turned on. If the pixel is already on, leave it on. The center pixel of the edge map allows the use of alternative rules for modifying the center pixel of the source map. The extended not join function works on binary maps to correct irregularities without putting neighboring objects in the way. The 3x3 input matrix and the four precomputed result pixels are examined to determine if the result of the center pixel should be turned on. If the pixel is already on, leave it on. The extended special not join function works in the same way as the above extended not join function with the addition of a mask bit. The center pixel of the mask map allows the adoption of alternative rules for modification of the center pixel of the source map. The extended label not join function works on the source label map, the resulting label map and the edge map to correct irregularities without putting neighbors in the way. The 3x3 matrix of the source and the resulting map is examined to determine whether the center pixel should be turned on based on the neighboring pixel that is on. If the pixel is not already zero or the edge map is zero, its value is not changed. The center pixel of the mask map allows the adoption of alternative rules for modification of the center pixel of the source map. In addition to the above, the following tasks are performed in hardware as part of the noise reduction scheme in the split pipeline 120: Subadd 2 module—total input bits of 0, 1, or 2 and more return. Sub-ad 3 module--Returns the total number of input bits as 0, 1, 2, or 3 and more. Sub-ad 4 module--returns the total number of input bits as 0, 1, 2, 3, or 4. Subsum 3 Module—Returns the sum of the two inputs as 0, 1, 2 or 3 and more. Subsum 6 Module—Returns the sum of the two inputs as 0, 1, 2, 3, 4, 5, or 6. Subjoin module-returns the sub-sum of one edge and one corner together. Join Module-Return to true if the extension operation matches the two regions. Order 1 Module—Return to true if one or more nearest neighbors are on. Order 2 Module—Return to true if two or more nearest neighbors are on. Order 3 Module—Return to true if three or more nearest neighbors are on. Order 4 Module—Return to true if four or more nearest neighbors are on. Order 5 Module-Returns true if 5 or more nearest neighbors are on. Order 6 Module-Returns true if 6 or more nearest neighbors are on. Order 7 Module-Returns true if seven or more nearest neighbors are on. Order 8 Module—Return to true if the 8 nearest neighbors are on. After the noise reduction is completed, the pipeline processor 100 proceeds to a process for detecting a cytoplasmic substance or a nuclear substance in the image I. The "detect" function is also performed in the split pipeline 120 and may include both a nucleus detection operation and a cytoplasm detection operation, or may include only a nucleus detection operation. This module in the split pipeline 120 receives the Sobel, NetCal and BitCyt bit streams from the input conditioning module 150 (described above) via the video in bus 205b. These signals are processed in parallel and in a fine pipelined manner, resulting in an unfiltered Nuc, BinNuc, NucPlus and BinCyt bit stream. These results from the split pipeline 120 are used to calculate a feature set at various stages of the feature extraction module in the feature extraction pipeline 130. These feature sets are used for subsequent image classification. These signals to be labeled and the binary cytoplasmic image are sent to the labeling module via video out bus 205a. If only the nucleus detection function is performed, the unfiltered Nuc, BinNuc, NucPlus and BinCyt intermediates are sent to the cytoplasmic detection module using another set of pins on the video out bus 205a. The main task of labeling is to give the segmented regions of the image (nuclear or cytoplasmic material) a unique number so that they can be identified later when the classification is completed. This operation is performed before the feature extraction is started. Therefore, feature extraction can be performed on the labeled segmentation target. The location of a completely heterogeneous nucleus within any single cytoplasmic region can itself be an important feature when attempting to classify cytological material. This function can be performed at the gate level of a Field Programmable Gate Aray, or alternatively, an application specific integrated circuit (ASIC) can be used. Once both the nuclear material and the cytoplasmic material in the image have been segmented and their appropriate labeling completed, processing proceeds to the feature extraction pipeline 130. The main function of this pipeline 130 is to extract features based on mathematical or rationale to be used in classifying each segmented object in image I. The feature extraction pipeline 130 is shown in FIG. . . . 130m includes a number of feature extraction modules 132 individually indicated as 130m. FIG. 12 shows the feature extraction module 132 in more detail. The feature extraction module 132 includes a feature calculator 210 and a plurality of accumulator arrays 212, each indicated as 212a, 212b, 212c, 212d. One block of the accumulator array 212 is assigned to each one feature, and each accumulator in each accumulator block is assigned to one label. Each block is expected to have a total of over 21,000 accumulators. In image processing for the applications described herein, features that can be extracted fall into five categories: (1) morphological features; (2) organizational features; (3) chromatic features; ) Optical densitometer features, (5) features viewed from the context. Morphological features refer to the overall shape and size of the divided object. The organizational features refer to the distribution and interrelationship of the light and dark levels of the divided objects. Chromatic features refer to the spectral characteristics of the segmented object. Optical densitometric features relate to the light intensity within the segmented object. The features viewed from the context are features relating to the physical relationship between the respective divided objects. Referring back to FIG. 4, the uplink transfer module 140 includes an uplink buffer 141 and an uplink communication interface 142. The uplink buffer 141 stores the leveled image data from the image bus 203c and the divided bus 203b. Each image is written separately to separate banks of memory in buffer 141 as follows: all three leveled images, cytoplasmic label, nuclear label and binary image. Once the images have been stored, the image banks can be transferred as needed. At the end of each image frame, feature information is input from the feature bus 203a based on the frame synchronization signal 189 and the line synchronization signal 190. This data is written in the same block of the buffer memory 141 as the image. The feature memory start column and the number of columns are used to determine the end of feature storage. When all feature data from all feature cards has been saved, this data is automatically sent to the host computer 56 via the fiber-optic communication interface 142. The image data is transferred when requested by the host computer 56. The present invention may be embodied in many other forms without departing from its spirit and essential characteristics. Therefore, the above embodiments should be considered as non-limiting illustrative examples, and the scope of the present invention is indicated by the appended claims rather than by the foregoing description. It is therefore to be understood that all modifications that come within the meaning and range of equivalency of the claims are to be embraced within the scope of the invention.

[Procedure for Amendment] Article 184-8, Paragraph 1 of the Patent Act [Date of Submission] December 23, 1998 (December 23, 1998) [Details of Amendment] Claims 1. In a pipeline processor for image processing, (a) an input stage for receiving an image; (b) a divided pipeline stage connected to an output of the input stage, wherein the divided pipeline stage includes a plurality of selected image stages. And (c) a feature extraction pipeline stage connected to the output of said split pipeline stage, said feature extraction pipe comprising: The line stage includes means for associating a feature with the selected portion, and the means for associating the feature comprises a fine particle processing element; and (d) for outputting information related to processing of the image. A pipeline processor for image processing, comprising: an output stage; and (e) a controller for controlling the operation of the pipeline stage. 2. The pipeline processor of claim 1, wherein the image comprises a digitized image of a monolayer biological sample prepared according to the Papanicolaou protocol. 3. 3. The pipeline processor according to claim 2, further comprising an input conditioning stage connected to an output of said input stage, and means for conditioning said image prior to processing by said split pipeline stage. 4. 4. The pipeline processor according to claim 3, wherein said conditioning means comprises a leveling pipeline function having means for leveling each distinct image element in said image. 5. The pipeline processor of claim 1, further comprising a rework module connected to the controller, the rework module including means for storing information corresponding to operation of the pipeline stage. 6. The pipeline processor of claim 1, wherein the controller includes an interface module for controlling a camera system that generates images. 7. The means for associating a feature with the selected portion comprises a feature calculator and a plurality of accumulator arrays, each of the accumulator arrays corresponding to a predetermined feature, and wherein the feature calculator is the selected calculator. 2. The pipeline processor according to claim 1, further comprising means for selecting one of said accumulator arrays based on said portion. 8. An input stage for receiving an image of a biological sample, a split pipeline stage connected to an output of the input stage for dividing the image into selected portions, and associating features with the selected portions. A pipeline processor for image processing comprising an output stage for outputting information related to image processing, and a plurality of slots for receiving a card having electronic circuitry. (B) the card comprises a processor card for holding a control processor, a memory circuit for storing information processed by a pipeline stage, and a communication for transferring the information to another computer. An output card for holding an interface, including one or more module cards, where Wherein each module card includes means for receiving a plurality of pipeline cards, each of the pipeline cards including a module of a split pipeline stage and a feature extraction pipeline stage, wherein the split pipeline stage and the feature The extraction pipeline stage comprises a fine particle processing element; (c) the back surface includes busbar means for transmitting information and control signals between cards inserted in slots on the back surface. Hardware mechanism. 9. 9. The hardware mechanism of claim 8, wherein said bus means comprises a plurality of parallel data buses and a video image bus, said video image bus being serially connected between said cards. 10. 9. The hardware mechanism of claim 8, wherein each of said module cards is capable of holding four of said pipeline cards. 11. 9. The hardware mechanism of claim 8, wherein said module card includes means for controlling a direction of data flow between the card and said back surface, said control means responsive to command signals issued by said control processor.

────────────────────────────────────────────────── ─── Continuation of front page    (81) Designated countries EP (AT, BE, CH, DE, DK, ES, FI, FR, GB, GR, IE, IT, L U, MC, NL, PT, SE), OA (BF, BJ, CF) , CG, CI, CM, GA, GN, ML, MR, NE, SN, TD, TG), AP (GH, KE, LS, MW, S D, SZ, UG, ZW), EA (AM, AZ, BY, KG) , KZ, MD, RU, TJ, TM), AL, AM, AT , AU, AZ, BA, BB, BG, BR, BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, F I, GB, GE, HU, ID, IL, IS, JP, KE , KG, KP, KR, KZ, LC, LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, M X, NO, NZ, PL, PT, RO, RU, SD, SE , SG, SI, SK, SL, TJ, TM, TR, TT, UA, UG, US, UZ, VN, YU

Claims (1)

[Claims] 1. In a pipeline processor for image processing,   (A) an input stage for receiving an image; (b) an input stage connected to the output of the input stage. Split pipeline stage, which splits an image into selected portions Including means for breaking;   (C) a feature extraction pipeline stage connected to the output of the split pipeline stage; The feature extraction pipeline stage includes means for associating features with the selected portion. MU;   (D) an output stage for outputting information related to the processing of the image; and   (E) a controller for controlling the operation of the pipeline stage; Pipeline processor for image processing. 2. Monolayer biological assays where images were prepared by the Papanicolaou protocol 2. The pipeline processor according to claim 1, comprising a digitized image of the feed. 3. An input conditioning stage connected to the output of the input stage; and Steps to condition the image prior to processing by the split pipeline stage 3. The pipeline processor of claim 2, including a stage. 4. The conditioning means levels each separate image element in the image. A contract comprising a leveling pipeline function having means for performing The pipeline processor according to claim 3. 5. And a rework module connected to the controller. The module includes means for storing information corresponding to the operation of the pipeline stage. The pipeline processor according to claim 1. 6. An input for controlling the camera system that generates the image by the controller; The pipeline processor according to claim 1, further comprising an interface module. 7. The means for associating a feature with the selected portion comprises a feature calculator and a plurality of features. Accumulator array, each of the accumulator arrays being predetermined. Corresponding to the selected feature, and the feature calculator corresponds to the selected portion. Means for selecting one of the accumulator arrays based on Item 4. The pipeline processor according to item 1. 8. An input stage for receiving an image of the biological sample, connected to the output of the input stage; A divided pipeline stage for dividing the image into selected portions, For feature extraction pipelines to correlate selected parts and image processing Pipeline pipeline for image processing consisting of output stages for outputting related information In the processor, (A) a rear side having a plurality of slots for receiving a card having electronic circuits; Become; (B) the card Processor card, pipeline to hold control processor A memory circuit for storing the information processed by the stage and another An output card for holding a communication interface for transferring to a computer, Includes one or more module cards, where each module card is Means for receiving a number of pipeline cards, wherein Each includes a module for the split pipeline stage and a feature extraction pipeline stage Yes; (C) The back side transmits information and control signals between cards inserted into the slots on the back side. A hardware mechanism comprising busbar means for reaching. 9. The bus means comprises a plurality of parallel data buses and a video image bus; 9. The hardware device according to claim 8, wherein an image bus is connected in series between said cards. Structure. 10. Each of the module cards holds four of the pipeline cards. 9. The hardware mechanism according to claim 8, which can be held. 11. The module card controls the direction of data flow between the card and the back. Control means, the control means of which is controlled by the control processor. The hardware mechanism of claim 8 responsive to a command signal issued.