JP5246201B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing program.

There is a technique for extracting a target object from an image.
As a technology related to this, for example, in Patent Document 1, it is an object to perform inspection processing of target cells automatically and accurately and to significantly shorten the inspection time, and the optical microscope has an XY stage mechanism. The control PC has an electric stage, an objective lens control unit, and a CCD camera. The control PC includes a stage (XY) moving unit that moves in the XY directions, and an automatic focus calculation unit that performs automatic focusing in the Z direction. An image input unit that inputs an image from the CCD camera, an image recognition unit, and a result storage database unit that stores an image processing result, and an image captured by the CCD camera is an image input unit The image is displayed on the monitor, sent to the image recognition unit and the automatic focus calculation unit, and the image recognized by the image recognition unit is stored in the result storage database, and the image is input. Parts input to the image is also sent to autofocus calculation unit, it is used in the automatic focusing is disclosed.

  In addition, for example, Patent Document 2 aims to construct a cell-based automatic classification logic corresponding to a facility using adaptive learning samples by making use of the characteristics of a hierarchical network that facilitates the construction of cell classification logic. Using the network, the learning reference sample memory that stores the reference learning sample, and the learning target sample memory that stores the learning sample for each facility, creates a learning sample corresponding to the facility, and adjusts the number of learning samples. Is disclosed.

  Further, for example, Patent Document 3 can facilitate observation of changes in cells and the like by time-lapse observation, and easily calculate the area of a living cardiomyocyte without gene transfer or staining. An object of the present invention is to provide a cell monitoring method and a monitoring device that can be digitized, and in order to monitor the cells in the culture vessel, the imaging means is constant for the cells or the cell population in the culture vessel containing the cells. Images are taken at each time, and the computer compares the images taken one after the other at a specific location (the same location in the cell), detects fluctuations between these images, and corrects the detected fluctuations. In addition, it is disclosed that each image is displayed, the culture vessel is not moved, and the above-described imaging means is moved three-dimensionally by the moving means.

  In addition, for example, Patent Document 4 has an object to quickly and efficiently evaluate cell motility characteristics and confirm the state of cells according to conditions such as administration of a reagent. A difference obtained by performing image processing on the phase difference image at the first time point and the second time point after a certain time to form the first and second cell extraction images and taking the difference between the two images An image is formed, and the moving direction of the center of gravity of the cell at the previous and subsequent time points is detected in the difference image, and then the direction is obtained as a reference direction. The change amount of one cell part is obtained by obtaining the change amount of the pixel value in every predetermined angle over 360 °, the change amount is obtained for N cells under the same conditions, and the cells in the direction of the angle φ are obtained. Of pixel values It is disclosed that a value obtained by averaging the amount of conversion for N cells is obtained for each angle φ, and the change amount of the pixel value of the cell corresponding to the angle φ over a range of 360 ° is displayed as indicating the movement characteristics of the cell. Has been.

  In addition, for example, Patent Document 5 has a problem of observing an entire image of a cell with high resolution and high accuracy and determining the activity of the cell from the obtained image. A dispenser that injects into a plurality of wells of a plate, a sensor unit that irradiates excitation light to the wells to receive fluorescence generated from the cells and obtains a fluorescence image of the cells, and determines the activity of the cells based on the fluorescence images A drug discovery screening apparatus for observing cell dynamics and the like provided with a determination unit, wherein the sensor unit moves a nipo-type confocal scanner, an objective lens, and a focal position of the objective lens in an optical axis direction. A focal position changing unit and a camera for capturing an output image of the confocal scanner; the focal position of the objective lens is moved in the optical axis direction by the focal position changing unit, Lens is irradiated with excitation light to said cells via, to have a function to obtain a slice image of a cell is disclosed.

JP 2004-248619 A JP-A-5-99920 JP 2008-076088 A JP 2007-2222073 A JP 2005-095012 A

  In the present invention, when a first object is extracted using images before and after removing the first object, an object that may have moved when the first object is removed is defined as the first object. It is an object of the present invention to provide an image processing apparatus and an image processing program that suppress the extraction.

The gist of the present invention for achieving the object lies in the inventions of the following items.
The first aspect of the present invention is a receiving means for receiving a first image obtained by photographing a plurality of objects and a second image obtained by photographing a state in which the first object which is an object included in at least one of the plurality of objects is removed. And based on the result of collating the first image and the second image so that the object photographed in the first image and the object photographed in the second image are matched. The first object is extracted based on the first area extracted by the first area extracting means and the first area extracting means for extracting an area where the first object may exist. A second region extracting means for extracting a region where the second object, which is an object that may have moved when removed, and a region extracted by the second region extracting means; As a target, the amount of movement of the second object and Based on the calculation means for calculating the movement direction, and the movement amount and movement direction of the second object calculated by the calculation means, the second image is transformed into an image in a state before the movement of the second object. An image processing apparatus comprising: an image deforming unit configured to extract the first object based on a difference between the image deformed by the image deforming unit and the first image. is there.

  The invention according to claim 2 is a first storage unit that stores an image of the first object extracted by the extraction unit, a detection unit that detects the first object from the second image, and the detection The second storage means for storing the image of the first object detected by the means, and the image of the first object stored in the first storage means as a positive example and stored in the second storage means Learning means for learning the negative image of the first object as a negative example, wherein the detection means detects the first object based on a result learned by the learning means. An image processing apparatus according to claim 1.

  According to a third aspect of the present invention, the object is a cell, the image is an image obtained by a microscope, and the detection unit detects a first cell as a target that is the first object from the first image. When the first cell is detected by the detection means, the control means for controlling the microscope stage to move so that the first cell is in the center, and the first cell are removed. Photographing the previous state as a first image and photographing the state after the microscope stage is moved by the control means as the second image after the first cell is removed The image processing apparatus according to claim 2, further comprising: a receiving unit configured to receive the first image and the second image captured by the capturing unit.

  According to a fourth aspect of the present invention, a first image obtained by photographing a plurality of objects and a second image obtained by photographing a state where the first object which is an object included in at least one of the plurality of objects is removed. The result of collating the first image and the second image so that the reception function to be accepted matches the object photographed in the first image and the object photographed in the second image Based on the first region extraction function for extracting a region where the first object may exist and the first region extracted by the first region extraction function. A second region extracting function for extracting a region where the second object, which may have moved when the object is removed, is extracted by the second region extracting function In the region, the second Based on the calculation function for calculating the movement amount and the movement direction of the object, and the movement amount and movement direction of the second object calculated by the calculation function, the second image is converted into a state before the movement of the second object. An image transformation function for transforming the image into a state image and an extraction function for extracting the first object based on a difference between the image transformed by the image transformation function and the first image are realized. This is an image processing program.

  According to the image processing apparatus of the first aspect, when the first object is extracted using the images before and after removing the first object, there is a possibility that the first object has moved when the first object is removed. It can suppress extracting an object as a 1st object.

  According to the image processing apparatus of the second aspect, learning can be performed without preparing a learning sample in advance.

  According to the image processing apparatus of the third aspect, the stage of the microscope can be controlled to move so that the first cell is in the center.

  According to the image processing program of claim 4, in the case where the first object is extracted using the images before and after removing the first object, there is a possibility that the first object has moved when the first object is removed. It can suppress extracting an object as a 1st object.

It is a conceptual module block diagram about the structural example of this Embodiment. It is a flowchart which shows the process example by this Embodiment. It is a conceptual module block diagram about the structural example in an image reception module and an image difference extraction module. It is a flowchart which shows the process example by an image reception module and an image difference extraction module. It is explanatory drawing which shows the example of the image before cell collection | recovery, and the image after cell collection | recovery. It is explanatory drawing which shows the process example of a global positioning process module. It is explanatory drawing which shows the process example of a cell presence candidate area | region extraction module. It is explanatory drawing which shows the process example of a cell movement candidate area | region extraction module. It is explanatory drawing which shows the process example of a cell movement candidate area | region extraction module. It is explanatory drawing which shows the process example of a displacement vector calculation module. It is explanatory drawing which shows the process example of an image deformation | transformation module. It is explanatory drawing which shows the process example of a target extraction module. It is a block diagram which shows the hardware structural example of the computer which implement | achieves this Embodiment.

Hereinafter, an example of a preferred embodiment for realizing the present invention will be described with reference to the drawings.
FIG. 1 shows a conceptual module configuration diagram of a configuration example of the present embodiment.
The module generally refers to components such as software (computer program) and hardware that can be logically separated. Therefore, the module in the present embodiment indicates not only a module in a computer program but also a module in a hardware configuration. Therefore, the present embodiment also serves as an explanation of a computer program, a system, and a method for functioning as those modules. However, for the convenience of explanation, the words “store”, “store”, “save”, and equivalents thereof are used, but these words are stored in the storage device when the embodiment is a computer program. It is the control to be stored or stored in the storage device. Modules may correspond to functions one-to-one, but in mounting, one module may be configured by one program, or a plurality of modules may be configured by one program, and conversely, one module May be composed of a plurality of programs. The plurality of modules may be executed by one computer, or one module may be executed by a plurality of computers in a distributed or parallel environment. Note that one module may include other modules. Hereinafter, “connection” is used not only for physical connection but also for logical connection (data exchange, instruction, reference relationship between data, etc.). “Predetermined” means that the process is determined before the target process, and not only before the process according to the present embodiment starts but also after the process according to the present embodiment starts. If it is before the target processing, it is used in accordance with the situation / state at that time or with the intention to be decided according to the situation / state up to that point.
In addition, the system or device is configured by connecting a plurality of computers, hardware, devices, and the like by communication means such as a network (including one-to-one correspondence communication connection), etc., and one computer, hardware, device. The case where it implement | achieves by etc. is included. “Apparatus” and “system” are used as synonymous terms. Of course, the “system” does not include a social “mechanism” (social system) that is an artificial arrangement.
In addition, when performing a plurality of processes in each module or in each module, the target information is read from the storage device for each process, and the processing result is written to the storage device after performing the processing. is there. Therefore, description of reading from the storage device before processing and writing to the storage device after processing may be omitted. Here, the storage device may include a hard disk, a RAM (Random Access Memory), an external storage medium, a storage device via a communication line, a register in a CPU (Central Processing Unit), and the like.

In the following, a description will be given mainly by exemplifying an image obtained by photographing a cell as an object, a nucleated red blood cell as a cell to be extracted, and a cell as an image. In addition, extracting or detecting cells from an image means extracting (cutting out) an image of the cells from the image.
With the development of cell handling technology, DNA diagnosis is performed by sorting out specific cells, and behavior of drugs is analyzed by introducing and observing specific molecules in cells. . For example, it is expected that fetal nucleated red blood cells contained in trace amounts in the blood of pregnant women will be extracted and subjected to DNA diagnosis to enable noninvasive fetal DNA diagnosis without risk to the mother and infant. In order to take out such specific cells, a person in charge of inspection usually searches for cells visually using a microscope. For example, the proportion of nucleated red blood cells derived from a fetus is about 1/8 to the power of 10, and this is visually searched while scanning the field of view of a microscope.

The image processing apparatus according to the present embodiment detects specific cells to be collected, and controls the stage so that the detected cells automatically come to the center of the field of view of the microscope. As shown, the sample manipulation module 110 and the image recognition module 150 are provided.
The sample operation module 110 controls the microscope, handles the sample, and acquires an image. The microscope includes an objective lens and is connected to an image acquisition module 116 that is an imaging device.
The sample manipulation module 110 includes a cell collection module 112, a sample ID reading module 114, an image acquisition module 116, and an electric stage 118.

The electric stage 118 is connected to the stage control module 170, and moves the position of the specimen under the control of the stage control module 170. Specifically, for example, what is called an XY stage performs a specimen positioning operation. Note that the specimen is obtained by applying a liquid containing cells (for example, blood) to a slide glass.
The image acquisition module 116 is connected to the image reception module 154 and acquires a specimen image including specific cells magnified by a microscope. Specifically, for example, a digital camera provided in a microscope is applicable, the state before the target cells are removed by the cell recovery module 112 is taken as an image before cell recovery, and the state after the target cells are removed is shown. Take an image after cell recovery.

The sample ID reading module 114 is connected to the sample ID receiving module 152. An information image is given to the specimen. The information image includes an ID (IDentification) for uniquely identifying the specimen, the type of examination, etc. (here, “specimen ID, examination type, etc.” includes “specimen ID, examination type only”). Is described). The sample ID reading module 114 reads the sample ID, the type of examination, and the like from this information image. The information image refers to an image code that is systematically created to represent electronic data in a machine-readable manner, and specifically includes a one-dimensional bar code, a two-dimensional code, and the like. In particular, a QR code (Quick Response code) has recently been used as a two-dimensional code.
The cell collection module 112 is connected to the image difference extraction module 156, the specific cell detection module 158, and the pre-cell collection image database 160, and collects target cells individually. Specifically, for example, the target cells are sucked and collected through a small-diameter nozzle. One or more target cells may be collected. In addition, the cell collection module 112 has a function of notifying the image recognition module 150 (specifically, the image reception module 154, the image difference extraction module 156, the specific cell detection module 158, etc.) that the collection has been completed.

  The image recognition module 150 has a function of specifying the position of the target cell from the image. The sample ID reception module 152, the image reception module 154, the image difference extraction module 156, the specific cell detection module 158, and before cell collection. An image database 160, a positive example database 162, a negative example database 164, a discriminator learning module 166, an identification model storage module 168, and a stage control module 170 are included.

The image reception module 154 is connected to the image acquisition module 116, the image difference extraction module 156, the specific cell detection module 158, and the pre-cell recovery image database 160, and receives an image from the image acquisition module 116. As an image, a cell-pre-collection image (first image) obtained by photographing a plurality of cells and a first cell (for example, the above-mentioned nucleated red blood cell, etc.) that is at least one of the cells are cells. The post-cell recovery image obtained by photographing the state removed by the recovery module 112 is received.
The sample ID reception module 152 is connected to the sample ID reading module 114, the image difference extraction module 156, the specific cell detection module 158, and the pre-cell recovery image database 160. Accept.

  The pre-cell recovery image database 160 is connected to the cell recovery module 112, the sample ID reception module 152, the image reception module 154, and the image difference extraction module 156, and the image before the cells are recovered by the cell recovery module 112 is sampled. Save with ID, inspection type, stage position. That is, the image before the cells received by the image receiving module 154 are collected (image before cell collection), the sample ID in the information image given to the photographed sample received by the sample ID receiving module 152, the examination The position of the electric stage 118 at the time of photographing, such as the type of the image, is stored in the pre-cell recovery image database 160 in correspondence with each other.

The image difference extraction module 156 is connected to the cell collection module 112, the specimen ID reception module 152, the image reception module 154, the pre-cell collection image database 160, and the positive example database 162, and is stored in the pre-cell collection image database 160. A difference image is extracted by comparing the pre-cell recovery image and the post-cell recovery image (second image), and stored in the positive example database 162. Details of the processing by the image difference extraction module 156 will be described later. Since the target cell exists in the pre-cell recovery image and does not exist in the post-cell recovery image, the difference image (target cell image) is used as a positive learning image.
The positive example database 162 is connected to the image difference extraction module 156 and the discriminator learning module 166, and stores the target cell image extracted by the image difference extraction module 156. That is, the difference image from the image difference extraction module 156 is stored as a positive example learning image. Further, the stored learning image of the positive example is passed to the classifier learning module 166.

The discriminator learning module 166 is connected to the positive example database 162, the negative example database 164, and the identification model storage module 168, and stores the cell images stored in the positive example database 162 as positive examples and stored in the negative example database 164. Learn the negative cell image as a negative example. Specifically, for example, whether each cell is desired to be identified from the positive learning image acquired by the image difference extraction module 156 and the negative learning image acquired by the specific cell detection module 158 described later. Learn a two-class identification model that is not. As an identification model, for example, a Haar-like feature is used and an Ada-Boost discriminator is used. However, the feature amount and the discriminator are not limited to this combination.
The identification model storage module 168 is connected to the specific cell detection module 158 and the classifier learning module 166, and stores the identification model learned by the classifier learning module 166 for each type of examination.

The specific cell detection module 158 is connected to the cell collection module 112, the specimen ID reception module 152, the image reception module 154, the negative example database 164, the identification model storage module 168, and the stage control module 170, and the target cell detection module 158 Detect cells. That is, the target cell is detected based on the identification model stored in the identification model storage module 168 that is the result learned by the classifier learning module 166. Specifically, for example, the type of the target cell to be specified is switched according to the type of examination from the specimen ID reception module 152, and the model corresponding to the cell to be identified stored in the identification model storage module 168 with respect to the pre-cell recovery image. The target cells are detected based on the above.
The stage control module 170 is connected to the electric stage 118 and the specific cell detection module 158. When the target cell is detected by the specific cell detection module 158, the stage control module 170 is electrically operated so that the detected target cell is at the center of the image. The stage 118 is controlled to move. Here, the center of the image is the so-called center of the visual field, and the visual field is the field of view of the operator who operates the microscope to collect the cells, and may be the center of the pre-cell recovery image or the post-cell recovery image. If it is displayed on a monitor or the like, it may be the center of the monitor.

The specific cell detection module 158 may detect the target cell from the post-cell recovery image. Since the target cell does not exist in the image after cell recovery, if a cell is detected, it is a false detection. Therefore, the image detected at this time is stored in the negative example database 164 as a negative example for learning.
The negative example database 164 is connected to the specific cell detection module 158 and the discriminator learning module 166, and stores the image of the target cell detected from the post-cell recovery image by the specific cell detection module 158. That is, the false detection image from the specific cell detection module 158 is stored as a negative example learning image. Further, the stored learning image of the negative example is passed to the discriminator learning module 166.

FIG. 2 is a flowchart showing an example of processing according to this embodiment. In particular, a procedure for learning an identification model for cell detection is shown.
In step S202, the specimen is loaded on the stage and an image is acquired. That is, the sample is loaded on the electric stage 118 according to the operator's instruction, and the microscope image of the sample is acquired by the image acquisition module 116. Usually, since the field of view of the microscope is narrow, the entire specimen cannot be covered, and the stage control module 170 moves the stage and acquires a plurality of images.

In step S204, the pre-cell recovery image is stored. The image acquired in step S202 is received by the image receiving module 154 and stored in the pre-cell recovery image database 160 in correspondence with the position of the electric stage 118 at the time of imaging, such as the specimen ID and the type of examination.
In step S206, target cell detection processing is performed. The specific cell detection module 158 performs target cell detection processing on the acquired image. The detection process is performed using, for example, an identification model learned in advance for each type of examination with respect to an image of a rectangular area of 20 pixels × 20 pixels in the image to determine whether the rectangular area is a target cell. . By scanning this rectangular area over the entire image (for example, 2000 pixels × 2000 pixels), target cells in the image are detected. Note that this detection process may be performed in advance on all samples (meaning all samples to be processed) by batch processing and the results may be stored. The rectangular area of 20 pixels × 20 pixels is an example of a predetermined rectangular area, but the size of the rectangular area may be changed according to the size of the cell to be detected.

In step S208, it is determined whether or not the target cell is detected. If it is detected, the process proceeds to step S212. Otherwise (the detection accuracy is low and the target cell is not detected), the process proceeds to step S210.
In step S210, the target cell is searched visually. That is, the operator operates the electric stage 118 to search for the target cell, and the target cell is set as the detected target cell.
In step S212, the stage is moved. That is, the stage control module 170 controls the electric stage 118 and moves the stage so that the detected cell is at the center of the image.

In step S214, target cells are collected. That is, the cell collection module 112 collects the detected target cells, and the collected cells are used for, for example, a DNA test.
In step S216, an image after cell recovery is acquired. That is, the cell collection module 112 notifies the image recognition module 150 that the target cell has been collected, and the image reception module 154 acquires an image after cell collection using the notification signal as a trigger.

In step S218, a difference image is extracted. That is, the image difference extraction module 156 compares the post-cell recovery image acquired in step S216 with the pre-cell recovery image at the same specimen ID and the same stage position by a method described later, and extracts a differential image.
In step S220, the positive example is stored. That is, since the difference image extracted in step S218 is an image of a cell to be detected, it is stored in the positive example database 162 as a positive example image for learning together with the specimen ID, the type of examination, and the like.

In step S222, target cell detection processing is performed. That is, the specific cell detection module 158 performs target cell detection processing (processing equivalent to step S206) again on the image after cell recovery. At this time, since the target cell does not already exist in the image after cell recovery, if the target cell is detected, it is erroneously detected.
In step S224, it is determined whether or not the target cell has been detected. If it is detected (so-called erroneous detection), the process proceeds to step S226, and otherwise, the process proceeds to step S228.
In step S226, a negative example is stored. That is, since the image detected in step S222 is an erroneously detected image, the image is stored in the negative example database 164 as a negative example image for learning together with the sample ID, the type of examination, and the like.

In step S228, it is determined whether or not the increment of the learning sample is greater than or equal to a predetermined value. If it is greater than or equal to the predetermined value, the process proceeds to step S230. Otherwise, the process ends (step S299). That is, if the increment of the learning sample from the previous learning process is greater than or equal to the specified value, the process proceeds to step S230.
In step S230, the identification model is learned. That is, the discriminator learning module 166 uses the positive example database 162 and the negative example database 164 to learn the identification model.

FIG. 3 is a conceptual module configuration diagram of a configuration example in the image reception module 154 and the image difference extraction module 156.
The image difference extraction module 156 includes a global alignment processing module 312, a cell presence candidate area extraction module 314, a cell movement candidate area extraction module 316, a displacement vector calculation module 318, an image deformation module 320, and a target extraction module 322. Yes.

The image reception module 154 is connected to the global alignment processing module 312, receives the pre-cell recovery image 301 and the post-cell recovery image 302, and passes both images to the global alignment processing module 312. The pre-cell recovery image 301 is an image of a specimen (a first image obtained by photographing a plurality of cells, an image before cell recovery), and the post-cell recovery image 302 is an image after the target cell is recovered (from the pre-cell recovery image 301). A second image obtained by photographing the state in which the target cells are removed, an image after cell recovery).
FIG. 5 is an explanatory diagram showing an example of an image 301 before cell recovery and an image 302 after cell recovery.
In the pre-cell recovery image 301 illustrated in FIG. 5A, six cells (cells 501 to 506) are photographed. Among these, the target cell is the cell 502. The post-cell recovery image 302 illustrated in FIG. 5B shows a state after the cell 502 is recovered, and five cells (cell 511, cell 513, cell 514, cell 515, cell 516) are photographed. ing. Cell 511, cell 513, cell 514, cell 515, and cell 516 correspond to cell 501, cell 503, cell 504, cell 505, and cell 506 in the pre-cell recovery image 301, respectively.
Comparing the two images, nucleated red blood cells (cells 502) are not recovered in the post-cell recovery image 302. Then, as a recovery operation of the nucleated red blood cells (cells 502), the stripping solution is injected around the cells to be recovered so that the cells can be easily detached. In this case, the red blood cells (cells 501) adjacent to the nucleated red blood cells (cells 502) have moved to the position of the cells 511 in the post-cell recovery image 302.
Moreover, the position of each cell has shifted | deviated uniformly. This occurs due to a stage misalignment, for example, when images before cell collection are taken by batch processing for a plurality of specimens and then cell collection is performed. As described above, since the pre-cell recovery image 301 and the post-cell recovery image 302 have a global positional shift or a local positional shift, the region of the nucleated red blood cell (cell 502) cannot be extracted by simple difference processing.

  The global alignment processing module 312 is connected to the image reception module 154 and the cell presence candidate region extraction module 314, receives the pre-cell recovery image 301 and the post-cell recovery image 302 from the image reception module 154, and receives the pre-cell recovery image. The post-cell recovery image 302 is moved so that the cells captured in 301 and the cells captured in the post-cell recovery image 302 are aligned. That is, the global positional deviation that occurs in the entire image is corrected. Specifically, for example, the probability distribution of luminance of both images is calculated, and the mutual information amount of both images is monitored using the probability distribution. That is, one image is translated by a predetermined amount, and the mutual information amount between the two images is calculated. Then, the parallel movement amount that maximizes the mutual information amount is searched. The result of translating the post-cell recovery image 302 so that the mutual information amount is maximized is the global post-alignment image 601 illustrated in FIG. The cells 611 and 613 to 616 in the image 601 after global alignment correspond to the cells 511 and 513 to 516 in the image 302 after cell recovery. This alignment is called global alignment. Although the post-cell recovery image 302 is illustrated as an image to be moved, the pre-cell recovery image 301 may be moved. Of course, the direction in which the pre-cell recovery image 301 is moved is opposite to that in the post-cell recovery image 302.

The cell presence candidate region extraction module 314 is connected to the global alignment processing module 312 and the cell movement candidate region extraction module 316, and the post-cell recovery image 302 moved by the global alignment processing module 312 (FIG. 6). A region in which the target cell may exist is extracted based on the post-global registration image 601) and the pre-cell recovery image 301 illustrated in FIG.
Specifically, for example, a difference between the post-global positioning image 601 after global positioning by the global positioning processing module 312 and the pre-cell recovery image 301 is extracted, and an area having a large absolute value is selected as a candidate area. A. Candidate area A is an area where the recovered cells may exist. FIG. 7 is an explanatory diagram illustrating a processing example of the cell presence candidate region extraction module 314. The difference image 701 includes a candidate area A 711 and a candidate area B 712. In this difference processing, an image that is present in the pre-cell recovery image 301 but not in the global post-alignment image 601 is extracted. In addition, although the case where the image moved by the global alignment processing module 312 is the post-cell recovery image 302 is illustrated, the pre-cell recovery image 301 is moved by the global alignment processing module 312. Based on the obtained pre-cell recovery image 301 and post-cell recovery image 302, an area where the target cell may exist is extracted. In this case, the difference process is to extract the pre-cell recovery image 301 that has been subjected to the global alignment, but not the post-cell recovery image 302. Hereinafter, similarly, the case where the image moved by the global alignment processing module 312 is the post-cell recovery image 302 will be exemplified. However, when the pre-cell recovery image 301 is moved, the global alignment is performed. Is the same processing except that it is in the opposite direction to that illustrated (that is, based on the position of the cell in the post-cell recovery image 302).

The cell migration candidate region extraction module 316 is connected to the cell presence candidate region extraction module 314 and the displacement vector calculation module 318, and removes the target cell based on the candidate region A extracted by the cell presence candidate region extraction module 314. An area where the second cell, which is a cell that may have moved, may have moved is extracted.
Specifically, for example, an area within a predetermined distance from the candidate area A is set as the candidate area B. Candidate area B is an area where cells may have moved. This within the predetermined distance is in accordance with the above-mentioned definition of “predetermined”. In this case, however, it is an area where the stripping solution diffuses. For example, it can be obtained by calculating fluid dynamics from the amount and viscosity of the stripping solution. Also good.
This will be described more specifically. FIG. 8 is an explanatory diagram showing a processing example of the cell movement candidate region extraction module 316. The centers (may be the center of gravity) of candidate area A 711 and candidate area B 712 are calculated, and circles 821 and 822 having radii that are predetermined distances from calculated centers 811 and 812 are generated, and the circles are generated. An area surrounded by 821 and 822 may be set as a candidate area B 830.
Another example will be described. FIG. 9 is an explanatory diagram showing a processing example of the cell movement candidate region extraction module 316. A region within a predetermined distance from the periphery of the candidate region A 711 and the candidate region B 712 may be set as the candidate region B 930.

The displacement vector calculation module 318 is connected to the cell movement candidate region extraction module 316 and the image deformation module 320, and the second cell movement amount and the region extracted by the cell movement candidate region extraction module 316 are set as targets. The moving direction is calculated. That is, the amount of cell movement relative to the candidate area B is estimated. This is done as follows.
An image having no cells photographed in advance is set as a background image. Then, the image has a luminance different from that of the background image of the post-cell recovery image (post-global registration image 601), and the candidate region B (for example, candidate region B 830) in the pre-cell recovery image 301 and the post-global registration image 601. , For example, a local area of 10 pixels × 10 pixels is displaced by S1 (x) by a predetermined amount Δx from the coordinates x in the pre-cell recovery image 301. Let S0 (x + Δx) be the local region at the coordinates, and calculate the mutual information amount of S1 (x) and S0 (x + Δx). Then, the maximum amount of parallel movement in which the mutual information amount is equal to or greater than a predetermined value is searched, and the movement amount and the movement direction are calculated. This is called a local displacement vector at the coordinate x of the post-cell recovery image.
FIG. 10 is an explanatory diagram illustrating a processing example of the displacement vector calculation module 318. The local displacement vector 1031 takes a non-zero value only at the position of the cell that has been displaced due to the collection operation (the cell 1011 before movement is displaced to the cell 1012 after movement). Note that the collected cell portion has the same luminance as the background, so the local displacement vector is not calculated.

The image deformation module 320 is connected to the displacement vector calculation module 318 and the object extraction module 322, and after global alignment based on the movement amount and movement direction of the second cell calculated by the displacement vector calculation module 318. The image 601 is transformed into an image in a state before the movement of the second cell. Specifically, for example, the cells in the candidate region B in the post-global alignment image 601 are moved using the movement amount and the movement direction. When the movement direction calculated by the displacement vector calculation module 318 is the direction before and after the movement, the movement direction is opposite to the movement direction. When the movement direction is the direction after the movement and before the movement, the movement direction is calculated. Move in the same direction as.
For example, a post-local alignment image 1101 illustrated in FIG. 11 is obtained by deforming an image using a local displacement vector. This post-local alignment image 1101 is a modification of the global post-alignment image 601 illustrated in FIG. In the post-local registration image 1101, the cell 611 in the global registration image 601 illustrated in FIG. 6 moves to the cell 1111 in the same position as the cell 501 in the pre-cell recovery image 301 illustrated in FIG. It has moved, and the corrected cells have been corrected. Of course, the cells 613 to 616 in the global post-alignment image 601 illustrated in FIG. 6 correspond to the cells 1113 to 1116 in the post-local alignment image 1101 illustrated in FIG. There is no deformation, so it remains in the same position.

The target extraction module 322 is connected to the image deformation module 320 and extracts target cells based on the difference between the image deformed by the image deformation module 320 and the pre-cell recovery image 301. Specifically, for example, the difference between the pre-cell recovery image 301 and the post-local alignment image 1101 is taken, and an area where the absolute value of the difference is equal to or greater than a predetermined value is extracted as an area where the collected target cells existed. .
FIG. 12 is an explanatory diagram illustrating a processing example of the target extraction module 322. Mask areas 1211 to 1213 are extracted as areas where target cells exist. Black areas (mask area 1211 and mask area 1213) also exist in other areas due to noise, deformation of moved cells, and the like, but these areas exclude areas having an area equal to or smaller than a predetermined value. This is possible.
An image of the target cell is acquired by extracting an image corresponding to the region (mask region 1212) determined as described above from the pre-cell recovery image 301. Then, in order to make this a positive example for learning by the classifier learning module 166, it is stored in the positive example database 162.

FIG. 4 is a flowchart illustrating an example of processing performed by the image reception module 154 and the image difference extraction module 156.
In step S402, the image receiving module 154 receives the pre-cell recovery image 301 and the post-cell recovery image 302.
In step S404, the global alignment processing module 312 performs global alignment of the pre-cell recovery image 301 and the post-cell recovery image 302.
In step S406, the cell presence candidate area extraction module 314 extracts an area where the recovered cells may exist.
In step S408, the cell movement candidate area extraction module 316 extracts an area around the area extracted in step S406 and possibly having a cell moved.
In step S410, the displacement vector calculation module 318 calculates a displacement vector of the moved cell.
In step S412, the image deformation module 320 deforms the post-cell recovery image 302 using the displacement vector.
In step S414, the target extraction module 322 takes the difference between the pre-cell recovery image 301 and the transformed post-cell recovery image 302, and extracts the recovered cells.

  A hardware configuration example of the image recognition module 150 of the present embodiment will be described with reference to FIG. The configuration shown in FIG. 13 is configured by a personal computer (PC), for example, and shows a hardware configuration example including a data reading unit 1317 such as a scanner and a data output unit 1318 such as a printer. This hardware configuration is also applied to the sample operation module 110.

  A CPU (Central Processing Unit) 1301 describes the execution sequence of each module such as the image difference extraction module 156, the specific cell detection module 158, and the classifier learning module 166 described in the above embodiment. It is a control part which performs the process according to a computer program.

  A ROM (Read Only Memory) 1302 stores programs used by the CPU 1301, calculation parameters, and the like. A RAM (Random Access Memory) 1303 stores programs used in the execution of the CPU 1301, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus 1304 including a CPU bus.

  The host bus 1304 is connected to an external bus 1306 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 1305.

  A keyboard 1308 and a pointing device 1309 such as a mouse are input devices operated by an operator. The display 1310 includes a liquid crystal display device or a CRT (Cathode Ray Tube), and displays various types of information as text or image information.

  An HDD (Hard Disk Drive) 1311 has a built-in hard disk, drives the hard disk, and records or reproduces a program executed by the CPU 1301 and information. The hard disk stores an image before cell collection, an image after cell collection, an image of a cell as a positive example, an image of a cell as a negative example, an identification model, and the like. Further, various computer programs such as various other data processing programs are stored.

  The drive 1312 reads data or a program recorded on a removable recording medium 1313 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and reads the data or program into an interface 1307 and an external bus 1306. , The bridge 1305, and the RAM 1303 connected via the host bus 1304. The removable recording medium 1313 can also be used as a data recording area similar to a hard disk.

The connection port 1314 is a port for connecting the external connection device 1315 and has a connection unit such as USB and IEEE1394. The connection port 1314 is connected to the CPU 1301 and the like via the interface 1307, the external bus 1306, the bridge 1305, the host bus 1304, and the like. A communication unit 1316 is connected to a network and executes data communication processing with the outside. Communication between the sample operation module 110 and the image recognition module 150 is performed via the connection port 1314 or the communication unit 1316.
The data reading unit 1317 is, for example, a scanner, and executes document reading processing. The data output unit 1318 is, for example, a printer, and executes document data output processing.

  Note that the hardware configuration illustrated in FIG. 13 illustrates one configuration example, and the present embodiment is not limited to the configuration illustrated in FIG. 13, and is a configuration that can execute the modules described in the present embodiment. I just need it. For example, some modules may be configured with dedicated hardware (for example, Application Specific Integrated Circuit (ASIC), etc.), and some modules are in an external system and connected via a communication line Alternatively, a plurality of systems shown in FIG. 13 may be connected to each other via communication lines so as to cooperate with each other. Further, the sample manipulation module 110 and the image recognition module 150 are not separate and may be configured as one system.

  In the above-described embodiment, the cell is shown as an example of the object to be detected from the image. However, in addition to this, for example, the defective product when the defective product is removed in the manufacturing process at the factory may be the target. The sold product when the product is sold in the store may be the target. That is, in addition to the target (defective product, product, etc.), the image A in which a plurality of objects equivalent to the target (including not only the same shape but also different shapes, patterns, and colors) are captured, and the target It may be applied to detecting an object by comparing the image B taken after the image is removed. Note that the image B is an image of a state in which there is a possibility that the surrounding objects may have moved to remove the target.

The program described above may be provided by being stored in a recording medium, or the program may be provided by communication means. In that case, for example, the above-described program may be regarded as an invention of a “computer-readable recording medium recording the program”.
The “computer-readable recording medium on which a program is recorded” refers to a computer-readable recording medium on which a program is recorded, which is used for program installation, execution, program distribution, and the like.
The recording medium is, for example, a digital versatile disc (DVD), which is a standard established by the DVD Forum, such as “DVD-R, DVD-RW, DVD-RAM,” and DVD + RW. Standard “DVD + R, DVD + RW, etc.”, compact disc (CD), read-only memory (CD-ROM), CD recordable (CD-R), CD rewritable (CD-RW), Blu-ray disc ( Blu-ray Disc (registered trademark), magneto-optical disk (MO), flexible disk (FD), magnetic tape, hard disk, read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM), flash Includes memory, random access memory (RAM), etc. .
The program or a part of the program may be recorded on the recording medium for storage or distribution. Also, by communication, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a wired network used for the Internet, an intranet, an extranet, etc., or wireless communication It may be transmitted using a transmission medium such as a network or a combination of these, or may be carried on a carrier wave.
Furthermore, the program may be a part of another program, or may be recorded on a recording medium together with a separate program. Moreover, it may be divided and recorded on a plurality of recording media. Further, it may be recorded in any manner as long as it can be restored, such as compression or encryption.

DESCRIPTION OF SYMBOLS 110 ... Sample operation module 112 ... Cell collection module 114 ... Sample ID reading module 116 ... Image acquisition module 118 ... Electric stage 150 ... Image recognition module 152 ... Sample ID reception module 154 ... Image reception module 156 ... Image difference extraction module 158 ... Specific Cell detection module 160 ... Image database before cell recovery 162 ... Positive example database 164 ... Negative example database 166 ... Classifier learning module 168 ... Identification model storage module 170 ... Stage control module 301 ... Image before cell recovery 302 ... Image after cell recovery 312 ... global alignment processing module 314 ... cell existence candidate area extraction module 316 ... cell movement candidate area extraction module 318 ... displacement vector calculation module 320 ... image deformation module Joule 322 ... Object extraction module

Claims (4)

Receiving means for receiving a first image obtained by photographing a plurality of objects and a second image obtained by photographing a state in which the first object, which is an object included in at least one of the plurality of objects, is removed;
Based on the result of collating the first image and the second image so that the object photographed in the first image and the object photographed in the second image are matched, the First region extraction means for extracting a region where the first object may exist;
An area in which a second object that may have moved when the first object is removed based on the first area extracted by the first area extracting means may have moved Second region extracting means for extracting
Calculation means for calculating a movement amount and a movement direction of the second object for the area extracted by the second area extraction means;
Image deformation means for deforming the second image into an image in a state before the movement of the second object based on the movement amount and movement direction of the second object calculated by the calculation means;
An image processing apparatus comprising: extraction means for extracting the first object based on a difference between the image deformed by the image deformation means and the first image. First storage means for storing an image of the first object extracted by the extraction means;
Detecting means for detecting the first object from the second image;
Second storage means for storing an image of the first object detected by the detection means;
Learning means for learning the first object image stored in the first storage means as a positive example, and learning the first object image stored in the second storage means as a negative example;
The image processing apparatus according to claim 1, wherein the detection unit detects the first object based on a result learned by the learning unit. The object is a cell, and the image is a microscope image;
The detection means detects a first cell as a target that is the first object from the first image,
Control means for controlling the microscope stage to move so that the first cell is in the center when the first cell is detected by the detection means;
A state before the first cell is removed is photographed as a first image, and the state after the first cell is removed after the microscope stage is moved by the control means. A photographing means for photographing as an image of 2,
The image processing apparatus according to claim 2, wherein the reception unit receives the first image and the second image captured by the imaging unit. On the computer,
A reception function for receiving a first image obtained by photographing a plurality of objects and a second image obtained by photographing a state in which the first object that is at least one of the plurality of objects is removed;
Based on the result of collating the first image and the second image so that the object photographed in the first image and the object photographed in the second image are matched, the A first region extraction function for extracting a region in which the first object may exist;
An area where a second object, which may have moved when the first object is removed, may have moved based on the first area extracted by the first area extracting function A second region extraction function for extracting
A calculation function for calculating a movement amount and a movement direction of the second object for the area extracted by the second area extraction function;
An image transformation function for transforming the second image into an image in a state before the movement of the second object based on the movement amount and movement direction of the second object calculated by the calculation function;
An image processing program that realizes an extraction function for extracting the first object based on a difference between an image deformed by the image deformation function and the first image.

Images