JP4444928B2 - Image processing method, apparatus, and program for microscope image - Google Patents

Description

  The present invention relates to an image processing apparatus and method for displaying an image by stopping the movement of a subject of interest or suppressing the movement.

In the field of image processing, several methods are known for realizing camera shake correction and subject tracking functions of video cameras. In order to correct camera shake, for example, as described in Patent Document 1, vibration of the camera itself is detected using a gyroscope, and the optical system is displaced so as to cancel the vibration. There is a method to shake the image. Furthermore, Patent Document 2 discloses an image processing that allows a subject tracked by a photographer to always be output to the center of the screen, and an image processing that does not cause blurring during playback of an image that has been blurred during input. Is described. According to the method of Patent Document 2, a reference point on a subject is set, the amount of deviation of the reference point is detected in each image frame, the entire screen is moved by the amount of deviation, and the subject's screen is detected. Correct the position inside. The deviation amount is detected by pattern matching.
Japanese Examined Patent Publication No. 56-21133 JP 11-187303 A

  For example, when observing a living cell with a biological microscope, the cell to be observed moves under the influence of the movement of other cells around it. In recent years, in biological research, it has become important to obtain biological image data as time-sequential data as it is and analyze its dynamic parameters. Unlike cells in cultured cell systems, the scaffold in which the cells to be observed adhere or the environment in which the cells are placed changes and moves in vivo. For this reason, it is difficult to record the spontaneous movement of one cell in a living body as easy-to-see and meaningful data, not just a moving image.

  The present invention provides an apparatus and a method for measuring a deviation amount between frames constituting a digital moving image at high speed and correcting the deviation amount.

  The present invention is a step of extracting the nth image and the (n + 1) th image from an image stack depicting a continuous movement composed of N digital images (N is an integer of 2 or more) stored in a storage device. A step in which n is an integer greater than or equal to 1 and less than or equal to N−1, and a first rectangular rectangle that partitions M squares vertically and horizontally at corresponding positions in the n th image and the n + 1 th image. A first grid setting step for performing an operation of placing each grid, wherein M is a multiple of 2 including 2, and one of the intersection points of the first grid of the nth image and the first of the n + 1th image A superposition step of superimposing the nth image and the n + 1th image so that the center of one grid comes, and a predetermined color for the overlapping pixel or the corresponding pixel between the nth image and the n + 1th image The difference of the sex is obtained, the variance of the difference is calculated, and this calculation is performed for all the intersections of the first grid of the nth image, thereby calculating the intersection of the first grid that minimizes the variance. A second grid for performing an operation of placing a second grid similar to the first grid, having a variance minimizing step, and having an area of 1 squared M of the first grid around the determined intersection An installation step, a regression step of repeating the dispersion minimization step and the second grid installation step a predetermined number of times, considering the second grid as the first grid in the variance minimization step, and the regression step are completed The position of the intersection at which the variance obtained from time to time with respect to the center of the grid 1 in the nth image is the amount of deviation between the nth image and the (n + 1) th image. A shift amount determination step for performing calculation, a correction step for performing a calculation for changing the relative position of the nth image and the (n + 1) th image corresponding to the shift amount, and from the first grid setting step to the correction step And a step of repeating the above while incrementing the value of n.

  Further, the present invention is a step of obtaining an n-th image and an (n + 1) -th image in which a continuous motion is digitally depicted from a movable imaging device, wherein n is an integer greater than or equal to 1 and less than or equal to N−1; A first grid setting step for performing an operation of placing a first grid of rectangles that divide M square planes at the corresponding positions of the nth image and the (n + 1) th image, wherein M is 2. The nth image and the n + 1th image are superimposed so that the center of the first grid of the (n + 1) th image comes to one of the intersections of the first grid of the nth image and the step that is a multiple of 2 A predetermined color characteristic difference between the overlapping step and the nth image and the (n + 1) th image with respect to the overlapping pixel or the corresponding pixel is obtained, a variance of the difference is calculated, and this calculation is performed on the nth image. The first of A variance minimizing step for performing an operation for determining the intersection of the first grid that minimizes the variance by performing the calculation for all the intersections of the lid, and the square of M of the first grid around the determined intersection A second grid installation step for performing an operation of placing a second grid similar to the first grid, and sequentially considering the second grid as the first grid in the variance minimization step , A regression step that repeats the variance minimization step and the second grid setting step a predetermined number of times, and the position of the intersection that minimizes the variance obtained when the regression step is completed with respect to the center of the grid 1 in the nth image Is a shift amount determination step for calculating the shift amount between the nth image and the (n + 1) th image, and a movable imaging device corresponding to the shift amount. Move the also provides a subject tracking image processing method comprising a correction step of performing tracking of the object.

  These methods according to the present invention can be carried out by using one of any color characteristics such as brightness, hue, and saturation, and by combining a plurality of them. Which color characteristic is used can be determined by the characteristics of the subject to be tracked and the characteristics of the background. In addition to the case of using a set of a plurality of grids having an area where M is 4 and 25 intersections are successively reduced to 1/16, M is 2 , A method of using a grid set with 9 intersections and sequentially becoming a quarter size, M is 6 and the number of intersections is 49, and the size is successively 36 times smaller There is also a method that uses a set of grids.

  Further, the present invention includes a computer program including instructions for causing a computer to perform such a method, in addition to an apparatus for performing these methods.

  According to the method or apparatus of the present invention, it is possible to quantitatively measure and correct a shift between digital images, that is, a movement of an environment to which cells adhere. Therefore, it becomes possible to eliminate the influence of the scaffold to which the cell adheres or its environment, which becomes an obstacle when providing the movement of the cell to be observed itself in the time-series image of the living body.

  In addition to biological systems such as cells, the present invention can be used for tracking observation of illegal intruders and animals.

  FIG. 1 shows the flow of image processing according to an embodiment of the method of the present invention. First, a series of images is obtained as a result of acquiring two-dimensional images continuously over time. Such an image can be obtained by a device in which a video camera is attached to a confocal laser microscope or an optical microscope, or more generally by a video camera. The series of two-dimensional images are stacked to form a three-dimensional image stack (S100), and can be stored in an appropriate storage means such as a hard disk, RAM, or flash memory.

  Two-dimensional images are taken out one by one from such a three-dimensional image stack (S102), and the amount of deviation in the biaxial direction from the previous two-dimensional image is calculated by the computing means (S104). Based on the calculated shift amount, a correction process is performed on the two-dimensional image (S106). That is, correction is performed by moving the image with respect to the center of the screen so as to eliminate the shift amount.

  The amount of deviation at this time is calculated as follows. This will be described with reference to FIG. First, an image immediately before is set as an image 1 and an image whose deviation is to be corrected is set as an image 2.

  Then, grid 1 is placed on image 1. This grid 1 is divided into four equal parts vertically and horizontally, and is composed of 10 line segments, and divides 16 faces. The size of the grid 1 may be the same as the size of the image 1 or may be smaller than the image 1. The position where the grid 1 is placed may be the center of the image 1 or may be placed at an appropriate position in the image 1 so as to cover the subject to be tracked. The grid 1 may have a rectangular shape as well as a square shape.

  Next, grid 1 is placed at the same position in image 2. The grid 1 of the image 2 is indicated by a broken line. What is important in the image 2 is the center point of the grid 1, and the whole image of the grid 1 is useful in some cases, but is not always essential.

  Further, a rectangular grid 2 in which the height and width of the grid 1 are reduced to 1/4 is created. The grid 2 is a grid that divides the vertical and horizontal directions into four, similar to the grid 1. That is, the grid 2 has a similar shape having an area that is 1/16 of the grid 1. This grid 2 is indicated by a solid thin line at the center of the grid 1 of the image 1.

  This grid 2 has 25 intersections, like the grid 1. Then, the image 2 is superimposed on the image 1 without rotating the entire image so that the center of the grid 1 of the image 2 comes to one of the 25 intersections. In this state, the brightness difference between the image 1 and the image 2 is calculated. This difference calculation can be performed for all the pixels constituting each image and overlapping between image 1 and image 2, or for the remaining pixels that are thinned out based on a certain rule. It is. It is also possible to calculate this difference for pixels that overlap between the grid 1 of image 1 and image 2. That is, the difference calculation can be performed over the entire overlap of the image 1 and the image 2, or can be performed for a partial overlap of the image 1 and the image 2. Then, the variance of this difference is calculated for all differences. This calculation is repeated by sequentially shifting the center of the grid 1 of the image 2 to each of the 25 intersections.

  At this time, other color characteristics can be used instead of brightness. A color is composed of hue, brightness, and saturation. Hue represents color (redness, yellowness, greenness, blueness, purpleness, etc.). Brightness represents lightness and darkness when viewed as an achromatic color, and saturation represents color intensity. is there. Any of these color characteristics can be used in the practice of the present invention. Also, RGB notation for expressing a color as a combination of three colors of red (R), green (G), and blue (B), and other color notation methods can be used. There are color characteristics that are unique to the subject to be tracked, that is, color characteristics that are different from the color characteristics of surrounding objects (for example, the subject is particularly bright or the surroundings are bluish but the subject has redness). If so, the subject can be tracked more reliably by using the color characteristics.

  In this way, the position of the intersection point at which the variance value calculated for each of the 25 intersection points is minimized is determined. In FIG. 2, the point where the value of the variance is minimum is (x, y) = (1, 1) in the grid 2 with the origin of the grid 1 as the center.

  Next, with the origin of (x, y) = (1, 1) of the grid 2, the width and height are one-fourth of the grid 2, and the grid 2 is partitioned into 16 planes like the grid 2. Create a grid 3. Then, in the same manner as performed for the grid 2, the center point of the grid 1 of the image 2 is placed at each of the 25 intersections of the grid 3, and the variance is sequentially calculated. As a result of the calculation, the position of the center of the grid 1 of the image 2 that minimizes the variance is obtained as the position of the intersection in the grid 3 of the image 1. In FIG. 2, the center point of the grid 1 of the image 2 is at the position (x, y) = (0, 0) of the grid 3.

  Then, the 16 divisions of the image area and the comparison are repeated, for example, until the grid line segment interval corresponds to one pixel of the digital image.

  In other words, a new rectangular grid that has an area that is 1/16 of the previous grid and is equally divided into 16 planes is placed at the center of the previous grid, and one intersection of the new grid. The center of the grid 1 of the image 2 is placed at. Then, a predetermined color characteristic difference between overlapping or corresponding pixels is obtained between the image 1 and the image 2, and the variance of the difference is calculated. By performing this calculation for all the intersection points of the new grid, the intersection point having the minimum variance is determined. Then, a rectangular grid having the next 1/16 area is placed around the intersection.

  Finally, the position where the image 2 with the minimum variance is to be placed is determined based on the image 1. This is performed for all the remaining images in the three-dimensional image stack. That is, assuming that the total number of images is N, a shift that minimizes the above dispersion is obtained for images n from n + 1 = 2 to n + 1 = N.

  With reference to FIG. 3, the deviation calculation method according to the present invention will be summarized. That is, the nth image and the (n + 1) th image are taken out from the image stack depicting the continuous movement composed of N digital images (N is an integer of 2 or more) stored in the storage device (S200). Here, n is an integer greater than or equal to 1 and less than or equal to N−1, and a rectangular first grid that divides 16 planes of 4 vertically and horizontally at corresponding positions of the nth image and the n + 1th image. Each is placed (S202).

  The nth image and the (n + 1) th image are superimposed so that the center of the first grid of the (n + 1) th image comes to one of the intersections of the first grid of the nth image (S204). A difference in predetermined color characteristics is obtained between the nth image and the n + 1th image for the overlapping pixel or the corresponding pixel, and a variance of the difference is calculated (S206), and this calculation is performed for the nth image. The intersection of the first grid that minimizes the variance is determined by performing the process for all the intersections of one grid (S208).

  Then, a second grid having an area which is 1/16 of the first grid and centering on the determined intersection is placed (S210). The second grid is sequentially considered as the first grid (of the nth image) in the variance minimization step, and the variance minimization step and the second grid installation step are repeated a predetermined number of times (S212). The position of the intersection at which the variance obtained when the regression step is completed with respect to the center of the grid 1 in the nth image is the amount of deviation between the nth image and the (n + 1) th image ( S214).

  As described above, the case where the size of the next grid is 1/16 has been described. However, the size of the next grid can be set to 1/4, or can be set to 1/36. Is possible.

  The deviation calculation method of the present invention will be examined with reference to FIG. Here, the size of each image is 512 pixels both vertically and horizontally. The first grid 1 is 128 pixels wide and high. The calculation is performed by placing the center of the grid 1 of the image 2 at each of the 25 intersections of the grid 1 of the image 1. In the example shown in FIG. 4, the variance of the difference between the predetermined color characteristics of the image 1 and the image 2 is minimized at the point (x, y) = (1, 1) of the grid 1. Considering pixel units, (x, y) = (32, 32). Then, a similar grid 2 having a vertical and horizontal size of 1/4 (that is, 32 pixels) is placed around (1, 1) of the grid 1. Then, with respect to the grid 2, the dispersion is minimized at (x, y) = (1, −1). In pixel units, (40, 24).

  Similarly, a grid 3 having a similar shape with a vertical and horizontal size of 1/4 (that is, 8 pixels) is placed at the position (1, −1) of the grid 2. Then, the intersection of the grids 3 where the variance is further minimized is obtained. In FIG. 4, it is the (−1, −1) position of the grid 3. When expressed in pixels, (38, 22) is obtained. Next, the size of the grid is 2 pixels square, and the grid interval is less than 1 pixel. Therefore, the grid 4 is a grid with a 1-pixel interval that divides the surface into four. Since this grid 4 has nine intersections, the variance of the color characteristic difference is obtained for each intersection, and the point at which it is minimized is obtained. As a result, (38, 23) is the deviation to be obtained.

  The number of comparisons between images in this series of operations is 25 + 25 + 25 + 9 = 84. If an attempt is made to obtain a brute force shift for all the pixels of grid 1, 128 × 128 = 16384 comparisons must be made. Therefore, according to the method of the present invention, the calculation amount can be reduced to about 1/195.

  A second embodiment of the present invention will be described with reference to FIG. Image processing is performed in real time using a movable imaging device such as a microscope having an xy motorized stage and a camera mounted on a drive turntable in two directions such as up and down and left and right. Images are sequentially input from the movable imaging device. When the nth and n + 1th images are input, a shift of a specific subject in the n + 1th image with respect to the nth image is detected. The detection of the deviation is as described in the first embodiment. Then, the movable imaging device is moved in accordance with the detected amount of the deviation to prepare for photographing the n + 2th or n + 3th image. This is continued until shooting is completed. As a result, it is possible to obtain image data obtained by tracking a specific subject.

  The recorded image data can be read arbitrarily, and the image represented by the data can be displayed on a display device such as a CRT (Cathode Ray Tube) or printed by a predetermined printing device, or It can be taken into a personal computer or the like and subjected to predetermined processing.

It is a flowchart of a 1st embodiment of the present invention. It is a conceptual diagram which shows the structure of the grid used in 1st Embodiment of this invention. It is a flowchart which shows the calculation method of the deviation | shift in 1st Embodiment of this invention. It is a conceptual diagram which shows the structure of the grid for verifying the calculation method of deviation | shift in 1st Embodiment of this invention. It is a flowchart of 2nd Embodiment of this invention.

Claims (5)

Extracting an nth image and an (n + 1) th image from an image stack depicting successive movements of N digital images (N is an integer greater than or equal to 2) stored in a storage device, wherein n Is an integer greater than or equal to 1 and less than or equal to N−1;
a first grid setting step for performing an operation of placing a rectangular first grid that partitions M squares in vertical and horizontal positions at corresponding positions of an nth image and an (n + 1) th image, where M is 2 A step that is a multiple of 2 including
a superposition step of superimposing the n + 1 image and the n + 1 image so that the center of the first grid of the n + 1th image comes to one of the intersections of the first grid of the nth image;
A difference of a predetermined color characteristic between the nth image and the (n + 1) th image with respect to an overlapping pixel or a corresponding pixel is obtained, a variance of the difference is calculated, and this calculation is performed on the first grid of the nth image. A variance minimizing step for performing an operation to determine the intersection of the first grid that minimizes the variance by performing for all intersections;
And a second grid installation step for performing an operation of placing a second grid similar to the first grid, having an area of 1 square of M of the first grid with the determined intersection as the center,
The second grid is considered to be the first grid of the nth image in the variance minimizing step, and the variance minimizing step and the second grid setting step are set as the interval between the line segments of the first grid. A regression step that repeats a predetermined number of times until it corresponds to one pixel of the image ;
An operation in which the position of the intersection at which the variance obtained when this regression step is completed with respect to the center of the grid 1 in the nth image is the amount of deviation between the nth image and the (n + 1) th image A deviation amount determining step for performing
A correction step for performing an operation of changing a relative position between the nth image and the (n + 1) th image in accordance with the amount of deviation;
The image processing method of the microscope image until correction step from the first grid placed step comprising the steps of repeated while incrementing the value of n. Obtaining a n-th image and an (n + 1) -th image in which a continuous motion is digitally depicted from a movable imaging device, wherein n is an integer greater than or equal to 1 and less than or equal to N−1;
a first grid setting step for performing an operation of placing a rectangular first grid that partitions M squares in vertical and horizontal positions at corresponding positions of an nth image and an (n + 1) th image, where M is 2 A step that is a multiple of 2 including
a superposition step of superimposing the n + 1 image and the n + 1 image so that the center of the first grid of the n + 1th image comes to one of the intersections of the first grid of the nth image;
A difference of a predetermined color characteristic between the nth image and the (n + 1) th image with respect to an overlapping pixel or a corresponding pixel is obtained, a variance of the difference is calculated, and this calculation is performed on the first grid of the nth image. A variance minimizing step for performing an operation to determine the intersection of the first grid that minimizes the variance by performing for all intersections;
And a second grid installation step for performing an operation of placing a second grid similar to the first grid, having an area of 1 square of M of the first grid with the determined intersection as the center,
The second grid is considered to be the first grid of the nth image in the variance minimizing step, and the variance minimizing step and the second grid setting step are set as the interval between the line segments of the first grid. A regression step that repeats a predetermined number of times until it corresponds to one pixel of the image ;
An operation in which the position of the intersection at which the variance obtained when this regression step is completed with respect to the center of the grid 1 in the nth image is the amount of deviation between the nth image and the (n + 1) th image A deviation amount determining step for performing
A microscopic image processing method comprising: a correction step of moving the movable imaging device in accordance with the amount of deviation and tracking the subject. A storage device for storing an image stack depicting successive movements of N (N is an integer greater than or equal to 2) digital images;
The nth image and the (n + 1) th image are extracted from the stored image stack, where n is an integer greater than or equal to 1 and less than or equal to N−1, and the nth image and the (n + 1) th image are in corresponding positions. First grid installation means for performing an operation of placing rectangular first grids that define M squares each in length and breadth, wherein M is a multiple of 2 including 2;
superimposing means for superposing the nth image and the n + 1th image so that the center of the first grid of the n + 1th image comes to one of the intersections of the first grid of the nth image;
A difference of a predetermined color characteristic between the nth image and the (n + 1) th image with respect to an overlapping pixel or a corresponding pixel is obtained, a variance of the difference is calculated, and this calculation is performed on the first grid of the nth image. Dispersion minimizing means for performing an operation to determine the intersection of the first grid that minimizes the dispersion by performing all intersections;
And a second grid installation step for performing an operation of placing a second grid similar to the first grid, having an area of 1 square of M of the first grid with the determined intersection as the center,
The second grid is considered to be the first grid of the nth image in the variance minimizing step, and the variance minimizing step and the second grid setting step are set as the interval between the line segments of the first grid. A regression means that repeats a predetermined number of times until it corresponds to one pixel of the image ;
An operation in which the position of the intersection at which the variance obtained when this regression step is completed with respect to the center of the grid 1 in the nth image is the amount of deviation between the nth image and the (n + 1) th image Deviation amount determining means for performing
Correction means for performing an operation of changing the relative position of the nth image and the n + 1th image in accordance with the amount of deviation;
The image processing device of the microscopic images until correction step from the first grid placed steps comprising means for repeated while incrementing the value of n. A movable imaging device that obtains an n-th image and an (n + 1) -th image in which continuous motion is digitally depicted;
1st grid installation means for performing an operation of placing a rectangular first grid that divides M square planes at corresponding positions of the nth image and the (n + 1) th image, where M is 2. Means that are multiples of 2 including:
superimposing means for superposing the nth image and the n + 1th image so that the center of the first grid of the n + 1th image comes to one of the intersections of the first grid of the nth image;
A difference of a predetermined color characteristic between the nth image and the (n + 1) th image with respect to an overlapping pixel or a corresponding pixel is obtained, a variance of the difference is calculated, and this calculation is performed on the first grid of the nth image. Dispersion minimizing means for performing an operation to determine the intersection of the first grid that minimizes the dispersion by performing all intersections;
And a second grid installation step for performing an operation of placing a second grid similar to the first grid, having an area of 1 square of M of the first grid with the determined intersection as the center,
The second grid is considered to be the first grid of the nth image in the variance minimizing step, and the variance minimizing step and the second grid setting step are set as the interval between the line segments of the first grid. A regression means that repeats a predetermined number of times until it corresponds to one pixel of the image ;
An operation in which the position of the intersection at which the variance obtained when this regression step is completed with respect to the center of the grid 1 in the nth image is the amount of deviation between the nth image and the (n + 1) th image Deviation amount determining means for performing
A microscopic image processing apparatus, comprising: correction means for moving the movable imaging device in accordance with the amount of deviation and tracking the subject. The program for making a computer perform each step of the method of Claim 1 or 2.

Images