JP2015031599A - Image superposing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To superpose two images derived by photographing two similar samples one on top of another simply and accurately.SOLUTION: Provided is an image superposing method including: an image acquisition step S1 of acquiring a first image and a second image in which a first object and a second object having a mutually similar shape are displayed in equal size, respectively; a contour acquisition step S2 of acquiring the contour of the object displayed in each of the first image and the second image; a center of gravity detection step S3 of detecting the center of gravity of each contour; a superposition step S4 of superposing the first image and the second image one on top of another so that the centers of gravity of the contours match each other; and a rotation step S5 of rotating the first image and the second image centering on the matched center of gravity relative to each other and adjusting the relative angles of these images.

Description

  The present invention relates to an image superimposing method.

  Conventionally, a technique for aligning two images when two images obtained by photographing the same sample or two similar samples are overlapped is known (for example, see Patent Documents 1 to 3). In Patent Document 1, a fluorescent or luminescent image and a transmitted light image of a sample are photographed using the same optical system in a state where the sample is held in a holding portion provided with an optical identification mark, and the image is photographed in each image. An optical identification mark is used as a reference for alignment. In Patent Document 2, an image of a sample with an artificial marker attached to a measurement region is obtained using a mass spectrometer and an optical microscope, and the marker in the obtained image is used as a reference for alignment. In Patent Document 3, when a specimen from which a section is cut is embedded in a resin, a columnar marker is also embedded in the resin together with the specimen, the specimen is sliced perpendicularly to the marker, and the marker in the created section is aligned. It is used as a standard.

  On the other hand, a method of collecting a fragment of a specific region in a section by cutting a section attached on a substrate such as a cover glass together with the substrate is known (see, for example, Patent Document 4). In Patent Document 4, one of two sections cut out from a position close to a specimen of a living tissue is used for cutting and the other is used for staining. Since the two sections have similar tissue morphology, by referring to the stained section, the tissue morphology at each position in the other section is estimated, and an area to be collected such as a cancerous area Can be specified.

Japanese Patent No. 3766421 JP 2010-85219 A JP 2013-83474 A International Publication No. 2011/149909

  In Patent Document 4, in order to accurately specify a fragment of a section to be collected, a technique for superimposing images of these sections so that the two sections are accurately matched is required. However, all of Patent Documents 1 to 3 are inconvenient because a process of attaching a marker to a sample is required. Further, since Patent Documents 1 and 2 are techniques related to two images obtained by photographing the same sample, application to Patent Document 4 is difficult. Patent Document 3 relates to two images obtained by photographing two sections having similar tissue forms, as in Patent Document 4. However, when the marker is embedded obliquely with respect to the slice slicing direction, the positional relationship between the segment and the marker is shifted for each segment, so that the two images cannot be accurately superimposed on the basis of the marker. There's a problem.

  The present invention has been made in view of the above-described circumstances, and an object thereof is to provide an image superimposing method capable of easily and accurately superimposing two images obtained by photographing two similar samples. To do.

In order to achieve the above object, the present invention provides the following means.
The present invention is a method for superimposing a first image obtained by photographing a first object and a second image obtained by photographing a second object having a shape similar to the first object. The image acquisition step for acquiring the first image and the second image in which the first object and the second object are displayed at the same magnification, and the image acquired in the image acquisition step A contour acquisition step of acquiring a contour of the object displayed in each of the first image and the second image, and a centroid detection step of detecting a centroid of each of the contours acquired in the contour acquisition step; A superimposing step of superimposing the first image and the second image so that the centroids of the contours detected in the centroid detection step coincide with each other, and the centroid matched in the superimposition step Said first The image and the second image are relatively rotated to provide a method by superimposing images and a rotating step of adjusting the relative angle of the image.

According to the present invention, the contour of each object in the first image and the second image obtained in the image acquisition step is acquired in the contour acquisition step, and then the centroid of each contour is detected in the centroid detection step. Then, the two images are superimposed in the superimposing step so that the centers of gravity coincide with each other, and then the relative angle between the two images centered on the center of gravity is adjusted.
This makes it possible to accurately align two objects having similar but not identical contours. Moreover, the marker for alignment of two objects is not required, and two images can be superimposed only by a simple process using a normal image.

In the above invention, in the rotating step, the relative angle may be adjusted to an angle that maximizes an overlapping area of two regions surrounded by the contours.
In this way, it is possible to evaluate the alignment accuracy of the two objects using only simple calculations, and determine the relative angle at which the positions of the two objects most closely match.

Further, the above invention may include a translation step of adjusting the relative position of the first image and the second image by relatively translating the first image and the second image after the rotation step.
In this way, when the center of gravity of the contour of the target object is biased due to, for example, part of one target object deforming, the bias of the center of gravity position is corrected by translating the two images. The alignment accuracy of the two objects can be further increased.

Further, in the above invention, in the parallel movement step, the relative position is set at a position where the ratio of the overlapping area of the two regions to the average of the area of the two regions surrounded by the contours is maximized. May be adjusted.
In this way, it is possible to evaluate the alignment accuracy of the two objects using only simple calculations, and to determine the relative position where the positions of the two objects most closely match.

Further, the above invention may include a re-rotation step that finely adjusts the relative angle between the first image and the second image by rotating the first image and the second image around the center of gravity after the parallel movement step. Good.
By doing in this way, as a result of performing the parallel movement process, there is a possibility that an optimum relative angle is generated more than the relative angle determined in the rotation process. Therefore, by finely adjusting the relative angle between the two images, the alignment accuracy of the two objects can be further increased.

  According to the present invention, there is an effect that two images obtained by photographing two similar samples can be easily and accurately superimposed.

It is a flowchart which shows the image superimposition method which concerns on the 1st Embodiment of this invention. It is the 1st image and 2nd image which are used in the image superposition method of FIG. It is a figure explaining a superposition process. It is a figure explaining a rotation process. It is a flowchart which shows the image superimposition method which concerns on the 2nd Embodiment of this invention.

(First embodiment)
Hereinafter, an image superimposing method according to the first embodiment of the present invention will be described with reference to FIGS.
The image superimposing method according to the present embodiment includes a first image obtained by photographing a first object and a second image obtained by photographing a second object having a shape similar to that of the first object. As shown in FIG. 1, an image acquisition step S1 for acquiring first and second images, a contour acquisition step S2 for acquiring the contour of an object in each image, and each contour as shown in FIG. A center of gravity detection step S3 for detecting the center of gravity of the image, a superposition step S4 for superimposing the first and second images on the basis of the center of gravity, and a rotation step S5 for relatively rotating the first and second images around the center of gravity. including.

  FIG. 2 shows an example of the first image G1 and the second image G2 used in the present embodiment. In this embodiment, two sections 1 and 2 cut out from a specimen of a living tissue are assumed as two objects. The two sections 1 and 2 are cut out from positions adjacent to the specimen, preferably adjacent positions, and have substantially the same shape and tissue form. Each of the two sections 1 and 2 is attached to a separate substrate such as a cover glass. One section (first section) 1 is cut into small pieces together with the substrate, and a part of the selected pieces is subjected to genetic testing or the like. The other section (second section) 2 is subjected to pathological staining.

  Here, the image superimposing method according to the present embodiment is used to select fragments. That is, the first image G1 obtained by photographing the first slice 1 and the second image G2 obtained by photographing the second slice 2 by the image superimposing method according to the present embodiment, the first slice 1 and By superimposing the second section 2 so as to coincide with each other, the operator identifies a region of interest (for example, a cancerous region) in the second section 2, and a position corresponding to the identified region of interest. Can be selected from the first section 1.

  Note that the image superimposing method according to the present embodiment is not limited to the images of the slices 1 and 2 and can be applied to any image as long as it is an image of two objects having similar outer shapes. can do.

Next, each step S1 to S5 of the image superimposing method according to the present embodiment will be described in detail.
The image superimposing method according to the present embodiment is realized as an image processing application program executed on a computer such as a personal computer, for example. That is, a CPU (Central Processing Unit), a main storage device such as a RAM (Random Access Memory), an auxiliary storage device such as a ROM (Read Only Memory), an input device such as a keyboard and a mouse, and a display device. In the computer provided, the CPU executes each step S1 to S5 by reading out and executing a program stored in advance in the auxiliary storage device to the main storage device.
Alternatively, the superimposing method according to the present embodiment may be realized by an image processing apparatus including a unit that executes each of steps S1 to S5.

  First, in the image acquisition step S1, the first image G1 and the second image G2 in which the slices 1 and 2 are displayed at the same magnification are read out from, for example, the auxiliary storage device. These images G1 and G2 are obtained, for example, by photographing the first section 1 and the second section 2 at the same magnification using the same photographing apparatus. If the images G1 and G2 are obtained by photographing the first section 1 and the second section 2 at different magnifications, the two sections are displayed so that the sections 1 and 2 are displayed at the same magnification. At least one of the images G1 and G2 is enlarged or reduced according to the photographing magnification.

  In the image acquisition step S1, image processing may be appropriately performed on the images G1 and G2. For example, one of the two slices 1 and 2 may be attached to the substrate with the front and back opposite to each other, and as a result, the slices 1 and 2 may be inverted in the first image G1 and the second image G2. . In such a case, a process of inverting the image may be performed on one of the two images G1 and G2. Furthermore, in order to facilitate the subsequent steps S2 to S5, the brightness, contrast, gamma curve, and the like of each of the images G1 and G2 may be adjusted, and an effect may be given to the images G1 and G2.

  Next, in the contour acquisition step S2, the contours A and B of the slices 1 and 2 are extracted from the images G1 and G2 by image processing. As the image processing, known methods such as binarization processing of gradation values and differentiation processing are used.

  In the contour acquisition step S2, the contours A and B may be acquired by manual input by an operator instead of image processing. For example, the operator draws a Bezier curve using the input device along the contours of the segments 1 and 2 in the images G1 and G2 displayed on the display device, and acquires the drawn Bezier curves as the contours A and B. May be. Alternatively, the operator may trace the contours of the slices 1 and 2 in the images G1 and G2 displayed on the touch panel using a stylus pen to obtain the contours A and B, and touch pad, pointing Devices such as sticks and mice may be used. Further, the image processing and the manual input by the operator may be combined, and the contours A and B extracted by the image processing may be corrected by the operator manually.

  Next, in the centroid detection step S3, the positions of the centroids C1 and C2 are calculated by calculating the average of the coordinates of the pixels constituting the contours A and B. Here, in calculating the centroids C1 and C2, it is not always necessary to use the coordinates of all the pixels constituting the contours A and B, and some of the coordinates may be thinned out in order to reduce the amount of calculation.

Next, in the superimposition step S4, as shown in FIG. 3, the second image G2 is placed on the first image G1 so that the centroids C1 and C2 detected by the centroid detection step S3 coincide with each other. Overlapping.
In FIGS. 3 and 4, the first image G1 and the first slice G1, and the first slice G1 and the second slice 2 are easily identified. 1 is indicated by a broken line.

Next, in the rotation step S5, of the two images G1 and G2 superimposed so that the centroids C1 and C2 coincide with each other, the second image G2 is centered on the centroids C1 and C2 at a predetermined angle. Rotate within the range, preferably 360 °. Then, as shown in FIG. 4, the rotation angle of the second image G2 with respect to the first image G1 is adjusted to an angle at which the coincidence rate is maximized. The match rate is defined by the following equation.
Match rate = 2R / (r1 × r2)
Here, r1 is the area of the region surrounded by the contour A, r2 is the area of the region surrounded by the contour B, and R is the region where the two regions surrounded by the contours A and B overlap each other (FIG. 3). And in FIG. 4, the area is a hatched area. R does not exceed the smaller area of r1 and r2. For this reason, the maximum value of the matching rate is 1 or less depending on the values of r1 and r2 or the form of the first image G1 and the second image G2.

  Specifically, for example, the coincidence rate at each rotation angle is calculated while rotating the second image G2 in units of 10 °, and the rotation angle at which the calculated coincidence rate is maximum is determined. Thereafter, the second image G2 may be further rotated in units of 1 ° within a range of 9 ° before and after the determined rotation angle, and similarly the rotation angle with the maximum matching rate may be determined. By doing so, the overlay accuracy can be improved. The unit of the rotation angle is arbitrary, and may be 0.5 °, for example, or may be smaller.

  During the rotation step S5, the two contours A and B and the two superimposed images G1 and G2 are separately displayed so that the operator can easily recognize the overlapping state of the contours A and B. Alternatively, auxiliary information may be displayed. For example, a region where two regions surrounded by the outlines A and B overlap each other may be displayed in a predetermined color. Furthermore, a graphic such as a numerical value or a bar graph showing the matching rate may be displayed. The coincidence rate is preferably set to a value exceeding 0 and a maximum of 1 so as to be easily understood by an operator so that the matching rate is within a certain range regardless of the superimposed images. Further, instead of the matching rate, the value of the area R may be presented to the operator as it is.

  In the rotation step S5, the operator may input the rotation angle of the second image G2 using an input device, and rotate the first image G1 by image processing by the input rotation angle. Further, the first image G1 may be rotated with respect to the second image G2.

  In the rotation step S5, when a sufficiently high coincidence rate is obtained at a plurality of rotation angles, the plurality of rotation angles are presented to the operator as candidates, and one of the presented candidates is selected. The operator may select it. The coincidence rate is a value determined based only on the overall shape of the two contours A and B, and the tissue form in the region surrounded by the contours A and B and the local shapes of the contours A and B It does not consider features. Therefore, when the contours A and B have shapes that are substantially symmetrical with respect to the centers of gravity C1 and C2, the coincidence rate is high even when the second image G2 is rotated, for example, by 180 ° from an appropriate rotation angle. . When the contours A and B are substantially circular, even if the second image G2 is rotated around the center of gravity C2, the coincidence rate does not change much. Therefore, when a high coincidence rate is obtained at a plurality of rotation angles, the operator may select an optimal rotation angle based on the tissue morphology of the two sections 1 and 2.

  By the above procedure, the first image G1 and the second image G2 can be overlaid so that the first slice 1 and the second slice 2 coincide. In this case, according to the image superposition method according to the present embodiment, the centroids C1 and C2 of the contours A and B of the segments 1 and 2 in the images G1 and G2 are detected, and two segments around the centroids C1 and C2 are detected. By adjusting the relative angles 1 and 2, the two images G1 and G2 can be overlaid so that the two segments 1 and 2 can be accurately matched. Further, since the markers for aligning the two slices 1 and 2 are not necessary and the normal images G1 and G2 can be used, special members and processing are not necessary, and simple image processing and Only the calculation process is sufficient, and there is an advantage that the images G1 and G2 can be superimposed in a short time.

(Second Embodiment)
Next, an image superimposing method according to the second embodiment of the present invention will be described with reference to FIG.
As shown in FIG. 5, the image superposition method according to the present embodiment is different from the first embodiment in that it further includes a parallel movement step S6 and a re-rotation step S7 after the rotation step S5. Therefore, in the present embodiment, these two steps S6 and S7 will be mainly described, and the same reference numerals are given to the configurations common to the first embodiment, and the description will be omitted.

  In the translation step S6, the second image G2 is translated with respect to the first image G1, the coincidence rate is calculated at each position, and the second image G2 is positioned at a position where the coincidence rate is maximized. To do. At this time, it is preferable to set in advance a range in which the second image G2 is translated.

  For example, the horizontal direction of the first image G1 is defined as the X axis and the vertical direction is defined as the Y axis. Then, the second image G2 is translated in the X-axis direction within a distance range of 5% of the maximum length of the contour A in the X-axis direction with reference to the position where the centroids C1 and C2 coincide. The unit of the movement amount is preferably 1 pixel, but may be about 2 to 10 pixels in order to reduce the calculation amount. Similarly, the second image G2 is translated in the Y-axis direction within a distance range of 5% of the maximum length of the contour A in the Y-axis direction with reference to the position where the centroids C1 and C2 coincide. Then, the coincidence rate is calculated for all combinations of the respective positions in the X-axis direction and the respective positions in the Y-axis direction, and the position where the coincidence rate is maximized is determined.

  In the translation step S6, similarly to the rotation step S5 described above, when a sufficiently high coincidence rate is obtained at a plurality of positions, the plurality of positions are presented to the operator as candidates, and the presented candidates One may be selected by the operator.

  The re-rotation step S7 is performed in the same manner as the rotation step S5 described above. However, since the relative angle between the first image G1 and the second image G2 has already been adjusted once in the rotation step S5, in this step S7, the rotation step is performed within an angle range narrower than that of the rotation step S5. It is preferable to rotate the second image G2 by a unit smaller than S5.

  The thin sections 1 and 2 may be distorted when the thin sections 1 and 2 are attached to the substrate or the edges may be bent, whereby the outlines A and B may be deformed. When such deformation has occurred, the positions of the centers of gravity C1 and C2 deviate from the original positions. As a result, in the superimposition step S4, the two images G1 and G2 deviate from their original relative positions. Is superimposed. According to the present embodiment, the shift of the positions of the centroids C1 and C2 is corrected by the translation step S6, and the two images G1 and G2 are overlaid so that the two segments 1 and 2 are more accurately matched. it can.

  Furthermore, as a result of performing the parallel movement step S6, an angle that is more optimal than the relative angle determined in the rotation step S5 may be generated. According to this embodiment, the overlay accuracy of the two images G1 and G2 can be further increased by adjusting the relative angle between the two images G1 and G2 again by the re-rotation step S7.

  In the present embodiment, the result of superimposing the two images G1 and G2 obtained in each of the rotation step S5, the parallel movement step S6, and the re-rotation step S7, and the angle or position candidate, together with the matching rate are used. And after the re-rotation step S7, all the overlay results may be presented to the operator together with the matching rate, and the operator may select the final overlay result.

1, 2 sections (object)
A, B Contour C1, C2 Center of gravity G1, G2 Image S1 Image acquisition process S2 Contour acquisition process S3 Center of gravity detection process S4 Overlay process S5 Rotation process S6 Parallel movement process S7 Re-rotation process

Claims (5)

A method of superimposing a first image obtained by photographing a first object and a second image obtained by photographing a second object having a shape similar to the first object,
An image acquisition step of acquiring the first image and the second image in which the first object and the second object are displayed at the same magnification;
A contour acquisition step of acquiring a contour of the object displayed in each of the first image and the second image acquired in the image acquisition step;
A centroid detection step of detecting the centroid of each of the contours acquired in the contour acquisition step;
A superimposing step of superimposing the first image and the second image so that the centroids of the contours detected in the centroid detection step coincide with each other;
An image superimposing method including a rotating step of relatively rotating the first image and the second image around the center of gravity matched in the superimposing step to adjust a relative angle between the images.   The image superimposing method according to claim 1, wherein, in the rotation step, the relative angle is adjusted to an angle at which an overlapping area of two regions surrounded by the contours is maximized.   3. The image superposition according to claim 1, further comprising a translation step of translating the first image and the second image relative to each other to adjust a relative position between the first image and the second image after the rotation step. How to match.   In the parallel movement step, the relative position is adjusted to a position where the ratio of the overlapping area of the two regions to the average of the average of the areas of the two regions surrounded by the contours is maximized. The image overlay method described.   5. The method according to claim 3, further comprising a re-rotation step after the parallel movement step, wherein the first image and the second image are rotated relative to each other about the center of gravity to finely adjust a relative angle between these images. The image overlay method described.

Images