JP5209100B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus capable of displaying a moving image generated by a camera that captures an inspection object on a mosaic image having a wider field of view than the actual field of view of the camera.

An image mosaic is conventionally known as a technique for creating a single wide-field image having a wider field of view than the actual field of view of the camera by pasting together a plurality of still images taken while changing the field of view. In the case of an imaging apparatus such as a digital microscope that photographs a subject magnified by an objective lens, the field of view can be changed by moving a movable stage on which an inspection target is placed. The wide-field image is called a mosaic image, and is created by connecting a plurality of still images captured while changing the field of view based on the relative positions between the images.

For example, in the case of a conventional imaging device that includes a sensor that detects the position of the movable stage and automatically captures an imaging range designated by the user, the relative positional relationship between the images is determined from the control information of the movable stage, The still images are joined together. In such an imaging apparatus, once the shooting range is designated and shooting is started, the shooting range cannot be changed midway. Further, since it is necessary to detect the position of the movable stage with high accuracy, there is a problem that the system configuration becomes complicated and the cost is increased.

On the other hand, there is also an imaging device that determines the relative positional relationship between images by pattern matching between images and performs still image joining. However, there has been no imaging apparatus that can capture a still image and connect it to the mosaic image while allowing the user to confirm the positional relationship between the field of view being photographed and the mosaic image being created on the display. In view of this, an imaging device that captures a still image at a timing instructed by a user and displays it on the mosaic image while displaying the field of view as a moving image on the mosaic image is conceivable. In this imaging device, the image size of the mosaic image changes due to the pasting, so that a part of the mosaic image is not displayed on the screen, and the positional relationship between the field of view being shot and the mosaic image being created is not understood. There was a problem.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an imaging apparatus that can easily recognize the positional relationship between a field of view being shot and a mosaic image being created. This is especially true when the image size of the mosaic image changes greatly before and after stitching when the still image is captured and pasted to the mosaic image while displaying the field of view as a moving image at an appropriate position on the mosaic image. Even if it exists, it aims at providing the imaging device which can display the whole mosaic image for a display on a screen.

An imaging apparatus according to the present invention includes a movable stage on which an object to be inspected can be placed and movable in a plane that intersects the central axis of the objective lens, and the movable stage that is disposed to face the movable stage. By combining two or more images having overlapping areas obtained by changing the field of view of the camera by moving the movable stage and moving the movable stage and shooting different positions of the inspection object, Mosaic image generation means for generating a mosaic image having a wider field of view than the actual field of view of the camera, feature quantity extraction means for extracting a feature quantity from the mosaic image and a frame image newly generated by the camera, and the feature quantity By comparing the relative position determination means for determining the relative position between the mosaic image and the frame image, the relative position determination When the relative position between the mosaic image and the frame image is successfully determined by the means and a new mosaic image in which the mosaic image and the frame image are connected is generated by the mosaic image generation means, the new image is generated. A mosaic image storage means for storing a mosaic image, and a display mosaic image generated by reducing the mosaic image stored in the mosaic image storage means for storage so that the entire mosaic image is displayed on the screen. Reduction means, display mosaic image storage means for storing the display mosaic image reduced by the display reduction means, and the display mosaic image is read from the display mosaic image storage means. And image display means for displaying the whole.

In this imaging device, the field of view being captured is displayed as a moving image composed of display frame images at an appropriate position on the display mosaic image, and the still image captured by the camera is captured and pasted onto the mosaic image. Thus, a new mosaic image is generated. Then, when generating the display frame image and the display mosaic image by reducing the frame image and the mosaic image, the image is reduced based on the image size of the mosaic image. According to such a configuration, since the image is reduced based on the image size of the mosaic image, even if the image size of the mosaic image changes greatly before and after the pasting, the entire display mosaic image is reduced. It can be displayed in the screen. Therefore, it is possible to easily recognize the positional relationship between the field of view being shot and the mosaic image being created.

Another imaging apparatus according to the present invention includes a movable stage that can be moved in two different directions with an inspection object placed thereon, and is arranged to face the movable stage. A camera that generates a moving image including a frame image, a mosaic image generating unit that generates a mosaic image having a wider field of view than the actual field of view of the camera by combining two or more still images captured by the camera, and the frame Relative position determination means for determining the relative position between the image and the mosaic image, and the frame image and the mosaic image are respectively reduced based on the image size of the mosaic image to generate a display frame image and a display mosaic image The display reduction means for displaying and the display mosaic of the display frame image based on the relative position determination result. It updates the display position on the image, and includes a live image display means for displaying a moving picture constituted by the display frame image on the display mosaic images. In addition to the above-described configuration, another imaging apparatus according to the present invention reduces the frame image and the mosaic image so that the display reduction unit displays the entire display mosaic image on the screen. Configured as follows. According to such a configuration, when displaying a moving image consisting of a display frame image on a display mosaic image as a live image, the entire mosaic image is always displayed in the screen, so that the field of view during shooting and creation The user can easily recognize the positional relationship with the mosaic image inside.

In addition to the above-described configuration, another imaging device according to the present invention may be configured such that the display reduction unit displays the frame image and the display frame image so that the entire display mosaic image and the entire display frame image are displayed simultaneously. It is configured to reduce the mosaic image. According to such a configuration, when the moving image composed of the display frame image is displayed as a live image on the display mosaic image, the entire mosaic image and the entire frame image are always displayed at the same time. The user can easily recognize the positional relationship between the visual field and the mosaic image being created.

Another imaging apparatus according to the present invention is configured such that, in addition to the above-described configuration, the display reduction unit reduces the frame image and the mosaic image while maintaining a constant aspect ratio of the image.

In addition to the above-described configuration, the imaging apparatus according to the present invention may further include the frame image and the mosaic image each time the display reduction unit updates the display position of the display frame image with respect to the display mosaic image. It is configured to update the reduction rate when reducing. According to such a configuration, every time the display position of the display frame image is updated, the reduction ratio when the frame image and the mosaic image are reduced is updated. Therefore, the display mosaic is made to follow the update of the display position. The entire image and the display frame image can be appropriately displayed on the screen.

In addition to the above configuration, another imaging apparatus according to the present invention has a region that overlaps with the mosaic image, and is recommended to display a frame image and a recommended moving area of the frame image that can determine the relative position between the mosaic images. Area display means, and the display reduction means is configured to reduce the frame image and the mosaic image so that the frame image and the mosaic image in the recommended movement area are simultaneously displayed. .

In addition to the above-described configuration, another imaging apparatus according to the present invention reduces the frame image and the mosaic image at a certain reduction ratio, respectively, and generates an alignment frame image and an alignment mosaic image. And a feature amount extracting unit that extracts a feature amount from the alignment frame image and the registration mosaic image, and the relative position determination unit compares the feature frame amount and the alignment frame image by comparing the feature amounts. A relative position between the mosaic images for alignment is determined, and the display reduction unit overlaps the entire mosaic image for display and a part of the mosaic image for alignment, and is necessary for determining the relative position. The entire display frame image corresponding to the above-described alignment frame image at an arbitrary position having an overlap area is displayed on the screen. As appears, configured to reduce the frame image and the mosaic image. According to such a configuration, since the relative position between these images is determined by comparing the feature amounts extracted from the alignment frame image and the alignment mosaic image, it is possible to prevent the system configuration from becoming complicated. Can do. In addition to the above configuration, the image pickup apparatus according to the eighth aspect of the present invention provides a reduction when the display reduction means reduces the frame image and the mosaic image when the image size of the mosaic image exceeds a certain size. Configured to fix rate. According to such a configuration, when the image size of the mosaic image exceeds a certain size, the reduction rate when the frame image and the mosaic image are reduced is fixed. It is possible to prevent the positional relationship with the mosaic image from being recognized.

According to the imaging apparatus according to the present invention, the image is reduced based on the image size of the mosaic image. Therefore, even if the image size of the mosaic image changes greatly before and after the pasting, the entire mosaic image for display is displayed. Can be displayed on the screen. Therefore, the entire display mosaic image is displayed in the screen when the still image is captured and pasted to the mosaic image while displaying the field of view as a moving image at an appropriate position on the mosaic image. It is possible to easily recognize the positional relationship between the field of view being shot and the mosaic image being created.

1 is a system diagram showing an example of a schematic configuration of an imaging apparatus according to an embodiment of the present invention, and a magnification observation apparatus 1 is shown as an example of the imaging apparatus. FIG. 2 is a block diagram illustrating a configuration example of a main part of the magnifying observation apparatus 1 in FIG. FIG. 3 is a block diagram illustrating a configuration example of a matching processing unit 125c in the system main body unit 100 of FIG. It is explanatory drawing which showed typically an example of the operation | movement at the time of the live image display in the magnification observation apparatus 1 of FIG. 1, and the moving image A1 and the mosaic image A3 for a display are shown. It is explanatory drawing which showed typically an example of the operation | movement at the time of the live image display in the magnification observation apparatus 1 of FIG. 1, and the live screen 111 is shown. It is the figure which showed an example of the pattern matching operation | movement in the magnification observation apparatus 1 of FIG. 1, and the mode that the positive correspondence between feature points is extracted by the comparison of all the feature points B3 is shown. It is the figure which showed an example of the operation | movement at the time of the live image display in the magnification observation apparatus 1 of FIG. 1, and the live screen 111 on which the mosaic image and the live image were displayed is shown. It is the figure which showed an example of the operation | movement at the time of the live image display in the expansion observation apparatus 1 of FIG. 1, and the mosaic image for a display updated by bonding a still image is shown. 3 is a flowchart showing an example of an operation at the time of capturing images for pasting in the magnification observation apparatus 1 in FIG. 1. It is the figure which showed an example of the operation | movement at the time of the live image display in the magnification observation apparatus by Embodiment 2 of this invention, and the live screen 111 on which the live image was displayed is shown. It is the block diagram which showed the structural example in the principal part of the magnification observation apparatus by Embodiment 4 of this invention, and shows an example of the function structure in the system main-body part 100a. It is the figure which showed an example of the operation | movement at the time of the live screen display in the enlarged observation apparatus of FIG. 11, and the movement recommendation area displayed on the mosaic image is shown.

Embodiment 1 FIG.
<Magnification observation device>
FIG. 1 is a system diagram illustrating an example of a schematic configuration of an imaging apparatus according to Embodiment 1 of the present invention. As an example of the imaging apparatus, a magnification observation apparatus 1 including a system main body unit 100, a camera unit 200, and a console 300 is illustrated. It is shown. The magnification observation apparatus 1 is a digital microscope that can capture a subject magnified by an objective lens, generate a moving image, and display the moving image on the display 110 of the system main body 100.

The camera unit 200 is an imaging unit for imaging an inspection object while changing the field of view, and includes a camera 210, a movable holder 220, and a movable stage 230. The camera 210 is a reading device that photographs an inspection object as a subject and generates a moving image composed of a plurality of frame images continuous at a constant frame rate, and includes an objective lens, a CCD image sensor, and an illumination in a cylindrical casing. It is configured by arranging devices and the like.

The movable holder 220 is a holding unit that holds the camera 210 so as to be movable in a direction parallel to the central axis of the objective lens. Here, the direction parallel to the central axis of the objective lens of the camera 210 is referred to as the z-axis direction, and the position of the camera 210 in the z-axis direction can be adjusted by turning the position adjustment knob 221.

The movable stage 230 is a holding unit that holds the inspection target, and is movable in a plane that intersects the z-axis with the inspection target placed thereon. Here, a plane perpendicular to the z-axis is referred to as an xy plane, and the position of the movable stage 230 in the xy plane can be adjusted by turning the position adjustment knobs 231 and 232. That is, the movable stage 230 is a stage that can be moved in two different directions while the inspection object is placed by turning the position adjustment knobs 231 and 232.

Specifically, the position in the x-axis direction can be adjusted by turning the position adjustment knob 231, and the position in the y-axis direction can be adjusted by turning the position adjustment knob 232. The camera 210 is arranged to face such a movable stage 230.

The console 300 is an input device for instructing the system main body unit 100 to start and end shooting and to take captured image data.

The system main unit 100 displays a moving image captured by the camera 210 on the display 110, and combines the frame images constituting the moving image to generate a mosaic image having a wider field of view than the actual field of view of the camera 210. It is.

<System body>
FIG. 2 is a block diagram illustrating a configuration example of a main part of the magnification observation apparatus 1 in FIG. 1, and illustrates an example of a functional configuration in the system main body 100. In addition to the display 110, the system main unit 100 includes display reduction units 121 and 128, a display mosaic image storage unit 122, a live image update unit 123, a storage mosaic image storage unit 124, a live positioning unit 125, An image acquisition unit 126 and a mosaic image generation unit 127 are provided.

The display reduction unit 121 performs the operation of processing the moving image data from the camera 210 and generating reduced moving image data with a reduced image size. Specifically, an operation of generating frame images for display by reducing frame images continuously obtained from the camera 210 at a predetermined reduction ratio and outputting the frame images to the live image update unit 123 is performed. The reduction of the frame image is performed by, for example, pixel thinning processing or pixel value averaging processing. Here, it is assumed that the reduction process is performed so that the aspect ratio, that is, the aspect ratio of the frame image does not change before and after the reduction.

The display mosaic image storage unit 122 is a mosaic image holding unit that holds a display mosaic image, and includes, for example, a volatile semiconductor memory. The live image update unit 123 controls the display 110 based on the reduction rate information from the display reduction unit 121, and sets the display position of the display frame image obtained continuously from the display reduction unit 121 with respect to the display mosaic image. By updating, a live image is displayed on the display mosaic image. A live image is a moving image composed of a plurality of continuous display frame images.

The storage mosaic image storage unit 124 is a mosaic image storage unit that stores the storage mosaic image, and includes a nonvolatile storage element, for example, an HDD (Hard Disk Drive) device.

The live registration unit 125 includes registration reduction units 125a and 125b and a matching processing unit 125c, and the frame image obtained continuously from the camera 210 and the storage mosaic image read from the storage mosaic image storage unit 124. Each of them is reduced to perform matching processing.

The alignment reduction unit 125a performs an operation of reducing the frame image from the camera 210 at a constant reduction rate for alignment to generate an alignment frame image and outputting it to the matching processing unit 125c. The alignment reduction unit 125b generates a registration mosaic image by reducing the storage mosaic image read from the storage mosaic image storage unit 124 at a predetermined reduction rate for alignment, and outputs the generated mosaic image to the matching processing unit 125c. The operation to be performed.

The matching processing unit 125c determines the relative position between these images by pattern matching between the alignment frame image and the alignment mosaic image, generates relative position information, and outputs the relative position information to the live image update unit 123. It is carried out.

The live image update unit 123 determines the relative position between the display frame image and the display mosaic image based on the relative position information from the live position alignment unit 125, and the display position of the display frame image with respect to the display mosaic image The operation of updating is performed.

Here, the live alignment unit 125 is a processing unit that executes a lower-precision matching process than the storage alignment unit 127a and outputs low-accuracy coordinate data as relative position information.

The still image acquisition unit 126 performs an operation of acquiring a still image taken by the camera 210 based on an instruction for taking in from the console 300 and outputting the still image to the mosaic image generation unit 127. The pasting still image acquired from the camera 210 may be an image having a different imaging condition such as an exposure time from the moving image, or may be one of frame images constituting the moving image.

The mosaic image generation unit 127 includes a storage alignment unit 127a and an image connection unit 127b, and performs an operation of generating a storage mosaic image by combining a plurality of still images.

The storage alignment unit 127 a performs an operation of determining a relative position between the still image acquired by the still image acquisition unit 126 and the storage mosaic image read from the storage mosaic image storage unit 124. The determination of the relative position is performed by pattern matching between the still image and the storage mosaic image, and the relative position between the still image and the storage mosaic image is estimated with a higher resolution than the registration mosaic image.

The image connecting unit 127b combines the still image and the storage mosaic image based on the determination result by the storage registration unit 127a, generates a new storage mosaic image, and stores the image in the storage mosaic image storage unit 124. An operation to update the mosaic image is performed. Specifically, the still image acquired by the still image acquisition unit 126 and the storage mosaic image read from the storage mosaic image storage unit 124 are in a relative position between the images estimated by the storage registration unit 127a. Based on these, a new storage mosaic image is generated.

The still image and the mosaic image for storage are joined together by connecting both images based on the relative positions between these images. In addition, when connecting a still image and a storage mosaic image, a blending process of pixel values is performed on an overlapping area of both images in order to make the joint inconspicuous. The blending process is an image process in which pixel values are weighted and averaged between both images to obtain a pixel value of a composite image. ing.

Here, the storage mosaic image is divided and held in a plurality of image tiles of a certain size, for example, 512 × 512 pixels, and when the storage mosaic image is updated, only necessary portions are read out on the memory and updated. Shall be performed.

The display reduction unit 128 reads the updated storage mosaic image from the storage mosaic image storage unit 124 each time the storage mosaic image is updated, and reduces the read storage mosaic image for display. An operation to generate a mosaic image is performed.

In the display reduction units 121 and 128, an operation of reducing the frame image and the storage mosaic image based on the image size of the storage mosaic image and generating the display frame image and the display mosaic image is performed. Specifically, a reduction ratio for reducing the frame image and the storage mosaic image is determined every time the storage mosaic image is updated, and a process of reducing these images is performed.

The reduction ratio is determined based on the resolution of the display 110 and the size of the monitor screen of the live image so that the entire display mosaic image and the entire display frame image are displayed on the screen at the same time, for example. . Each time the saved mosaic image is updated, the reduction rate is determined from the image size of the mosaic image and the display image is reduced. The display magnification is automatically adjusted according to the size of the mosaic image. Live images can be displayed.

Here, the entire display mosaic image and a part of the display mosaic image overlap with the alignment mosaic image, and the display frame corresponding to the alignment frame image at an arbitrary position having an overlap area necessary for determining the relative position It is assumed that the reduction ratio is determined so that the entire image is displayed on the screen.

In general, when determining the relative positional relationship between a still image and a storage mosaic image by pattern matching between these images, in order to ensure the accuracy of alignment above a certain level, the still image and the storage mosaic image There must be a certain overlap area between them. In addition, when connecting the still image and the mosaic image for storage, if a joint is made inconspicuous by blending processing, a certain amount between the still image and the mosaic image for storage is necessary to smoothly connect the images with a smoothness of a certain level or more. The above overlapping area needs to exist.

In the display reduction units 121 and 128, the entire display frame image corresponding to the alignment frame image at an arbitrary position having the overlapping region as described above as the overlap region, and the entire display mosaic image. The reduction ratio is determined so that it is always displayed in the screen.

Also, at first, for each image tile of the mosaic image for storage, if the original size image is held as an on-memory image for displaying the mosaic image, and the display mosaic image does not fit within the monitor screen of the live image, The image tile is reduced, for example, a ½ image is created and used as an on-memory image for display. At this time, the full-size image is deleted from the memory. Further, when the display mosaic image does not fit in the monitor screen, for example, a ¼ image is created from ½ image for each image tile, and the display on-memory image is updated.

In this way, each time the display mosaic image does not fit in the monitor screen due to the joining of the new still images, the image tiles are sequentially reduced. Image tiles that have been changed by pasting still images or newly added image tiles are read out and are reduced in accordance with the size of the current on-memory image for display.

<Matching processing unit>
FIG. 3 is a block diagram illustrating a configuration example of the matching processing unit 125c in the system main body 100 of FIG. The matching processing unit 125c includes a feature amount extraction unit 131 and a relative position determination unit 132.

The feature amount extraction unit 131 performs an operation of extracting feature amounts from the alignment frame image and the alignment mosaic image. The feature amount may be anything as long as it serves as a mark for comparing images. Here, it is assumed that a vertex at which a plurality of edges intersect is extracted as a feature point.

The relative position determination unit 132 includes a comparison unit 141, a relative position calculation unit 142, and an overlapping region estimation unit 143, and performs an operation of determining a relative position between the alignment frame image and the alignment mosaic image by comparing feature points. Is going. The comparison unit 141 compares the feature points extracted from the alignment frame image with the feature points extracted from the alignment mosaic image, and outputs the comparison result to the relative position calculation unit 142. Yes.

The feature points are compared by, for example, extracting a region including the feature points from one image as a template and searching for the region most similar to the template region from the other image. As an index for measuring the similarity between regions, there are a method using a sum of squared errors of luminance values obtained for pixels in the region, and a method using a normalized correlation obtained by normalizing the luminance value of each pixel in the region by the average luminance. Conceivable.

The relative position calculation unit 142 determines the relative position between the alignment frame image and the alignment mosaic image based on the comparison result by the comparison unit 141, and outputs the determination result to the overlapping region estimation unit 143. An operation of outputting information to the live image update unit 123 is performed.

The overlapping area estimation unit 143 performs an operation of estimating an overlapping area between the current positioning frame image and the positioning mosaic image based on the determination result of the relative position regarding the past positioning frame image. For example, an overlapping area between the frame image and the positioning mosaic image is determined from the relative position determination result regarding the positioning frame image one frame before, and the overlapping area is an overlapping area between the current frame image and the mosaic image. The operation of determining that there is is performed.

In the comparison unit 141, the feature points in the overlap region estimated by the overlap region estimation unit 143 are compared, and an operation of outputting the comparison result to the relative position calculation unit 142 is performed. Then, as a result of comparing the feature points in the overlapping region, if the relative position cannot be determined, the feature points of the alignment frame image are compared with all the feature points of the alignment mosaic image, and compared. An operation of outputting the result to the relative position calculation unit 142 is performed.

Further, for the first alignment frame image, all feature points of the alignment frame image are compared with all feature points of the alignment mosaic image, and an operation of outputting the comparison result to the relative position calculation unit 142 is performed. . That is, for the first alignment frame image, the relative position is determined by comparing all the feature points of each image. On the other hand, for the second and subsequent frame images for alignment, first, the relative position is determined by comparing the feature points in the overlapping region estimated from the relative position determination result for the past frame image. At this time, if the relative position cannot be determined, the relative position is determined by comparing all the feature points of the images.

Here, the first frame image is, for example, a frame image that is first acquired after resuming photographing in a case where photographing is temporarily interrupted during the creation of a mosaic image and then photographing is resumed.

In general, when extracting feature points from two still images that overlap part of an image and finding a pair of corresponding feature points between these images, extract the feature points from the overlapping area of both images. Searching for a pair of corresponding feature points has a lower probability of occurrence of a false response than extracting and searching for feature points from the entire image. That is, by comparing the feature points in the overlapping region with priority and determining the relative position, it is possible to improve the probability of successful alignment of the alignment frame image. Furthermore, the alignment speed can be improved as compared with the case where all feature points in the image are compared.

Here, when the feature amount extraction unit 131 extracts feature points from the current alignment frame image, the feature amount extraction unit 131 extracts feature points from the overlap region estimated by the overlap region estimation unit 143. Then, when the relative position cannot be determined only by the feature points in the overlapping area, an operation of extracting the feature points from the area other than the overlapping area is performed.

The live image update unit 123 updates the display position when the moving image constituted by the display frame image is displayed as a live image on the display mosaic image based on the relative position determination result by the relative position calculation unit 142. Then, an operation of outputting the display data to the display 110 is performed.

<Live screen>
4 and 5 are explanatory views schematically showing an example of an operation at the time of displaying a live image in the magnification observation apparatus 1 of FIG. FIG. 4 shows a moving image A1 and a display mosaic image A3 captured by the camera 210. FIG. 5 shows a live screen 111 in which the moving image A1 is arranged as a live image on the mosaic image A3.

The moving image A1 is composed of a display frame image A2 that is repeatedly generated at a constant frame rate. For example, the display frame image A2 is generated at 15 fps. Here, it is assumed that the photographing magnification and the focus position are fixed.

The display mosaic image A3 is a mosaic image created by reducing the storage mosaic image for live screen display.

The live screen 111 is a monitor screen displayed on the display 110, and displays a display mosaic image A3 and a moving image A1 that are being created. On the live screen 111, the moving image A1 is arranged at a display position determined from a relative position determined by pattern matching between the current alignment frame image and the alignment mosaic image.

That is, since the moving image A1 being shot is displayed as a live image at an appropriate position on the display mosaic image A3 being created, the user confirms the positional relationship between the field of view being shot and the mosaic image being created. However, it is possible to capture a still image and connect it to a storage mosaic image.

<Pattern matching>
FIGS. 6A and 6B are diagrams showing an example of the pattern matching operation in the magnification observation apparatus 1 of FIG. 1, and all feature points extracted from the reference image B1 and the alignment frame image B2, respectively. A state in which a positive correspondence between these feature points is extracted by comparison of B3 is shown. FIG. 6A shows a state in which the feature point B3 extracted from the reference image B1 is compared with each feature point B3 in the frame image B2, and FIG. 6B shows the feature point B3. A positive correspondence between feature points extracted based on the comparison is shown.

The reference image B1 is a part of the mosaic image being created, and is extracted in advance from the positioning mosaic image as a pattern matching processing target. For example, the last connected still image is extracted as the reference image B1.

If the positional relationship between the reference image B1 and the frame image B2 is unknown, the feature point B3 is extracted for the entire image. Then, for each feature point B3 extracted from the reference image B1, whether or not a similar feature point exists in the frame image B2 is determined by comparison between the feature points.

The similarity between feature points can be measured by a squared error sum or normalized correlation of luminance values calculated for a predetermined region including the feature point B3, for example, a rectangular region of 5 pixels × 5 pixels.

Positive correspondences between feature points are extracted based on such comparison results. For example, the correspondence between feature points moving in parallel in the same direction is extracted as a positive correspondence. The relative position between the reference image B1 and the frame image B2 determines the movement amount of the feature point in the image based on the positive correspondence between the extracted feature points, and the movement of the frame image B2 with respect to the reference image B1 based on the movement amount. It is determined by determining the amount.

On the other hand, when a rough positional relationship between the reference image B1 and the frame image B2 is known in advance, a template region is appropriately extracted from one image, and by searching for the vicinity of the corresponding region of the other image, The relative position between these images can be determined with higher accuracy.

That is, the overlapping region B4 of these images is obtained from the determination result of the relative position between the (n-1) th frame image one frame before and the reference image. The overlapping area B4 is determined to be the overlapping areas B5 and B6 between the current nth frame image and the reference image. Then, for each feature point B3 in the overlapping region B5 of the reference image, a similar feature point is extracted from the overlapping region B6 of the nth frame image, thereby determining the relative position between these images.

Similar feature points are extracted by extracting a predetermined region near the position corresponding to the feature point from the overlap region B6 of the nth frame image for each feature point B3 in the overlap region B5 of the reference image. Done by exploring.

In the present embodiment, a method of comparing feature points in the overlapping region is used for the alignment of the alignment frame image and the alignment mosaic image in the second and subsequent frames. On the other hand, if the relative position cannot be determined by the alignment of the first frame alignment frame image and the alignment mosaic image or the comparison of the feature points in the overlapping region, or the image is captured based on the capture instruction. In order to align the still image and the storage mosaic image, a method of comparing all feature points is employed.

<Display range>
FIGS. 7 and 8 are diagrams illustrating an example of an operation at the time of displaying a live image in the magnification observation apparatus 1 in FIG. 1, and shows a live screen 111 on which a display mosaic image and an entire frame image are simultaneously displayed. Has been. FIG. 7 shows a display range 112 that includes the entire mosaic image for display and the entire live image that overlaps the mosaic image and part thereof.

In this example, a live image composed of continuous display frame images is displayed in a rectangular frame indicated by a solid line. The display position of the rectangular frame indicating the live image is determined based on the determination result of the relative position between the positioning mosaic image and the current positioning frame image.

The reduction ratio when creating a display frame image and a mosaic image by reducing the frame image and the storage mosaic image, respectively, overlaps the entire mosaic image for display and a part of the mosaic image for alignment. It is determined that the entire display frame image corresponding to the alignment frame image at an arbitrary position having an overlap area necessary for position determination is simultaneously displayed in the live screen 111.

Here, the display range 112 is set so that the display frame image and the display mosaic image corresponding to the alignment frame image at an arbitrary position having the minimum overlap area necessary for the determination of the relative position are included. The image within the display range 112 is defined and displayed on the live screen 111. That is, the display range 112 has a shape obtained by enlarging the display mosaic image outward.

In the state of FIG. 7, if an instruction is given to capture a still image for pasting, a still image having the same field of view as the live image being displayed is captured and a new mosaic image for storage is created by linking with the mosaic image for storage. . By reducing the new storage mosaic image, the display mosaic image being displayed is updated.

FIG. 8 shows a display mosaic image updated by pasting still images corresponding to the live image of FIG. 7 and a display range 112 changed in accordance with the update of the display mosaic image. The reduction ratio when the frame image and the storage mosaic image are reduced is updated every time the storage mosaic image is updated, so that the entire display mosaic image and the entire display frame image are included. Range 112 is adjusted.

Steps S101 to S110 in FIG. 9 are flowcharts showing an example of the operation at the time of capturing images for pasting in the magnification observation apparatus 1 in FIG. First, the still image acquisition unit 126 acquires a still image for pasting from the camera 210 based on a capture instruction from the console 300 (step S101).

The mosaic image generation unit 127 determines the relative position between these images by pattern matching between the acquired still image and the storage mosaic image read from the storage mosaic image storage unit 124, and pastes these images together A new mosaic image for storage is generated (steps S102 to S104).

When the storage mosaic image in the storage mosaic image storage unit 124 is updated with the new storage mosaic image (step S105), the display reduction units 121 and 128 update the image size of the updated storage mosaic image. Based on this, the display range 112 is determined, and the reduction ratio is determined (step S106).

At this time, if the display range 112 cannot be accommodated in the live screen 111, the reduction process is performed by changing the reduction ratio so that the entire display range 112 is displayed (steps S107 to S109). On the other hand, if the display range 112 can be accommodated in the live screen 111, the reduction process is performed at the same reduction rate (steps S107 and S109).

When the display mosaic image in the display mosaic image storage unit 122 is updated, the live image update unit 123 reads the updated mosaic image and updates the display mosaic image on the live screen 111 (step S110). .

According to this embodiment, since the display image is reduced based on the image size of the storage mosaic image, the display mosaic can be used even when the image size of the mosaic image changes greatly before and after the pasting. The entire image can be displayed on the screen. Therefore, it is possible to easily recognize the positional relationship between the field of view being shot and the mosaic image being created. In particular, when displaying a live image on a display mosaic image, the entire display mosaic image and the entire frame image are always displayed at the same time, so the positional relationship between the field of view being shot and the mosaic image being created can be determined. The user can easily recognize it.

Embodiment 2. FIG.
In the first embodiment, an example in which the reduction ratio when reducing the frame image and the storage mosaic image is determined so that the entire display mosaic image and the entire display frame image are displayed simultaneously is described. explained. In contrast, in the present embodiment, a case will be described in which the reduction ratio is fixed when the image size of the mosaic image exceeds a certain size.

In the display reduction units 121 and 128, when the image size of the storage mosaic image exceeds a certain size, an operation of fixing the reduction ratio when the frame image and the storage mosaic image are reduced is performed. That is, the reduction ratio is determined so that the entire display mosaic image and the entire display frame image are simultaneously displayed in the live screen 111 until the image size of the storage mosaic image exceeds a certain size. Processing is performed.

On the other hand, when the image size of the storage mosaic image exceeds a certain size, the reduction process is performed at the reduction rate up to that point. In this case, a part of the display mosaic image and the entire live image are displayed in the live screen 111.

FIG. 10 is a diagram illustrating an example of an operation at the time of live image display in the magnification observation apparatus according to the second embodiment of the present invention, and shows a live screen 111 on which a live image is displayed on a mosaic image for display. ing. In this example, a part of the mosaic image for display and the entire live image in which the mosaic image and the part overlap are displayed.

In the present embodiment, the reduction ratio is changed according to the size of the mosaic image as in the first embodiment up to a predetermined reduction ratio. On the other hand, if the image is reduced too much, it becomes difficult to grasp the positional relationship between the field of view being shot and the mosaic image being created.Therefore, when the predetermined reduction rate is reached, the reduction rate is not changed and the field of view being shot is fixed. The live image and the surrounding mosaic image are displayed at the display magnification that is the size.

By configuring in this way, when the image size of the storage mosaic image exceeds a certain size, the reduction ratio when the frame image and the storage mosaic image are reduced is fixed, so the field of view during shooting is too small. It is possible to prevent the positional relationship between the image and the mosaic image being created from becoming unrecognizable.

Embodiment 3 FIG.
In the first embodiment, the entire display mosaic image and the entire display frame image corresponding to the alignment frame image at an arbitrary position having an overlap area necessary for determining the relative position are displayed. The example of was described. On the other hand, in the present embodiment, the reduction rate when the frame image and the storage mosaic image are reduced is updated every time the display position of the display frame image with respect to the display mosaic image is updated. A case will be described in which the entire mosaic image and the updated display frame image are displayed.

In the display reduction units 121 and 128, every time the display position of the display frame image with respect to the display mosaic image is updated, an operation of updating the reduction rate when the frame image and the storage mosaic image are reduced is performed.

By configuring in this way, every time the display position of the display frame image is updated, the reduction ratio when the frame image and the mosaic image for storage are reduced is updated, so that the display follows the update of the display position. Thus, the entire mosaic image and frame image can be appropriately displayed in the live screen 111.

Embodiment 4 FIG.
In the first embodiment, the reduction ratio when the frame image and the mosaic image for storage are reduced for live screen display is determined so that the entire display mosaic image and the entire display frame image are displayed simultaneously. An example of the case has been described. On the other hand, in this embodiment, the recommended moving area of the frame image for which the relative position can be determined is displayed on the mosaic image, and the frame image and the mosaic image in the recommended moving area are displayed at the same time. A case where the reduction ratio is determined will be described.

FIG. 11 is a block diagram illustrating a configuration example of a main part of the magnification observation apparatus according to the fourth embodiment of the present invention, and illustrates an example of a functional configuration in the system main body 100a. The system main body 100a is different from the system main body 100 of FIG. 2 in that a recommended area display 151 is provided.

The recommended area display unit 151 performs an operation of displaying the recommended moving area of the frame image on the mosaic image. This movement recommendation area is an area having a certain shape and size having an overlap area with the mosaic image, and the display position for the mosaic image is determined based on the image size of the storage mosaic image. Specifically, of the areas where the relative position between the frame image and the mosaic image can be determined, an area where the area of the overlapping area with the mosaic image is the smallest is displayed as the recommended movement area.

In the display reduction units 121 and 128, an operation of reducing the frame image and the mosaic image for storage is performed so that the frame image and the mosaic image in the recommended movement area are displayed simultaneously. Specifically, there is an operation for determining a reduction ratio when reducing the frame image and the mosaic image for storage for live screen display so that at least the entire frame image and the entire mosaic image in the recommended movement area are displayed. Done.

The display position of the recommended movement area is updated every time the saving mosaic image is updated, and the reduction ratio is changed according to the update of the mosaic image and the recommended movement area.

<Recommended moving area>
FIG. 12 is a diagram illustrating an example of an operation at the time of live screen display in the magnification observation apparatus of FIG. 11, and shows a recommended movement area displayed on the mosaic image. On this live screen 111, a display mosaic image corresponding to the mosaic image for storage being created, a live image showing the field of view being photographed, and a recommended movement area are displayed.

In this example, a live image composed of display frame images is arranged in a rectangular frame indicated by a solid line. The display position of the rectangular frame is changed based on the determination result of the relative position between the reference image (a part of the mosaic image) and the current alignment frame image.

On the other hand, the recommended movement area represents a position relative to the mosaic image that is optimal for pasting still images, and is indicated by a rectangular frame indicated by a broken line. In this example, the recommended moving area is slightly larger than the display frame image, the still images are pasted with a certain level of alignment accuracy, and a still image necessary to create a mosaic image of a desired size. Is formed so as to obtain an optimum overlapping region that can minimize the number of sheets.

In Embodiments 1 to 4, an example in which vertices where edges intersect is extracted as a feature amount when comparing the alignment frame image and the alignment mosaic image has been described. It is not limited to this. For example, for a predetermined area on the image, a contrast value in the area may be extracted as a feature amount from the alignment frame image and the alignment mosaic image, and compared between these images.

Moreover, although Embodiment 1-4 demonstrated the example in case the relative positional relationship between these images was determined by the pattern matching between the mosaic image for alignment, and a frame image, this invention is based on this. It is not limited. For example, a position sensor that detects the position of the movable stage 230 in the x-axis direction and the y-axis direction is provided, and the relative position between the frame image and the mosaic image is determined based on the output of the position sensor. good.

DESCRIPTION OF SYMBOLS 1 Magnification observation apparatus 100,100a System main-body part 110 Display 111 Live screen 112 Display range 121,128 Display reduction part 122 Display mosaic image memory | storage part 123 Live image update part 124 Storage mosaic image memory | storage part 125 Live alignment part 125a , 125b Alignment reduction unit 125c Matching processing unit 126 Still image acquisition unit 127 Mosaic image generation unit 127a Storage registration unit 127b Image connection unit 131 Feature amount extraction unit 132 Relative position determination unit 141 Comparison unit 142 Relative position calculation unit 143 Overlap area estimation unit 151 Recommended area display unit 200 Camera unit 210 Camera 220 Movable holders 221, 231, 232 Position adjustment knob 230 Movable stage 300 Console A1 Moving image A2 Display frame image A3 Display mosaic Image B1 Reference image B2 Frame image B3 for alignment Feature points

Claims (7)

A movable stage on which an object to be inspected can be placed and moved in a plane intersecting the central axis of the objective lens;
A camera that is arranged opposite to the movable stage and generates an image obtained by photographing the inspection object through the objective lens;
By moving the movable stage, the field of view of the camera is changed, and two or more images having overlapping regions obtained by photographing different positions of the inspection object are bonded together, so that the field of view is wider than the actual field of view of the camera. Mosaic image generating means for generating a mosaic image;
Feature amount extraction means for extracting feature amounts from the mosaic image and a frame image newly generated by the camera;
A relative position determination means for determining a relative position between the mosaic image and the frame image by comparing the feature amounts;
When the relative position determination unit succeeds in determining the relative position between the mosaic image and the frame image, and the mosaic image generation unit generates a new mosaic image that connects the mosaic image and the frame image. A storing mosaic image storing means for storing the new mosaic image;
A display reduction unit that generates a display mosaic image by reducing the mosaic image stored in the storage mosaic image storage unit so that the entire mosaic image is displayed in a screen;
Display mosaic image storage means for storing the display mosaic image reduced by the display reduction means;
An image pickup apparatus comprising: an image display unit that reads the display mosaic image from the display mosaic image storage unit and displays the entire display mosaic image. The display reduction means calculates the reduction ratio of the display mosaic image based on the image size of the mosaic image after the update every time the mosaic image stored in the storage mosaic image storage means is updated. The imaging apparatus according to claim 1, wherein the imaging apparatus is updated. The display reduction unit changes the reduction ratio when the updated display mosaic image does not fit within the screen, and displays the mosaic image without changing the reduction ratio when fits within the screen. The imaging apparatus according to claim 2. The display reduction unit changes the reduction rate up to a predetermined reduction rate according to the size of the mosaic image, and when the predetermined reduction rate is reached, fixing the reduction rate without changing , 4. The display mosaic image is reduced so that the whole or part of the display mosaic image is displayed on the screen, and the image display means displays the whole or part of the display mosaic image. The imaging device described in 1. The storage mosaic image storage means stores the mosaic image divided into a plurality of image tiles,
5. The imaging apparatus according to claim 1, wherein the image display unit reduces the plurality of image tiles when the display mosaic image does not fit within a screen. 2. The display device according to claim 1, further comprising a recommended area display unit that displays a recommended moving area of the frame image that has an overlapping area with the mosaic image and is capable of determining a relative position between the frame image and the mosaic image. 6. The imaging device according to any one of 1 to 5. The feature amount extraction unit extracts a feature amount from the reference image using the frame image and an image last connected to the mosaic image as a reference image,
The imaging apparatus according to claim 1, wherein the relative position determination unit determines a relative position between the reference image and the frame image.