JP5096301B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus capable of displaying a moving image generated by a camera that captures an inspection object on a mosaic image having a wider field of view than the actual field of view of the camera.

  An image mosaic is conventionally known as a technique for creating a single wide-field image having a wider field of view than the actual field of view of the camera by pasting together a plurality of still images taken while changing the field of view. In the case of an imaging apparatus such as a digital microscope that photographs a subject magnified by an objective lens, the field of view can be changed by moving a movable stage on which an inspection target is placed. The wide-field image is called a mosaic image, and is created by connecting a plurality of still images captured while changing the field of view based on the relative positions between the images.

  For example, in the case of a conventional imaging device that includes a sensor that detects the position of the movable stage and automatically captures an imaging range designated by the user, the relative positional relationship between the images is determined from the control information of the movable stage, The still images are joined together. In such an imaging apparatus, once the shooting range is designated and shooting is started, the shooting range cannot be changed midway. Further, since it is necessary to detect the position of the movable stage with high accuracy, there is a problem that the system configuration becomes complicated and the cost is increased.

  On the other hand, there is also an imaging device that determines the relative positional relationship between images by pattern matching between images and performs still image joining. However, there has been no imaging apparatus that can capture a still image and connect it to the mosaic image while allowing the user to confirm the positional relationship between the field of view being photographed and the mosaic image being created on the display.

  The present invention has been made in view of the above circumstances, and provides an imaging apparatus capable of easily recognizing the positional relationship between a field of view being shot and a mosaic image being created without complicating the system configuration. The purpose is to do. In particular, when displaying a moving image being shot as a live image at an appropriate position on a mosaic image being created, an imaging device that improves the accuracy of the alignment between the frame image and the mosaic image constituting the moving image The purpose is to provide.

An imaging apparatus according to a first aspect of the present invention includes a movable stage that can be moved in two different directions while placing an inspection object thereon, and a plurality of continuous stages that are arranged to face the movable stage and photograph the inspection object. A camera that generates a moving image composed of the frame images, a mosaic image holding unit that holds a mosaic image having a wider field of view than the actual field of view of the camera, and a feature amount extraction that extracts feature amounts from the frame image and the mosaic image A relative position determination means for determining a relative position between the frame image and the mosaic image by comparing the feature amount, and a display position of the frame image with respect to the mosaic image based on the determination result of the relative position. update the, and a live image display means for displaying the moving image on the mosaic image, the relative position determining means, The overlap region between the previous frame image and the mosaic image obtained from the determination result of the relative position between the frame image and the mosaic image is estimated as the overlap region between the current frame image and the mosaic image, and is estimated. configured to compare the feature amounts of the overlap region that is.

  In this imaging device, the relative position between these images is determined by comparing the feature amounts extracted from the frame image and the mosaic image, and the moving image is displayed on the mosaic image based on the determination result. The image can be displayed as a live image at an appropriate position on the mosaic image being created. According to such a configuration, since the moving image being shot is displayed as a live image at an appropriate position on the mosaic image being created, the positional relationship between the field of view being shot and the mosaic image being created is shown to the user. While confirming, it is possible to capture the frame image and connect it to the mosaic image. In addition, since the relative position between these images is determined by comparing the feature amounts extracted from the frame image and the mosaic image, it is possible to prevent the system configuration from becoming complicated. Furthermore, when comparing feature amounts, an overlap region between a frame image and a mosaic image is estimated based on a past relative position determination result, and the feature amounts in the overlap region are compared. The accuracy of alignment between the frame image to be performed and the mosaic image can be improved.

  In addition to the above-described configuration, the imaging apparatus according to the second aspect of the present invention is configured so that when the relative position determination unit fails to determine the relative position as a result of comparing the feature amounts in the overlapping region, The relative position is determined by comparing the feature amount of the mosaic image. According to such a configuration, when the relative position cannot be determined by comparing the feature amounts in the overlapping region, the entire feature amount of the frame image is compared with a part of or all of the feature amounts of the mosaic image. Since the position is determined, the relative position determination result can be reliably obtained even if the frame image is partially out of focus.

  In the imaging device according to the third aspect of the present invention, in addition to the above-described configuration, the relative position determination unit determines the relative position of the first frame image by comparing the total feature amount of the frame image with the feature amount of the mosaic image. Configured as follows. According to such a configuration, for the first frame image, the relative position is determined by comparing all the feature values of the frame image with some or all of the feature values of the mosaic image. Thereafter, when the shooting is resumed, the relative position between the images can be determined for the first frame image.

  In addition to the above configuration, the imaging device according to a fourth aspect of the present invention captures the frame image based on a capture instruction, and pastes the frame image on the mosaic image based on the determination result of the relative position regarding the captured frame image. A mosaic image generating means for generating a new mosaic image is provided. According to such a configuration, since the frame images are joined based on the determination result of the relative position regarding the captured frame image, an increase in processing load can be suppressed.

  In the imaging device according to a fifth aspect of the present invention, in addition to the above-described configuration, the feature amount extraction unit extracts a feature amount in the reference image using the frame image that is finally pasted on the mosaic image as a reference image. The relative position determination means is configured to determine the relative position between the frame image and the mosaic image by comparing the feature amount of the frame image with the feature amount of the reference image.

  In addition to the above-described configuration, the imaging apparatus according to a sixth aspect of the present invention includes a first alignment reduction unit that reduces the frame image forming the moving image to generate an alignment frame image, and a frame image before reduction. Mosaic image generation means for generating a mosaic image for storage by pasting together, second reduction means for alignment for generating a mosaic image for alignment by reducing the mosaic image for storage, and frame images constituting the moving image A display reduction unit that generates a display frame image by reducing the display mosaic image and generates a display mosaic image by reducing the storage mosaic image, and the feature amount extraction unit includes the alignment frame image and the alignment frame image. A feature amount is extracted from the mosaic image for alignment, and the relative position determination unit is configured to output the frame image for alignment and the mosaic for alignment. The live image display means displays a moving image constituted by the display frame image on the display mosaic image as a live image, and the mosaic image generation means determines the relative position between the images. It is configured to estimate a relative position between the frame image and the storage mosaic image at a higher resolution than the mosaic image for combination, and to combine the frame image with the storage mosaic image to generate a new storage mosaic image .

  According to the imaging apparatus of the present invention, since the moving image being shot is displayed as a live image at an appropriate position on the mosaic image being created, the positional relationship between the field of view being shot and the mosaic image being created is determined by the user. It is possible to capture the frame image and connect it to the mosaic image while confirming. In addition, since the relative position between these images is determined by comparing the feature amounts extracted from the frame image and the mosaic image, it is possible to prevent the system configuration from becoming complicated. Furthermore, when comparing feature amounts, an overlap region between the current frame image and the mosaic image is estimated based on the determination result of the relative position with respect to the past frame image, and the feature amounts in this overlap region are compared. Thus, it is possible to improve the accuracy of the alignment between the frame image and the mosaic image constituting the moving image. Therefore, it is possible to realize an imaging apparatus capable of easily recognizing the positional relationship between the field of view being shot and the mosaic image being created without complicating the system configuration.

<Magnification observation device>
FIG. 1 is a system diagram illustrating an example of a schematic configuration of an imaging apparatus according to an embodiment of the present invention. As an example of the imaging apparatus, a magnification observation apparatus 1 including a system main body unit 100, a camera unit 200, and a console 300 is illustrated. It is shown. The magnification observation apparatus 1 is a digital microscope that can capture a subject magnified by an objective lens, generate a moving image, and display the moving image on the display 110 of the system main body 100.

  The camera unit 200 is an imaging unit for imaging an inspection object while changing the field of view, and includes a camera 210, a movable holder 220, and a movable stage 230. The camera 210 is a reading device that photographs an inspection object as a subject and generates a moving image composed of a plurality of frame images continuous at a constant frame rate, and includes an objective lens, a CCD image sensor, and an illumination in a cylindrical casing. It is configured by arranging devices and the like.

  The movable holder 220 is a holding unit that holds the camera 210 so as to be movable in a direction parallel to the central axis of the objective lens. Here, the direction parallel to the central axis of the objective lens of the camera 210 is referred to as the z-axis direction, and the position of the camera 210 in the z-axis direction can be adjusted by turning the position adjustment knob 221.

  The movable stage 230 is a holding unit that holds the inspection target, and is movable in a plane that intersects the z-axis with the inspection target placed thereon. Here, a plane perpendicular to the z-axis is referred to as an xy plane, and the position of the movable stage 230 in the xy plane can be adjusted by turning the position adjustment knobs 231 and 232. That is, the movable stage 230 is a stage that can be moved in two different directions while the inspection object is placed by turning the position adjustment knobs 231 and 232.

  Specifically, the position in the x-axis direction can be adjusted by turning the position adjustment knob 231, and the position in the y-axis direction can be adjusted by turning the position adjustment knob 232. The camera 210 is arranged to face such a movable stage 230.

  The console 300 is an input device for instructing the system main body unit 100 to start and end shooting and to take captured image data.

  The system main unit 100 displays a moving image captured by the camera 210 on the display 110, and combines the frame images constituting the moving image to generate a mosaic image having a wider field of view than the actual field of view of the camera 210. It is.

<System body>
FIG. 2 is a block diagram illustrating a configuration example of a main part of the magnification observation apparatus 1 in FIG. In addition to the display 110, the system main unit 100 includes display reduction units 121 and 127, a display mosaic image storage unit 122, a live image update unit 123, a storage mosaic image storage unit 124, a live positioning unit 125, and a mosaic. An image generation unit 126 is provided.

  The display reduction unit 121 performs the operation of processing the moving image data from the camera 210 and generating reduced moving image data with a reduced image size. Specifically, an operation of generating frame images for display by reducing frame images continuously obtained from the camera 210 at a predetermined reduction ratio and outputting the frame images to the live image update unit 123 is performed. The reduction of the frame image is performed by, for example, pixel thinning processing or pixel value averaging processing. Here, it is assumed that the reduction process is performed so that the aspect ratio of the frame image does not change before and after the reduction.

  The display mosaic image storage unit 122 is a mosaic image holding unit that holds a display mosaic image, and includes, for example, a volatile semiconductor memory. The live image update unit 123 controls the display 110 based on the reduction rate information from the display reduction unit 121, and sets the display position of the display frame image obtained continuously from the display reduction unit 121 with respect to the display mosaic image. By updating, a live image is displayed on the display mosaic image. A live image is a moving image composed of a plurality of continuous display frame images.

  The storage mosaic image storage unit 124 is a mosaic image storage unit that stores the storage mosaic image, and includes a nonvolatile storage element, for example, an HDD (Hard Disk Drive) device.

  The live registration unit 125 includes registration reduction units 125a and 125b and a matching processing unit 125c, and the frame image obtained continuously from the camera 210 and the storage mosaic image read from the storage mosaic image storage unit 124. Each of them is reduced to perform matching processing.

  The alignment reduction unit 125a performs an operation of reducing the frame image from the camera 210 at a constant reduction rate for alignment to generate an alignment frame image and outputting it to the matching processing unit 125c. The alignment reduction unit 125b performs an operation of reducing the storage mosaic image read from the storage mosaic image storage unit 124 for alignment, generating a registration mosaic image, and outputting the image to the matching processing unit 125c. Yes.

  The matching processing unit 125c determines a relative position between these images by pattern matching between the alignment frame image and the alignment mosaic image, generates relative position information, and generates a live image update unit 123 and a mosaic image. The operation of outputting to the unit 126 is performed.

  The live image update unit 123 determines the relative position between the display frame image and the display mosaic image based on the relative position information from the live position alignment unit 125, and the display position of the display frame image with respect to the display mosaic image The operation of updating is performed.

  The mosaic image generation unit 126 includes a storage registration unit 126a and an image connection unit 126b, and performs an operation of generating a storage mosaic image by pasting frame images before reduction.

  The storage alignment unit 126a captures the frame image from the camera 210 based on the capture instruction from the console 300, and the relative relationship between the captured frame image and the storage mosaic image read from the storage mosaic image storage unit 124. An operation for determining the position is performed. The determination of the relative position is performed based on the relative position information from the live position alignment unit 125, and by using the relative position determination result regarding the captured frame image, the frame with a higher resolution than the mosaic image for alignment is used. A relative position between the image and the storage mosaic image is estimated.

  The image connecting unit 126b combines the frame image and the storage mosaic image based on the determination result by the storage registration unit 126a, generates a new storage mosaic image, and stores it in the storage mosaic image storage unit 124. An operation to update the mosaic image is performed. Specifically, the frame image from the camera 210 is captured based on the capture instruction from the console 300. Then, the captured frame image and the storage mosaic image read from the storage mosaic image storage unit 124 are pasted together based on the relative position between the images estimated by the storage registration unit 126a, and a new storage mosaic is created. An image is generated.

  The frame image and the mosaic image for storage are joined by connecting both images based on the relative position between these images. Also, when connecting the frame image and the storage mosaic image, a blending process of pixel values is performed on the overlapping area of both images in order to make the joint inconspicuous. The blending process is an image process in which pixel values are weighted and averaged between both images to obtain a pixel value of a composite image. ing.

  The display reduction unit 127 reads the updated storage mosaic image from the storage mosaic image storage unit 124 each time the storage mosaic image is updated, and reduces the read storage mosaic image for display. An operation to generate a mosaic image is performed.

  Here, the live alignment unit 125 is a processing unit that executes a lower-precision matching process than the storage alignment unit 126a and outputs low-accuracy coordinate data as relative position information.

<Matching processing unit>
FIG. 3 is a block diagram illustrating a configuration example of the matching processing unit 125c in the system main body 100 of FIG. The matching processing unit 125c includes a feature amount extraction unit 131 and a relative position determination unit 132.

  The feature amount extraction unit 131 performs an operation of extracting feature amounts from the alignment frame image and the alignment mosaic image. The feature amount may be anything as long as it serves as a mark for comparing images. Here, it is assumed that a vertex at which a plurality of edges intersect is extracted as a feature point.

  In the feature amount extraction unit 131, when extracting feature points from the mosaic image for alignment, the mosaic image for alignment is extracted from the viewpoint of shortening the processing time required for alignment when the size of the mosaic image becomes large. An operation of extracting a feature point in the reference image is performed by using a part of the frame image, for example, a frame image finally pasted on the mosaic image as a reference image for alignment.

  The relative position determination unit 132 includes a comparison unit 141, a relative position calculation unit 142, and an overlapping region estimation unit 143, and performs an operation of determining a relative position between the alignment frame image and the alignment mosaic image by comparing feature points. Is going. The comparison unit 141 compares the feature points extracted from the alignment frame image with the feature points extracted from the alignment mosaic image, and outputs the comparison result to the relative position calculation unit 142. Yes.

  The feature points are compared by, for example, extracting a region including the feature points from one image as a template and searching for the region most similar to the template region from the other image. As an index for measuring the similarity between regions, there are a method using a sum of squared errors of luminance values obtained for pixels in the region, and a method using a normalized correlation obtained by normalizing the luminance value of each pixel in the region by the average luminance. Conceivable.

  The comparison unit 141 compares the feature points extracted from the alignment frame image with the feature points extracted from the alignment mosaic image, and the feature points of the alignment frame image and the reference image. The operation of comparing the points is performed.

  The relative position calculation unit 142 determines the relative position between the alignment frame image and the alignment mosaic image based on the comparison result by the comparison unit 141, and outputs the determination result to the overlapping region estimation unit 143. An operation of outputting information to the live image update unit 123 and the mosaic image generation unit 126 is performed.

  The overlapping area estimation unit 143 performs an operation of estimating an overlapping area between the current positioning frame image and the positioning mosaic image based on the determination result of the relative position regarding the past positioning frame image. For example, an overlapping area between the frame image and the positioning mosaic image is determined from the relative position determination result regarding the positioning frame image one frame before, and the overlapping area is an overlapping area between the current frame image and the mosaic image. The operation of determining that there is is performed.

  The overlap area estimation unit 143 estimates an overlap area between the alignment frame image and the alignment reference image when estimating the overlap area between the current alignment frame image and the alignment mosaic image. Operation is performed.

  In the comparison unit 141, the feature points in the overlap region estimated by the overlap region estimation unit 143 are compared, and an operation of outputting the comparison result to the relative position calculation unit 142 is performed. Then, as a result of comparing the feature points in the overlap region, if the relative position cannot be determined, all the feature points of the alignment frame image and the mosaic image for alignment are subject to matching processing. All the feature points included in the existing reference image for alignment are compared, and an operation for outputting the comparison result to the relative position calculation unit 142 is performed.

  Further, for the first alignment frame image, all feature points of the alignment frame image are compared with all feature points included in the reference image, and an operation of outputting the comparison result to the relative position calculation unit 142 is performed. That is, the relative position of the first alignment frame image is determined by comparing all feature points of the frame image with all feature points of the reference image. On the other hand, for the second and subsequent frame images for alignment, first, the relative position is determined by comparing the feature points in the overlapping region estimated from the relative position determination result for the past frame image. At this time, if the relative position cannot be determined, the relative position is determined by comparing all the feature points of the frame image with all the feature points of the reference image.

  Here, the first frame image is, for example, a frame image that is first acquired after resuming photographing in a case where photographing is temporarily interrupted during the creation of a mosaic image and then photographing is resumed.

  In general, when extracting feature points from two still images that overlap part of an image and finding a pair of corresponding feature points between these images, extract the feature points from the overlapping area of both images. Searching for a pair of corresponding feature points has a lower probability of occurrence of a false response than extracting and searching for feature points from the entire image. That is, by comparing the feature points in the overlapping region with priority and determining the relative position, it is possible to improve the probability of successful alignment of the alignment frame image. Furthermore, the alignment speed can be improved as compared with the case where all feature points in the image are compared.

  Here, when the feature amount extraction unit 131 extracts feature points from the current alignment frame image, the feature amount extraction unit 131 extracts feature points from the overlap region estimated by the overlap region estimation unit 143. Then, when the relative position cannot be determined only by the feature points in the overlapping area, an operation of extracting the feature points from the area other than the overlapping area is performed.

  The live image update unit 123 updates the display position when the moving image constituted by the display frame image is displayed as a live image on the display mosaic image based on the relative position determination result by the relative position calculation unit 142. Then, an operation of outputting the display data to the display 110 is performed.

<Live screen>
4 and 5 are explanatory views schematically showing an example of an operation at the time of displaying a live image in the magnification observation apparatus 1 of FIG. FIG. 4 shows a moving image A1 and a display mosaic image A3 captured by the camera 210. FIG. 5 shows a live screen 111 in which the moving image A1 is arranged as a live image on the mosaic image A3.

  The moving image A1 is composed of a display frame image A2 that is repeatedly generated at a constant frame rate. For example, the display frame image A2 is generated at 15 fps. Here, it is assumed that the photographing magnification and the focus position are fixed.

  The display mosaic image A3 is a mosaic image created by reducing the storage mosaic image for live screen display.

  The live screen 111 is a monitor screen displayed on the display 110, and displays a display mosaic image A3 and a moving image A1 that are being created. On the live screen 111, the moving image A1 is arranged at a display position determined from a relative position determined by pattern matching between the current alignment frame image and the alignment mosaic image.

  That is, since the moving image A1 being shot is displayed as a live image at an appropriate position on the display mosaic image A3 being created, the user confirms the positional relationship between the field of view being shot and the mosaic image being created. However, it is possible to capture the frame image and connect it to the mosaic image.

<Pattern matching>
FIGS. 6A and 6B are diagrams showing an example of the pattern matching operation in the magnification observation apparatus 1 in FIG. 1, and by comparing all feature points B3 extracted from the reference image B1 and the frame image B2, respectively. The manner in which the positive correspondence between these feature points is extracted is shown. FIG. 6A shows a state in which the feature point B3 extracted from the reference image B1 is compared with each feature point B3 in the frame image B2, and FIG. 6B shows the feature point B3. A positive correspondence between feature points extracted based on the comparison is shown.

  The reference image B1 is a part of the mosaic image being created, and is extracted in advance from the mosaic image as a pattern matching processing target. For example, the last connected frame image is extracted as the reference image. Alternatively, when the size of the overlap region between the frame image last connected to the mosaic image and the current frame image is below a certain level, the frame image connected last to the mosaic image It is also conceivable to extract a frame image from a mosaic image as a reference image.

  If the positional relationship between the reference image B1 and the frame image B2 is unknown, the feature point B3 is extracted for the entire image. Then, for each feature point B3 extracted from the reference image B1, whether or not a similar feature point exists in the frame image B2 is determined by comparison between the feature points.

  The similarity between feature points can be measured by a squared error sum or normalized correlation of luminance values calculated for a predetermined region including the feature point B3, for example, a rectangular region of 5 pixels × 5 pixels.

  Positive correspondences between feature points are extracted based on such comparison results. For example, the correspondence between feature points moving in parallel in the same direction is extracted as a positive correspondence. The relative position between the reference image B1 and the frame image B2 determines the movement amount of the feature point in the image based on the positive correspondence between the extracted feature points, and the movement of the frame image B2 with respect to the reference image B1 based on the movement amount. It is determined by determining the amount.

  FIGS. 7A and 7B are diagrams showing an example of the pattern matching operation in the magnification observation apparatus 1 in FIG. 1, and are estimated from the relative positions between the reference image and the (n−1) th frame image. A state in which the feature point B3 in the overlapping regions B5 and B6 is compared is shown. FIG. 7A shows an overlapping region B4 of both images obtained from the relative position between the reference image and the (n−1) th frame image. FIG. 7B shows overlapping regions B5 and B6 of the reference image and the nth frame image estimated from the relative position between the reference image and the (n−1) th frame image.

  When a rough positional relationship between the reference image B1 and the frame image B2 is known in advance, a template region is appropriately extracted from one image, and these regions are searched by searching for the vicinity of the corresponding region of the other image. The relative position between images can be determined with higher accuracy.

  That is, the overlapping region B4 of these images is obtained from the determination result of the relative position between the (n-1) th frame image one frame before and the reference image. The overlapping area B4 is determined to be the overlapping areas B5 and B6 between the current nth frame image and the reference image. Then, for each feature point B3 in the overlapping region B5 of the reference image, a similar feature point is extracted from the overlapping region B6 of the nth frame image, thereby determining the relative position between these images.

  Similar feature points are extracted by extracting a predetermined region near the position corresponding to the feature point from the overlap region B6 of the nth frame image for each feature point B3 in the overlap region B5 of the reference image. Done by exploring.

  In the present embodiment, there is an overlap between the alignment of the alignment frame image and the alignment mosaic image in the second and subsequent frames, and the connection processing of the frame image acquired based on the acquisition instruction and the mosaic image. The method shown in FIG. 7 for comparing feature points in the region is employed. On the other hand, if the relative position cannot be determined by alignment between the alignment frame image of the first frame and the mosaic image for alignment, or comparison of feature points in the overlapping area, comparison is performed for all feature points. The method of FIG. 6 to perform is employ | adopted.

  FIGS. 8A and 8B are transition diagrams showing an example of an operation at the time of capturing a live image in the magnification observation apparatus 1 in FIG. 1. FIG. 8A shows a reference image and a part thereof. An overlapping live image is shown, and FIG. 8B shows a live image on the mosaic image after the capture instruction.

  In this example, the reference image extracted from the mosaic image is arranged in a rectangular frame indicated by a broken line. On the other hand, the live image is arranged in a rectangular frame indicated by a solid line. The display position of the rectangular frame indicating the live image is determined based on the determination result of the relative position between the reference image and the current alignment frame image.

  If the capture of the frame image is instructed in the state of FIG. 8A, a frame image corresponding to the display frame image being displayed is captured and connected to the storage mosaic image. A display mosaic image is created from the new storage mosaic image created by this connection, and the display mosaic image being displayed is updated. At this time, the reference image is also changed to one corresponding to the captured frame image.

  Steps S101 to S116 in FIG. 9 are flowcharts illustrating an example of an operation when displaying a live image in the magnification observation apparatus 1 in FIG. First, the feature quantity extraction unit 131 acquires a position alignment frame image, and if it is the first frame image, extracts a feature point from the entire image (steps S101 to S103).

  Next, the relative position determination unit 132 compares all feature points of the frame image with all feature points of the reference image, and determines a relative position between the current alignment frame image and the alignment mosaic image. (Step S104). At this time, if the relative position cannot be determined and alignment fails, a matching error is notified to the user (steps S105 and S114). If the alignment is successful, the display position of the live image is determined based on the relative position determination result, and the live image is updated (steps S105 to S107).

  On the other hand, if the acquired alignment frame image is not the first frame image, the relative position determination unit 132 determines the result of the previous alignment, that is, the alignment frame image one frame before and the alignment mosaic image. With reference to the determination result of the relative position between, the overlap region between the current alignment frame image and the reference image is estimated (steps S102, S110, S111). The feature amount extraction unit 131 extracts feature points from the estimated overlap region.

  Next, the relative position determination unit 132 compares the feature points in the overlapping region estimated from the current alignment frame image and the reference image, and determines the relative position between these images (step S112). . At this time, if the relative position cannot be determined and the alignment fails, the processing procedure after step S103 is executed (step S113). When the alignment is successful, the display position of the live image is determined based on the relative position determination result, and the live image is updated (steps S113, S106, and S107).

  After the live image is updated, if there is a frame image capture instruction, the frame image is captured based on the capture instruction and a connection process is performed to update the display and storage mosaic images (steps S108, S115, S115). S116). The processing procedure from step S101 to S108 is repeated until the end of photographing is instructed (step S109).

  According to the present embodiment, since the moving image being shot is displayed as a live image at an appropriate position on the mosaic image being created, the positional relationship between the field of view being shot and the mosaic image being created is shown to the user. While confirming, it is possible to capture the frame image and connect it to the mosaic image. Moreover, since the relative position between these images is determined by comparing the feature points extracted from the frame image and the mosaic image, it is possible to prevent the system configuration from becoming complicated. Furthermore, when comparing feature points, an overlap region between the current frame image and the mosaic image is estimated based on the determination result of the relative position with respect to the past frame image, and the feature points in this overlap region are compared. Thus, it is possible to improve the accuracy of the alignment between the frame image and the mosaic image constituting the moving image.

  Further, by using the relative position determination result by the live alignment unit 125 for the frame image captured based on the capture instruction, the frame image is aligned with higher resolution than the alignment mosaic image. Since the bonding is performed, the alignment accuracy can be improved without increasing the processing load.

  In the present embodiment, an example in which vertices where edges intersect is extracted as a feature amount when comparing the alignment frame image and the alignment mosaic image has been described. It is not limited. For example, for a predetermined area on the image, a contrast value in the area may be extracted as a feature amount from the alignment frame image and the alignment mosaic image, and compared between these images. Further, template matching may be executed on a predetermined area in the overlap region based on the feature amount in the area without extracting an edge or contrast.

  In the present embodiment, when the relative position cannot be determined by comparing the feature amounts in the overlapping region, all the feature points of the alignment frame image and all of the reference images extracted from the alignment mosaic image are used. Although the example in which the relative position is determined by comparing with the feature points has been described, the present invention is not limited to this. For example, if the relative position cannot be determined by comparing the feature values in the overlapping region, or if the relative position is determined for the first acquired frame image, all the feature values of the alignment frame image are used for alignment. The relative position may be determined by comparing all the feature values of the mosaic image.

1 is a system diagram showing an example of a schematic configuration of an imaging apparatus according to an embodiment of the present invention, and a magnification observation apparatus 1 is shown as an example of the imaging apparatus. FIG. 2 is a block diagram illustrating a configuration example of a main part of the magnification observation apparatus 1 in FIG. 1, in which an example of a functional configuration in a system main body unit 100 is illustrated. FIG. 3 is a block diagram illustrating a configuration example of a matching processing unit 125c in the system main body unit 100 of FIG. It is explanatory drawing which showed typically an example of the operation | movement at the time of the live image display in the magnification observation apparatus 1 of FIG. 1, and the moving image A1 and the mosaic image A3 for a display are shown. It is explanatory drawing which showed typically an example of the operation | movement at the time of the live image display in the magnification observation apparatus 1 of FIG. 1, and the live screen 111 is shown. It is the figure which showed an example of the pattern matching operation | movement in the magnification observation apparatus 1 of FIG. 1, and the mode that the positive correspondence between feature points is extracted by the comparison of all the feature points B3 is shown. It is the figure which showed an example of the pattern matching operation | movement in the magnification observation apparatus 1 of FIG. 1, and a mode that the feature point B3 in overlap area | region B5, B6 is compared is shown. It is the transition diagram which showed an example of the operation | movement at the time of live image taking in the magnification observation apparatus 1 of FIG. 3 is a flowchart illustrating an example of an operation when displaying a live image in the magnification observation apparatus 1 in FIG. 1.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Magnification observation apparatus 100 System main-body part 110 Display 121,127 Display reduction part 122 Display mosaic image memory | storage part 123 Live image update part 124 Storage mosaic image memory | storage part 125 Live position alignment part 125a, 125b Position reduction part 125c Matching processing unit 126 Mosaic image generation unit 126a Storage registration unit 126b Image connection unit 131 Feature amount extraction unit 132 Relative position determination unit 141 Comparison unit 142 Relative position calculation unit 143 Overlapping region estimation unit 200 Camera unit 210 Camera 220 Movable holder 221 , 231, 232 Position adjustment knob 230 Movable stage 300 Console A1 Moving image A2 Display frame image A3 Display mosaic image

Claims (6)

A movable stage that can be moved in two different directions with the inspection object placed thereon;
A camera that is arranged opposite to the movable stage, shoots the inspection object, and generates a moving image composed of a plurality of continuous frame images;
Mosaic image holding means for holding a mosaic image having a wider field of view than the actual field of view of the camera;
Feature amount extraction means for extracting feature amounts from the frame image and the mosaic image;
Relative position determination means for determining a relative position between the frame image and the mosaic image by comparing the feature amounts;
Live image display means for updating the display position of the frame image with respect to the mosaic image based on the determination result of the relative position, and displaying the moving image on the mosaic image,
The relative position determination unit is configured to determine an overlap region between the previous frame image and the mosaic image obtained from a determination result of a relative position between the previous frame image and the mosaic image, and to determine an overlap area between the current frame image and the mosaic image. An imaging apparatus characterized by estimating as an overlap area and comparing the estimated feature quantities in the overlap area.   When the relative position cannot be determined as a result of comparing the feature amounts in the overlapping area, the relative position determination unit determines the relative position by comparing all the feature amounts of the frame image and the feature amount of the mosaic image. The imaging apparatus according to claim 1.   3. The imaging apparatus according to claim 1, wherein the relative position determination unit determines the relative position of the first frame image by comparing all the feature amounts of the frame image and the feature amount of the mosaic image. .   Mosaic image generating means for capturing the frame image based on the capture instruction, pasting the frame image on the mosaic image based on the determination result of the relative position regarding the captured frame image, and generating a new mosaic image The imaging apparatus according to claim 1. The feature amount extraction unit extracts a feature amount in the reference image by using the frame image finally pasted to the mosaic image as a reference image,
The relative position determination unit determines a relative position between the frame image and the mosaic image by comparing a feature amount of the frame image with a feature amount of the reference image. The imaging device described. A first alignment reduction means for reducing the frame image constituting the moving image to generate an alignment frame image;
Mosaic image generation means for generating a storage mosaic image by pasting together the frame images before reduction,
A second alignment reduction unit that reduces the storage mosaic image to generate an alignment mosaic image;
Reducing the frame image constituting the moving image to generate a display frame image, and reducing the storage mosaic image to generate a display mosaic image;
The feature amount extraction means extracts a feature amount from the alignment frame image and the alignment mosaic image,
The relative position determination means determines a relative position between the alignment frame image and the alignment mosaic image;
The live image display means displays a moving image constituted by the display frame image on the display mosaic image as a live image,
The mosaic image generating means estimates a relative position between the frame image and the storage mosaic image at a higher resolution than the registration mosaic image, and pastes the frame image on the storage mosaic image, thereby creating a new storage mosaic. The imaging apparatus according to claim 1, wherein an image is generated.