JP3762382B2 - Image processing method and apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a specific point detection method and apparatus for detecting a specific point of a stationary object such as a landmark from an image.
[0002]
[Prior art]
In recent years, research on mixed reality (hereinafter referred to as MR technology) for the purpose of superimposing and displaying additional information and virtual objects (hereinafter collectively referred to as virtual images) in a real space has been actively conducted. Among them, a video see-through type head-mounted display (hereinafter referred to as “HMD”) is worn by an observer, and a real image captured by a camera built in or attached to the HMD is added to the real space and virtual. A system that superimposes and draws a virtual image in a state where the space is three-dimensionally aligned, and displays a mixed reality image (hereinafter referred to as MR image) generated as a result on the HMD in real time (in this specification, The device will be referred to as the MR system).
[0003]
The alignment of the virtual image and the real image is the biggest technical problem in the MR system, and the realization thereof requires accurate measurement of the position and orientation of the camera viewpoint. In general, if the positions of a plurality of known three-dimensional positions (theoretically 3 points or more, 6 points or more for stable solving) on the captured image can be obtained, the position of the camera viewpoint from the corresponding relationship Azimuth and orientation can be obtained (in the present specification, such a point is referred to as a landmark). That is, the alignment problem is reduced to how accurately the landmark is tracked or detected from the image taken by the moving camera and the position is obtained.
[0004]
The present inventors have so far developed an application device of MR technology in the field of games and the like. These devices were intended for indoor use.
[0005]
For indoor use as described above, a characteristic marker (a characteristic color such as red or green, or a characteristic pattern such as a checkered pattern or concentric circles) Are arranged in the target space and these are used as landmarks, the landmarks can be detected easily and stably by image processing, and high-precision positioning can be realized.
[0006]
As a marker detection method when using a color-based marker, for example, a marker is photographed under a certain illumination environment, a representative color of the marker area in the image is extracted, and the captured image is stored. There is known a method of detecting a marker as an area having the same color (or a color close to it) as the representative color of the marker area. In addition, as a marker detection method in the case of using a marker based on a pattern, for example, each marker is photographed under a certain illumination environment, and a region near the marker in the image is saved as a template image, thereby performing template matching. The marker can be detected by. That is, similarity calculation is performed between the template image and the partial area of the photographed image, and the position of the partial area most similar to the template image is detected as the marker position. In the present specification, image features used as clues for detecting a marker, such as the representative color of the marker area and the template image in the above, are collectively referred to as “detection parameters”.
[0007]
On the other hand, for example, there is an increasing demand for MR systems that are assumed to be used outdoors, such as displaying virtual images of guiders on HMDs to guide university campuses and sightseeing spots.
[0008]
There are many cases where it is difficult to put an artificial marker in the environment outdoors. In such a situation, the method of measuring the position and orientation of the observer's viewpoint is a point having features that can be detected by image processing in the captured image captured by the camera (for example, the corner of the structure, the structure There are known methods that use, as a landmark, a point having a lot of texture inside, a point where the color changes locally, and the like. A template matching technique can be applied to the detection of the landmark from the photographed image.
[0009]
[Problems to be solved by the invention]
However, in the outdoor environment, the appearance (brightness and color) of the landmark changes due to changes in ambient light depending on the weather (sunny / cloudy / rainy) and time zones (morning / daytime / night). For this reason, when trying to detect a landmark by template matching, even if a template image for matching is prepared as a detection parameter in advance, correct matching cannot be performed due to changes in ambient light, There is a problem that it becomes impossible to detect. Therefore, there is a problem that the correct position and orientation of the viewpoint cannot be obtained, and correct alignment between the real image and the virtual image cannot be performed. Further, even when artificial markers are used in an indoor environment, the same problem occurs when the lighting environment changes.
[0010]
The present invention has been made in view of the above-described problems, and even when the appearance of a landmark used as a specific point changes due to changes in the shooting environment, the specific point is reliably detected from the captured image. The purpose is to make it possible.
[0011]
[Means for Solving the Problems]
  According to the invention to achieve the above objectImage processingThe method includes, for example, the following steps. That is,
  Placed in real spacepluralAn image processing method for calculating a posture of a photographing unit using a specific point,
  From captured imagespluralSpecific pointEachHolding step for holding a plurality of detection parameters for detecting
  An input process for inputting a photographed image photographed by the photographing unit;
  The captured imageAverage brightness calculation to calculate the average brightness ofProcess,
  From the plurality of held detection parameters,For each of the plurality of specific points,SaidAverage brightnessA selection step of selecting a detection parameter according to
  A detection step of detecting a specific point from the captured image input by the input step using the detection parameter selected by the selection step;
  A calculating step of calculating the posture of the photographing unit using the position of the detected specific point in the photographed image.
[0012]
  In addition, according to the present invention to achieve the above objectImage processingThe apparatus includes, for example, the following configuration.Ie,
  Placed in real spacepluralAn image processing apparatus that calculates a posture of a photographing unit using a specific point,
  From captured imagespluralSpecific pointEachA holding unit for holding a plurality of detection parameters for detecting
  Enter the image taken by the shooting unitInput sectionWhen,
  The captured imageAverage brightness calculation to calculate the average brightness ofAnd
  From the plurality of held detection parameters,For each of the plurality of specific points,SaidAverage brightnessA selection unit for selecting a detection parameter according to
  A detection unit that detects a specific point from a captured image input by the input unit using the detection parameter selected by the selection unit;
  A calculating unit that calculates the posture of the photographing unit using the position of the detected specific point in the photographed image.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.
[0014]
<First Embodiment>
In the embodiment described below, a template image used for template matching is used as a detection parameter, and this template image is dynamically updated to improve landmark detection accuracy.
[0015]
FIG. 1 is a block diagram illustrating the configuration of an MR system according to the first embodiment. In FIG. 1, reference numeral 101 denotes a fixed camera corresponding to the second photographing means of the present invention, and its installation position, viewpoint orientation, focal length, etc. are fixed so that the same point in the scene is always observed. ing. That is, a captured image obtained from the fixed camera 101 (hereinafter referred to as a fixed viewpoint image I)._SIn the above, the landmark P to be detected_i(Where i is the number of landmarks) is always the same coordinate (x_i, Y_i)
[0016]
A template image creation module 102 is a fixed viewpoint image I._SFrom each landmark P_iTemplate image T corresponding to_iIs generated. There are various methods for generating a template image as will be described later. In this embodiment, the landmark P is used._iObservation coordinates (x_i, Y_i) Is assumed to be known. The template image T_iIs (x_i, Y_i) With a certain range of rectangular area R_iI_SGenerate by extracting from. This template image T_iIs used for template matching processing for landmark detection, as will be described later. This template image T_iIs updated at a predetermined timing, for example, every frame of the fixed camera 101.
[0017]
Reference numeral 110 denotes an HMD worn by an observer, and includes an observer viewpoint camera 111 and a display 112. The observer viewpoint camera 111 is fixed to the HMD 110, and the captured image is an image corresponding to the viewpoint position and direction of the observer (hereinafter referred to as an observer viewpoint image I). Here, the observer camera 111 corresponds to one aspect of the first photographing unit, and this observer viewpoint image corresponds to a target image that is a target for detecting a specific point (landmark).
[0018]
A landmark detection module 113 is a template image T provided from the template image creation module 102._iBy performing a search process by template matching using the image, a landmark P is obtained from the observer viewpoint image I provided by the observer viewpoint camera 111._iIs detected. As described above, since the template image creation module 102 updates the template image at a predetermined timing, the landmark detection module was photographed at almost the same time as the observer viewpoint image I (that is, the observer viewpoint image I). Template matching can be performed using a template image (taken under almost the same light source environment). Accordingly, stable template matching can always be performed even in a situation where the light source environment changes dynamically as in the outdoor environment, and accurate detection of the landmark position can be realized.
[0019]
The landmark detection module 113 further detects the detected landmark P._iCoordinate value (u) on the observer viewpoint image I of_i, V_i) And sent to the viewpoint position estimation module 114. (U_i, V_i) Is the center position of the region that matches the template image.
[0020]
The viewpoint position estimation module 114 is known based on the image coordinate values of the plurality of landmarks provided from the landmark detection module 113 and the positions of the landmarks in the real space that are measured in advance and stored as known information. The viewpoint position and orientation of the observer are calculated by the method. Theoretically, if there are coordinate values of three landmarks on the observer viewpoint image I, the viewpoint position and orientation of the observer viewpoint image can be calculated.
[0021]
The viewpoint position and azimuth orientation calculated as described above are provided to the virtual image generation module 115. The virtual image generation module 115 superimposes and draws the virtual image that will be observed from the viewpoint position and orientation orientation provided from the viewpoint position estimation module 114 on the observer viewpoint image I and displays it on the display 112 of the HMD 110. To do. As a result, an MR image in which the real space and the virtual space are merged based on accurate alignment is displayed on the display 112, and the observer observes it.
[0022]
Assuming that the observer moves outdoors, a unit (fixed portion) including the fixed camera 101 and the template image creation module 102, and a unit including the HMD 110 and the landmark detection module 113 (portion attached to the viewer) ) Is preferably a separate body. In this case, the template image is transmitted from the template image creation module 102 to the landmark detection module 113 by wire or wireless.
[0023]
FIG. 2 is a diagram for explaining the outline of the landmark detection process according to the first embodiment. Reference numeral 201 denotes a fixed viewpoint image I captured by the fixed camera 101._SIn this example, seven landmarks (P₁~ P₇) Is set. As described above, the landmark position (x_i, Y_i) Is known. Therefore, the template image creation module 102 determines each landmark position (x_i, Y_i) And a predetermined area R₁~ R₇Template image T by extracting₁~ T₇Can be generated. In this way, the template image creation module 102 has the latest fixed viewpoint image I at a predetermined timing._STemplate image T using_iIs generated.
[0024]
The landmark detection module 113 generates the latest template image T generated as described above._iIs used to perform template matching on the observer viewpoint image I (202) obtained from the observer viewpoint camera 111 provided in the HMD 110 to detect landmarks.
[0025]
FIG. 3 is a flowchart for explaining the procedure of template image creation processing by the template image creation module 102. First, in step S301, it is determined whether or not it is a template image update timing. In the present embodiment, the update timing of the template image is made to coincide with the frame period of the fixed camera 101, but of course it is not limited to this. For example, the template image is updated every elapse of a predetermined time, the template image is updated every time the fixed camera 101 finishes photographing a predetermined number of frames, and the fixed viewpoint image at the time of the previous template image update and the current It will be apparent that various modifications are possible, such as updating the template image when the difference between the average luminance values of the fixed viewpoint images exceeds a predetermined value, or a combination of these timings.
[0026]
If it is time to update the template image in step S301, the process proceeds to step S302, and the fixed viewpoint image I from the fixed viewpoint camera 101 is reached._SEnter. In step S303, the image I_SFrom among the landmarks P_iA predetermined rectangular area R corresponding to_i(E.g. (x_i-n <x <x_i+ n, y_i-n <y <y_i(x, y)) that satisfies + n; n is a constant) is extracted, and this is extracted as a template image T_iAnd In step S304, the template image T obtained in step S303 is displayed._iIs output to the landmark detection module 113.
[0027]
In step S305, the landmark P_iIt is determined whether or not the template image generation has been completed for all of the above, and if there is an unprocessed landmark, in step S306, the processing target is transferred to the landmark, and the process returns to step S303 to repeat the above processing. When the generation and output of template images for all landmarks are completed, the process returns from step S305 to step S301, and the next update timing is awaited.
[0028]
Through the above processing, the template image updated at a predetermined timing (in this embodiment, in units of frames) is provided to the landmark detection module 113.
[0029]
In the above embodiment, in step S303, the image I_SRectangular region R extracted from_iAs template image T_iHowever, the method for generating the template image is not limited to this. For example, the fixed viewpoint image I in the past plural frames_SMultiple rectangular regions R extracted from_iIs used to create an average image or a weighted average image, and this is used as a template image T_IIt is good. In this case, the fixed viewpoint image I_SIt can be expected to remove the noise component contained in.
[0030]
In the above embodiment, in step S304, all the template images generated in step S303 are output. However, the template image output method is not limited to this. For example, the last output template image T_i'And the template image T generated in step S303_iWhen the difference e is greater than a certain value (e ≧ TH)₁), It may be determined that the light source environment has changed, and the template image may be output. In this case, network traffic can be reduced by omitting unnecessary data transmission. In addition, a barrier enters between the landmark and the fixed camera 101, and the fixed viewpoint image I_SWhen the landmark is not observed above, in order to prevent the template image from being updated to an incorrect image obtained by capturing the obstacle, the difference is greater than or equal to a certain value (e ≧ TH₂) May determine that the landmark is concealed and not output the template image. For calculating the degree of difference between template images, a known image processing method such as cross-correlation or the sum of absolute differences of pixel values can be used.
[0031]
Next, processing by the landmark detection module 113 will be described. FIG. 4 is a flowchart for explaining a landmark detection procedure by the landmark detection module.
[0032]
Steps S401 and S402 are executed by the template image T from the template image creation module 102 described above._iIs output in a memory for use in template matching. In this embodiment, every time one template image is obtained in FIG. 3 described above, the template image is output (steps S303 and S304). Therefore, the template image is updated in steps S401 and S402 by one template image. Will be done every time. However, the template image update procedure is not limited to this. For example, in the template image creation module 102, the fixed viewpoint image I_SIf the template images for all the landmarks included in the template are generated and then output in a batch, the landmark detection module 113 updates all the template images in a batch. It will be.
[0033]
If no template image has been received in step S401, or after step S402 is completed, the process proceeds to step S403, where it is determined whether an observer viewpoint image I has been input. As described above, the observer viewpoint image I is image data output from the observer viewpoint camera 111, and a landmark is detected from the observer viewpoint image I by the processing in steps S404 to S407. Therefore, in this embodiment, each time an observer viewpoint image is input from the observer viewpoint camera 111 (that is, every frame), the landmark is detected.
[0034]
In step S404, the template image T_iThe landmark P from the observer viewpoint image I using_iIs detected. Any known template matching method may be used for this detection process. For example, each pixel (u_j, V_j) For each template image T around the pixel._iAn area of the same size as the partial image Q_jExtracted as a partial image Q_jAnd template image T_iDifference e between_jIs calculated. As a method of calculating the degree of difference, a cross-correlation between both images may be obtained, or the sum of absolute values of luminance values of corresponding pixels may be used, or the input image is a color image May be the sum of the RGB distances between the corresponding pixels. All pixels (u_j, V_j) Partial image Q_jAnd template image T_iDifference e between_jAnd the difference e_j(Ie, the template image T_iPartial image Q that most closely matches_jCenter coordinates (u_j, V_j)), The landmark P in the observer viewpoint image I_iDetection position (u_i, V_i).
[0035]
In step S405, coordinates (u_i, V_i) To the landmark P in the observer viewpoint image I_iIs output to the viewpoint position estimation module 114. In step S404, the template image T is added to the observer viewpoint image I._iIs determined that there is no matching part (for example, all the differences e_j) Exceeds the set threshold)_iIs output on the observer viewpoint image I, or this processing is skipped. In step S406, all landmarks P_iIt is determined whether or not the detection process has been completed. If there is still an unprocessed landmark, the process proceeds to step S407 and an unprocessed landmark P_iThe process from step S404 is repeated. All landmarks P_iWhen the process is finished, the process returns to step S401.
[0036]
Note that the present invention further increases the effect by operating the template image creation module 102 and the landmark detection module 113 in synchronization. That is, after receiving the template image in step S401, in step S403, the fixed viewpoint image I that is the origin of the received template image._SBy inputting the observer viewpoint image I photographed at the same time, template matching using a template image photographed under the same light source environment as the observer viewpoint image I becomes possible. Needless to say, in order to strictly realize this processing, it is desirable that the imaging of the fixed camera 101 and the observer viewpoint camera 111 be electrically synchronized.
[0037]
In the above embodiment, the detection process is performed for all the landmarks. However, the process may be terminated when a predetermined number of landmarks that enable calculation of the observer viewpoint position are detected.
[0038]
In the above processing, the template image creation module 102 outputs the updated template image to update the template image in the landmark detection module 113. The landmark detection module 113 uses the template as necessary. The latest template image stored in the image creation module 102 may be read. The reading timing is, for example, every time the observer viewpoint image I is input, every predetermined time interval, or the like. In this case, the template image creation module 102 holds the template image created in its own storage medium, and the latest template image is transmitted from the template image creation module 102 to the landmark detection module 113 in response to a request from the landmark detection module 113. Is done.
[0039]
In step S404, the entire observer viewpoint image I is scanned and the landmark P is scanned._iHowever, it is possible to apply various known techniques for improving the efficiency of the template matching process. An example is as follows.
[0040]
FIG. 5 is a diagram for explaining a method for limiting the search area during the landmark detection process. For each landmark, information such as the position and orientation of the observer camera in the previous frame (or past frame) of the observer viewpoint image I and the detection position of the landmark in the previous frame (or past frame) is used. Next, an approximate position in the observer viewpoint image I of the current frame is estimated, and a search area is set in the surrounding area. Of course, position data from the immediately preceding viewpoint position estimation module 114 may be used. The landmark P including the search area in the observer viewpoint image I of the current frame_iOnly for, search processing within the search area is performed. If it demonstrates in the example of FIG. 5, the landmark P shown in (a) will be described.₁~ P₇Are obtained as shown in (b) with respect to the observer viewpoint image I. In this case, in step S404, P_Three~ P_FiveAll search areas and P₂The corresponding landmark is searched for the portion included in the observer viewpoint image I of the search area. That is, the processing speed can be increased by narrowing down the search range.
[0041]
As described above, according to the first embodiment, since the template image is updated using the image captured by the fixed camera 101, a template image corresponding to the environment can be obtained following the change in the environment. Can do. For this reason, since it is possible to reliably detect the landmark from the observer viewpoint image I regardless of changes in the environment, it is possible to accurately determine the position and orientation of the observer's viewpoint in the outdoor environment. . Therefore, it is suitable for alignment between the real space and the virtual space, particularly when displaying an MR image on the display 112 provided in the HMD 110.
[0042]
In this embodiment, the position of each landmark in the fixed viewpoint image 201 is known. For example, the position of each landmark is stored in a memory (not shown) of the template image creation module, acquired as necessary, and stored in the template image creation module 102. Shall be supplied. In addition to this, the following method can be used as means for supplying the landmark position. That is, the operator may directly specify the position of the landmark on the fixed viewpoint image 201 by an input unit (not shown), or the position of each landmark in the three-dimensional space measured by some method and the camera of the fixed camera 101. Parameters (including at least position and orientation) are stored in a memory, and based on this information, a fixed viewpoint image 201 is displayed by a landmark position calculation unit (not shown) (corresponding to a specific point position calculation unit of the present invention). The position of each landmark on the top may be calculated. In addition, when a landmark to be detected is not determined in advance and it is only necessary to track some feature point in the observer image 202, the feature extraction unit (not shown) starts from the fixed viewpoint image 201 at an initial time. A feature point having a remarkable image feature (for example, an edge portion or a strong texture portion) is automatically extracted, and this position may be used as a landmark position.
[0043]
<Second Embodiment>
In the first embodiment, since the template image is updated by one fixed camera, the acquisition range of the template image is limited, and the movement and / or look-around range of the observer is limited. Therefore, in the second embodiment, a plurality of fixed cameras are installed so that the observer can move and / or look around in a wide range. However, since a plurality of fixed cameras are used, when there are a plurality of template images for one landmark (hereinafter referred to as “with overlap”), one fixed camera is used for one landmark. There is a case where only one template image is present by assignment (referred to as “no overlap”). In the second embodiment, a case where there is no overlap will be described, and a case where there is an overlap will be described in the third embodiment.
[0044]
When there is no overlap, the MR system provided with a plurality of fixed cameras can be realized with a configuration similar to that of the first embodiment. FIG. 6 is a block diagram showing the configuration of the MR system according to the second embodiment. In other words, the template image creation module 602 determines a predetermined region R from each of a plurality of fixed viewpoint images obtained from a plurality of fixed cameras 601._iIs extracted from the template image T_iOutput as.
[0045]
As in the first embodiment, the landmark detection module 613 updates the template image to be used with the template image transmitted from the template image creation module 602, and uses the template image to calculate the landmark image from the observer viewpoint image I. Perform mark detection. The camera selection module 616 selects a predetermined number of fixed cameras near the viewpoint position obtained from the viewpoint position estimation module 614 and notifies the landmark detection module 613 of the selection result. As will be described later, in the second embodiment, which fixed camera the camera selection module 616 uses based on the viewpoint position output from the viewpoint position estimation module 614 in order to improve the processing efficiency. To decide. The landmark detection module 613 performs template matching for landmark detection using the determined template image from the fixed camera.
[0046]
The virtual image generation module 115 and the HMD 110 are as described in the first embodiment.
[0047]
FIG. 7 is a diagram for explaining the outline of the landmark detection process according to the second embodiment. Each fixed viewpoint image I obtained by a plurality of fixed cameras 601 (A to E)._S1~ I_S5Landmark P above₁~ P₁₃The observation position is determined, and the surrounding rectangular area R₁~ R₁₃Template image T corresponding to each by extracting₁~ T₁₃Is generated. And what is necessary is just to detect a landmark from the observer viewpoint image I using these template images. The processing in this case is essentially the same as the case where there is one fixed camera, and it can be considered that the angle of view of one camera is widened, and the processing procedure described in FIGS. 3 and 4 is used. Can detect landmarks.
[0048]
As described above, in the second embodiment in which a plurality of fixed cameras are provided, the observer can perform the same processing as that in the first embodiment (that is, even in the configuration in which the camera selection module 616 in FIG. 6 does not exist). The position and orientation of the viewpoint can be detected. However, since the number of landmarks increases, the processing efficiency decreases if detection processing is performed for all landmarks each time. Therefore, in the second embodiment, processing efficiency is improved by limiting the number of landmarks to be detected in the landmark detection module 613 in advance. That is, the landmarks to be detected are narrowed down only to the landmarks observed by the fixed camera selected by the camera selection module 616.
[0049]
This can be realized, for example, by adding step S801 before step S404 as shown in FIG. 8 in the process shown in FIG. When the observer viewpoint image I is input, the process proceeds from step S403 to step S801, and the landmark P_iIs observed by the fixed camera selected by the camera selection module 616. Where landmark P_iIf it is not observed with the selected fixed camera, the landmark detection process (steps S404 and S405) is skipped, and the process proceeds to step S406 to detect the next landmark. On the other hand, landmark P_iIf it is observed by the selected fixed camera, the process proceeds to step S404 to detect the landmark.
[0050]
Also in the second embodiment, various well-known methods for improving the efficiency of the template matching process can be applied. For example, the technique for limiting the search area described in the first embodiment is also effective. In particular, by specifying the search area after limiting the template image to be used as described above, unnecessary position calculation of the search area can be eliminated, which is effective.
[0051]
FIG. 9 is a diagram for explaining a method for limiting the search area of the template image during the landmark detection process in the second embodiment. For example, assume that the camera selection module 616 selects the fixed cameras A, B, and C shown in FIG. 7 based on the detected viewpoint position. In this case, the object of detection is the landmark P₁~ P₈And other landmarks P₉~ P₁₃Is not considered. In step S404, these landmarks P₁~ P₈Of which the search area is included in the observer viewpoint image (P in the figure)₂~ P₆) Only for the corresponding template image T₂~ T₆A landmark detection process is performed by template matching using.
[0052]
As described above, according to the second embodiment, since the template image is updated using a plurality of fixed cameras, a wider range of movement of the observer is allowed.
[0053]
<Third Embodiment>
Next, a case where a plurality of template images exist at one landmark at one time point due to the provision of a plurality of fixed cameras, that is, a case where there is an overlap will be described.
[0054]
FIG. 10 is a diagram for explaining the outline of the landmark detection process when there is an overlap according to the third embodiment. Landmark P for fixed camera F₁And P₂Is observed, and a rectangular region R is defined around it.₁ ^F, R₂ ^FTemplate image T₁ ^F, T₂ ^FIs generated. The fixed camera G has a landmark P.₁~ P_ThreeIs observed, and a rectangular region R is defined around it.₁ ^G~ R_Three ^GTemplate image T₁ ^G~ T_Three ^GIs generated. Similarly, from the fixed camera H, the template image T₁ ^H~ T_Three ^HIs obtained. Here, for example, T₁ ^FAnd T₁ ^GAnd T₁ ^HIs the same landmark P in space₁Is a template image corresponding to.
[0055]
As described above, when a plurality of template images are obtained with respect to one landmark by different fixed cameras, it is necessary to determine which template image is used to detect the landmark. In the following, two cases will be described: (1) a case where the best template matching result is used, and (2) a case where a template image obtained by a fixed camera selected based on the observer position is used. In the third embodiment, for example, it is assumed that template images acquired from captured images obtained by the cameras F, G, and H are stored as shown in FIG. For example, from the photographed image of camera F, landmark P₁~ P₆Template image T₁ ^F~ T₆ ^FHowever, from the image taken by camera G, landmark P_Three~ P₈Template image T_Three ^G~ T₈ ^GHowever, from the image taken by camera H, landmark P_Three~ P₈Template image T₇ ^H~ T₁₂ ^HAre acquired and stored. Here, the same landmarks after the subscript number are the same landmarks. For example, landmark P₆The template images are acquired from the captured images of the camera F and the camera G.
[0056]
(1) When using the best template matching result
FIG. 11 is a flowchart for explaining a procedure in a case where landmark detection is performed using the best matching result when a plurality of template images exist in the same landmark. FIG. 11 shows a process that replaces the step S404 in FIG.
[0057]
When the observer viewpoint image I is input in step S403, the landmark P obtained by the fixed camera j in step S1100._iTemplate image T_i ^jThe landmark P from the observer viewpoint image I using_iIs detected. In step S1101, the landmark P_iHas a plurality of template images, and it is determined whether or not the coordinates have already been calculated by another template image. If the coordinates are not calculated by another template image, or if there are not a plurality of corresponding template images, the coordinate value obtained by the template image and the matching degree are stored in the memory in step S1104.
[0058]
On the other hand, if the coordinates have already been output by another template image, the process proceeds to step S1102, and the matching result based on the other template image stored in the memory is compared with the matching result based on the current template image. If the matching with the current template image is a better result (when the matching degree is higher), the process proceeds to step S1103, and the coordinates stored in the memory of the landmark are used as the current template image. Replace with the coordinate value and matching degree obtained. For example, T₆ ^GWhen matching is already done for T₆ ^FIf the matching using is performed and the matching degree is stored, T₆ ^GMatching degree when T is used and T₆ ^FThe matching degree when using is compared, and the one with the higher matching degree is adopted.
[0059]
Next, in step S1105, the landmark P_iAll template images T corresponding to_i ^jIf the processing has not been completed, the process proceeds to step 1106, and the unprocessed template image T_i ^jAre repeated as the processing target. On the other hand, landmark P_iAll template images T corresponding to_i ^jIf the processing is finished, the process proceeds to step S405, and the coordinates stored in the memory are changed to the landmark P._iIs output to the landmark detection module. As described above, when all the template images are processed, when there are a plurality of template images in one landmark, the coordinate value based on the template image having the best matching degree is adopted. Become.
[0060]
(2) When using a template image obtained by a fixed camera selected based on the observer position
FIG. 12 is a flowchart for explaining a procedure for performing landmark detection using a template image obtained by a fixed camera selected based on the observer position when a plurality of template images exist in the same landmark. is there. FIG. 12 shows a process added before step S404 in FIG.
[0061]
When the observer viewpoint image I is input in step S403, in step S1201, a landmark P to be detected from now on._iIt is determined whether or not there are a plurality of template images. If a plurality of template images do not exist, only one template image exists for the landmark, so the process proceeds to step S404, and landmark detection is performed by template matching.
[0062]
On the other hand, if there are a plurality of template images, in step S1202, a template image obtained from a fixed camera closest to the observer position is selected from the plurality of template images, and this is used for the detection process. T_iThen, the process proceeds to step S404. For example, in FIG. 16, if the observer position is closer to the camera G than the camera F, the landmark P_Three~ P₆Is a template image T obtained from an image taken by the camera G_Three ^G~ T₆ ^GIs adopted.
[0063]
By performing processing for all the template images as described above, when a plurality of template images exist for one landmark, the template image from the fixed camera closest to the observer position is adopted, and the land Mark detection is performed.
[0064]
As described above, according to the third embodiment, when a plurality of template images obtained from a plurality of fixed cameras exist in one landmark, it is possible to select an appropriate template image. In particular, as shown in FIG. 10, a template image obtained from each of a plurality of fixed viewpoint images obtained by photographing one landmark from different directions can be used appropriately. Template matching can be performed appropriately even when the appearance of the image is greatly different (for example, when it has a three-dimensional shape or a reflection characteristic close to a mirror surface).
[0065]
Note that the camera selection module 616 as described in the second embodiment can be used together. In this case, the landmarks to be processed in FIGS. 11 and 12 are only the landmarks obtained from the fixed camera selected by the camera selection module 616.
[0066]
In the third embodiment, it is needless to say that various known techniques for improving the efficiency of the template matching process can be applied.
[0067]
<Fourth Embodiment>
In the first to third embodiments, a template image used for template matching performed by the landmark detection module 113 is updated by creating a template image as needed from a fixed viewpoint image obtained using a fixed camera. . According to this method, a template image is generated by using images taken at each time point. At that time, the appearance of the landmark is reflected in the template image, and good template matching can be performed. it can. However, one or a plurality of fixed cameras must be prepared, which increases the device scale. Therefore, in the fourth embodiment, a plurality of types of template images are registered in advance for one landmark, and the template image is updated using them.
[0068]
FIG. 13A is a block diagram showing a configuration of an MR system according to the fourth embodiment. A template image storage unit 1301 registers a plurality of types of template images 1310 for each of a plurality of landmarks. A template image selection module 1302 selects one template image for each landmark from among a plurality of template images stored in the template image storage unit 1301. In this example, a template image to be used is selected based on the average luminance value by the average luminance value calculation module 1303 from the image captured at that time by the observer viewpoint camera 111 mounted on the HMD 110 (detailed later). Therefore, as shown in FIG. 13B, the template image storage unit 1301 classifies and stores the template images to be used according to the luminance value range. Note that since the luminance value for changing the template image differs for each landmark, as shown in FIG. 13B, the same template image may be used even if the luminance value range is different. For example, landmark # 1 has the same template image T in both luminance value ranges B and C._1BIs used.
[0069]
The landmark detection module 1313 performs template matching on the observer viewpoint image I using the template image acquired by the template image selection module 1302, and detects a landmark. The viewpoint position estimation module 114, the virtual image generation module 115, and the HMD 110 are as described in the first embodiment (FIG. 1).
[0070]
The average luminance value calculation module 1303 obtains an average luminance value from the captured image from the observer viewpoint camera 111 attached to the HMD 110 and provides the calculation result to the template image selection module 1302. The template image selection module 1302 selects a template image of each landmark from the template image storage unit 1301 based on this average luminance value, and outputs it to the landmark detection module 1313.
[0071]
FIG. 14 is a flowchart for explaining the processing procedure of the template image selection module according to the fourth embodiment. First, in step S1401, the average luminance value is taken from the average luminance calculation module 1303. In step S1402, it is determined whether the luminance value range has been changed. For example, when the luminance value range of the template image currently used is the range A, it is determined whether or not the average luminance value captured in step S1401 belongs to another luminance value range (B or C). If the brightness value range has changed, the process advances to step S1403 to read a template image group corresponding to the brightness range to which the new average brightness value belongs. In step S1404, the template image group is output to the landmark detection module 1313.
[0072]
As described above, according to the fourth embodiment, an appropriate image is selected from a plurality of types of template images prepared in advance and used for template matching without using a fixed camera. , Accurate template matching can be realized.
[0073]
Note that switching of template images is not limited to the average luminance value, and may be executed according to morning, noon, and night time zones. Alternatively, the observer can manually input weather conditions such as clear, cloudy, rain, etc., and the template image selection module 1302 can switch the template image accordingly.
[0074]
In the above example, a template image is selected from one template image group, but a plurality of template image groups should be prepared corresponding to landmarks observed from a plurality of positions. A template image group may be selected, and a template image may be acquired from the selected template image group according to the average luminance value. In this case, a plurality of template image groups can be considered in association with the plurality of fixed cameras of the second and third embodiments described above. Therefore, the template image group can be selected from the position of the observer.
[0075]
Furthermore, it goes without saying that the search range in template matching can be narrowed down (for example, the method described with reference to FIG. 5 of the first embodiment).
[0076]
<Fifth Embodiment>
In the first to third embodiments, the template image is defined as the detection parameter and the template matching is used for the landmark detection. However, the template matching is not necessarily used for the landmark detection. For example, when a marker using a color feature (color marker) is used as a landmark, the landmark is detected by defining a color parameter representing the color feature of the marker as a detection parameter and extracting a specific color region. Can do.
[0077]
FIG. 15 is a block diagram illustrating the configuration of the MR system according to the present embodiment. In FIG. 15, the fixed camera 101, the HMD 110, the observer camera 111, the display 112, the viewpoint position estimation module 114, and the virtual image generation module 115 are the same as those in the first embodiment.
[0078]
Reference numeral 1502 denotes a color parameter extraction module, which is a fixed viewpoint image I._SFrom each landmark P_iParameter C for detecting_iIs generated. For example, the fixed viewpoint image I_SLandmark P above_iObservation region R_iBased on the distribution in the RGB color space of each pixel in the pixel (assumed to be supplied by supply means not shown and not shown in the present embodiment), the existence range of landmarks in the RGB color space (minimum red value) Rmin, maximum red value Rmax, minimum green value Gmin, maximum green value Gmax, minimum blue value Bmin, maximum blue value Bmax)), and this is the color parameter C representing the color characteristics of the landmark_iAnd This color parameter C_iIs output to a landmark detection module described later at a predetermined timing.
[0079]
A landmark detection module 1513 is a color parameter C provided from the color parameter extraction module 1502._iFrom the observer viewpoint image I, the color parameter C_iBy extracting the pixels included in the color area defined by_iIs detected. As described above, the fixed camera image I photographed at almost the same time as the observer viewpoint image I (that is, photographed in almost the same light source environment as the observer viewpoint image I)._SColor parameter C based on_iTherefore, even in a situation where the light source environment changes dynamically as in an outdoor environment, stable color marker detection can always be performed, and accurate detection of the landmark position can be realized. In this embodiment, the color parameter C_iAs described above, although the landmark existence range in the RGB color space is used, it is needless to say that any color space or color feature extraction method generally used for color feature extraction may be used. It may be a parameter. Also, the types of detection parameters are not limited to template images and color features, and any detection parameter for detecting a landmark from an image may be used.
[0080]
<Sixth Embodiment>
In the first to fifth embodiments, there is one observer viewpoint camera for which the landmark position on the captured image is to be detected. However, the observer viewpoint camera is not necessarily one. For example, there are observer viewpoint cameras 111 </ b> A to 111 </ b> D corresponding to a plurality of observers (here, four persons A to D), and observer viewpoint images I photographed by them._A~ I_DWhen detecting the landmark positions on the upper side, the corresponding landmark detection modules 113A to 113D are provided, and the template image creation module 102 having the same configuration as that of the first to fourth embodiments is used. The template image may be updated for each of the landmark detection modules 113A to 113D.
[0081]
As described above, according to each of the above-described embodiments, it is possible to accurately detect a landmark from a captured image even if the environment at the time of shooting changes and the appearance of a specific point changes. In addition, according to each embodiment, accurate landmark detection is assured with respect to environmental changes, so that in MR technology, virtual and real high-precision alignment and free movement outdoors. Coexistence can be achieved.
[0082]
In the first to sixth embodiments, the application to the video see-through type MR system has been described. However, it is of course possible to apply to the use that requires the measurement of the viewpoint position, for example, the optical see-through type MR system. Any application other than MR can be applied as long as it is an application that detects the coordinates of a specific part of a stationary object from an image captured by a camera.
[0083]
【The invention's effect】
As described above, according to the present invention, it is possible to reliably detect a specific point from a captured image even if the environment at the time of shooting changes and the appearance of the specific point changes.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an MR system according to a first embodiment.
FIG. 2 is a diagram illustrating an outline of landmark detection processing according to the first embodiment.
FIG. 3 is a flowchart for explaining a procedure of template image creation processing by a template image creation module 102;
FIG. 4 is a flowchart illustrating a landmark detection procedure performed by a landmark detection module.
FIG. 5 is a diagram illustrating a method of limiting a search area during landmark detection processing.
FIG. 6 is a block diagram showing a configuration of an MR system according to a second embodiment.
FIG. 7 is a diagram illustrating an outline of landmark detection processing according to a second embodiment.
FIG. 8 is a flowchart for explaining processing in a case where a landmark to be detected is limited in the second embodiment.
FIG. 9 is a diagram illustrating a method of limiting a search area during landmark detection processing in the second embodiment.
FIG. 10 is a diagram for explaining an overview of landmark detection processing when there is an overlap according to the third embodiment;
FIG. 11 is a flowchart for explaining a procedure when landmark detection is performed using the best matching result when there are a plurality of template images in the same landmark.
FIG. 12 is a flowchart illustrating a procedure for performing landmark detection using a template image obtained by a fixed camera selected based on an observer position when a plurality of template images exist in the same landmark. is there.
FIG. 13A is a block diagram showing a configuration of an MR system according to a fourth embodiment.
FIG. 13B is a diagram illustrating a data configuration example of a template image.
FIG. 14 is a flowchart illustrating a processing procedure of a template image selection module according to the fourth embodiment.
FIG. 15 is a block diagram illustrating a configuration of an MR system according to a fifth embodiment.
FIG. 16 is a diagram illustrating a storage state of a template image according to the third embodiment.

Claims (5)

An image processing method for calculating a posture of a photographing unit using a plurality of specific points arranged in a real space,
A holding step for holding a plurality of detection parameters for detecting each of a plurality of specific points from the captured image;
An input process for inputting a photographed image photographed by the photographing unit;
An average luminance calculating step for calculating an average luminance of the captured image;
A selection step of selecting a detection parameter corresponding to the average luminance for each of the plurality of specific points from the plurality of detection parameters held;
A detection step of detecting a specific point from the captured image input by the input step using the detection parameter selected by the selection step;
And a calculation step of calculating an attitude of the photographing unit using a position of the detected specific point in the photographed image. The detection parameter is a template image. 1 The image processing method as described. In response to said average luminance, an image processing method according to claim 1 or 2, characterized in that to control the execution of the selected detection parameters by the selection step. An image processing apparatus that calculates an attitude of a photographing unit using a plurality of specific points arranged in a real space,
A holding unit that holds a plurality of detection parameters for detecting each of a plurality of specific points from a captured image;
An input unit for inputting a photographed image taken by the photographing unit ;
An average luminance calculation unit for calculating an average luminance of the captured image;
A selection unit that selects a detection parameter corresponding to the average luminance for each of the plurality of specific points from the plurality of detection parameters held;
A detection unit that detects a specific point from a captured image input by the input unit using the detection parameter selected by the selection unit;
An image processing apparatus comprising: a calculation unit that calculates an attitude of the photographing unit using a position of the detected specific point in a photographed image. A storage medium storing a control program for causing a computer to execute an image processing method for calculating a posture of a photographing unit using a plurality of specific points arranged in a real space, the image processing method comprising:
A holding step for holding a plurality of detection parameters for detecting each of a plurality of specific points from the captured image;
An input process for inputting a photographed image photographed by the photographing unit;
An average luminance calculating step for calculating an average luminance of the captured image;
A selection step of selecting a detection parameter corresponding to the average luminance for each of the plurality of specific points from the plurality of detection parameters held;
A detection step of detecting a specific point from the captured image input by the input step using the detection parameter selected by the selection step;
And a calculation step of calculating an attitude of the photographing unit using a position of the detected specific point in the photographed image.

Images