JP2007233971A - Image compositing method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To visualize measurement areas of a position attitude sensor and an objective camera. <P>SOLUTION: A three-dimensional computer graphics model, representing measurement areas of the position attitude sensor and the objective camera, is prepared previously. A composite image formed, by superimposing the measurement areas of the position attitude sensor and the objective camera to an actual image in a real space, in which the position attitude sensor is arranged, is produced as an image for a positioned virtual object. When the composite image is displayed, the measurement areas of the position attitude sensor and the objective camera can be visualized. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image composition technique for aligning and superimposing a computer generated image (CG: Computer Graphics) on a real world image.

  A virtual object exists in the real space as if it were in the real space by aligning and displaying the image of the virtual object generated by computer graphics (hereinafter referred to as CG) on the background of the real image. There is a mixed reality (MR) technology that allows the user to feel as if he / she is doing (see, for example, Patent Document 1).

  There are various application examples of MR technology. For example, a medical assistance application in which a built-in CG is aligned and superimposed on a live-action image of a patient's body surface to present to a doctor as if they are seeing through the body. In addition, examples of the use of assisting the assembly work by superimposing and displaying the assembly procedure on the actual image of the product at the factory in CG, and the guidance application for superimposing and displaying the name of the place and name on the image of the actual scenery, etc. can do.

  Note that the background on which the CG is superimposed does not necessarily have to be a captured image, and the same effect can be obtained by displaying only CG on a translucent display member (such as a half mirror) disposed on the observer's line of sight. Realized. Such a method is called an optical see-through method. On the other hand, a method using a real image as a background is called a video see-through method.

Japanese Patent Laid-Open No. 11-84307

  In order for the observer to experience mixed reality without a sense of incongruity, the alignment accuracy between the real space serving as the background and the CG superimposed thereon is important. For this reason, the viewpoint position and orientation of the observer are usually measured, and the CG is generated by using the measured viewpoint position and orientation and the three-dimensional model of the virtual object, thereby realizing superimposition of CG with less discomfort. Yes.

  Since the observer's viewpoint position and orientation is information having six degrees of freedom, there are only a limited number of sensors (position and orientation sensors) that can be used for the measurement, and magnetic sensors, optical sensors, and image sensors are generally used. ing.

  However, sensors for position and orientation measurement, including magnetic sensors and image sensors, have a preset measurement range, and can measure the position and orientation only within the measurement range. For this reason, the range in which the observer can act is limited by the arrangement of the sensors.

  Therefore, how to arrange a sensor becomes a big subject when installing a sensor in the field. However, since information such as magnetism and light used by the sensor cannot be grasped visually, the measurement range of the sensor cannot be visually observed, and eventually the sensor must be arranged by relying on experience. However, the arrangement of the sensors relying on intuition may not be able to ensure the planned measurement range, and in such a case, it is necessary to perform sensor installation adjustment again, which is inefficient.

  In addition to the sensor, an objective camera (a camera that captures images from the viewpoint position and orientation of a third party) may be used in addition to the sensor for the purpose of obtaining information for correcting the detection information of the sensor. However, like the sensor, since the range that can be photographed (measured) by the camera is limited, it is preferable that the measurable range of the objective camera can also be visualized.

  The present invention has been made in view of the above-described problems of the prior art, and an object of the present invention is to enable efficient setting and management of an image composition system by visualizing the measurement range of a sensor or an objective camera. .

  The above-described object is to provide a first setting step of setting a three-dimensional computer graphics model representing a measurement range of an objective camera whose position and orientation are known in an image composition method for aligning and synthesizing a computer graphics image with a photographed image. A viewpoint position / orientation information acquisition step for acquiring viewpoint position / orientation information, a CG image generation step for generating a computer graphics image representing the measurement range of the objective camera using the viewpoint position / orientation information and a three-dimensional computer graphics model; The image synthesis method of the present invention is characterized by comprising an image synthesis step of synthesizing a real image and a computer graphics image.

  In addition, the above-described object is an image composition method for aligning and synthesizing a computer graphics image with a real image using position and orientation information measured using a sensor and an objective camera each having a predetermined measurement range. A setting process for setting a three-dimensional computer graphics model representing a predetermined measurement range of the sensor and the objective camera, and position and orientation information of the camera that captures a real image from the detection information of the sensor and the image captured by the objective camera Camera position / orientation information acquisition step for acquiring image data, and CG image generation step for generating a computer graphics image representing a predetermined measurement range of the sensor and objective camera using the camera position / orientation information and the three-dimensional computer graphics model A live-action image and a computer graphics image Also achieved by an image synthesizing method according to the present invention characterized by having an image synthesis process.

  Furthermore, the above-described object is to use a 3D computer graphics model of a virtual object and camera position / orientation information prepared in advance for a real image taken by a camera provided in a head-mounted display device worn by an observer. 3 represents the measurement range of each of the sensor and the objective camera arranged in the space where the observer exists in the image composition method for compositing the generated virtual object images and displaying the composite image on the head-mounted display device. The present invention is also achieved by an image composition method of the present invention comprising a setting step for setting a three-dimensional computer graphics model and an image generation step for generating a measurement range as an image of a virtual object.

  Furthermore, the above-described object is also achieved by a program for causing a computer to execute the image composition method of the present invention and a computer-readable recording medium storing the program.

  With the above configuration, according to the present invention, it is possible to visualize a measurement range invisible to a sensor or an objective camera, and it is possible to efficiently set and manage an image composition system.

<First Embodiment>
Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings.
An example of the image composition device according to the first embodiment acquires position information of a sensor based on a marker attached to the sensor, and superimposes and displays the measurement range of the sensor on the position using 3D CG model data. Visualize the measurement range of the sensor. Note that the 3D CG model data for visualizing the measurement range may be model data corresponding to the measurement range according to the sensor specifications, or may be a measurement range obtained through experience. The model data display method is not particularly limited, and may be output as it is, or may be displayed after setting transparency.

(Configuration of image composition device)
FIG. 1 is a block diagram illustrating a configuration example of an image composition device according to the present embodiment. FIG. 2 is a diagram schematically showing a specific device arrangement when the image synthesizing apparatus of FIG. 1 is applied to a mixed reality application, and a mixed reality sensed by an observer.

  In FIG. 1, a small video camera 101 captures a real space. As shown in FIG. 2, the video camera 101 has a head-mounted image display device (Head Mounted Display, hereinafter referred to as an HMD) 108 in a portion close to both eyes of the observer 201 in a direction that matches the line of sight of the observer. It is fixed. Instead of using the right-eye and left-eye video cameras 101, a monocular video camera is used, or a video camera is attached to a display device (Hand Held Display, hereinafter referred to as HHD) that is held by an observer. Also good.

  In the following, in order to simplify the explanation and understanding, even if the processing is performed for each of a pair of images taken by the right-eye video camera and the left-eye video camera, it will be described as processing for one image. There is. However, when using a pair of video cameras (compound-eye cameras), the same processing is performed on the images captured by each video camera to generate a pair of composite images. For example, the right-eye display device or the left-eye included in the HMD Displayed on the display device.

  For example, an image input unit 102 that is a video capture board inputs a video (image) taken by the video camera 101. In FIG. 2, it is assumed that the observer is looking on a desk that is a real object. An image signal captured by the video camera 101 is processed by the image input unit 102 and supplied to the image composition unit 107 and the marker detection unit 103 as digital image data.

  The marker detection unit 103 detects the two-dimensional marker 205 shown in FIG. 2 from the image supplied from the image input unit 102. The two-dimensional marker is an index having a feature (color, shape, pattern, etc.) that can be extracted from an image, whose absolute position in the real space is known. Therefore, the marker detection unit 103 detects the two-dimensional marker 205 by searching the image for an area that matches the characteristics of the two-dimensional marker 205.

  The viewpoint position / orientation calculation unit 104 calculates the three-dimensional position and orientation of the HMD 108 mounted on the observer's head 202. For this calculation, the marker information (position in the image, direction of the marker 205, area, etc.) detected by the marker detection unit 103, the position of the feature point of the marker 205 in the marker coordinate system, the camera parameters of the video camera 101, and the like. Use. Details of processing of the viewpoint position and orientation calculation unit 104 will be described later.

  The image generation unit 105 generates a CG image to be observed from the viewpoint of the observer based on the position / orientation data of the HMD 108 calculated by the viewpoint position / orientation calculation unit 104 from the three-dimensional CG model data registered in the virtual space database 106. Is generated. The virtual space database 106 is, for example, a hard disk drive. The generated CG image is supplied to the image composition unit 107. In the generation of the CG image in the image generation unit 105, geometric information of the CG model, attribute information such as color and texture, and illumination information included in the virtual space database 106 are used. Since generation of a three-dimensional CG image is a known technique, a detailed description thereof is omitted.

  The image synthesis unit 107 synthesizes the real image from the image input unit 102 and the CG image (image of the virtual object 206) from the image generation unit 105, and outputs the synthesized image to the HMD 108. When the observer views the composite image displayed on the HMD 108, it appears as if the virtual object 206 is on the desk 204.

  Of these image composition apparatuses, functional blocks excluding the video camera 101 can be realized by a general-purpose computer apparatus equipped with a video capture board. Therefore, at least a part of the processing in the image composition apparatus described below can be realized by software by the CPU executing the control program.

(Visualization of sensor measurement range)
In the present embodiment, the measurement range of the position and orientation sensor is visualized using the image composition apparatus described above.
FIG. 3 is a diagram schematically showing visualization of the measurement range of the position and orientation sensor in the present embodiment. The observer 201 images the two-dimensional marker 205 attached to the pair of sensors 301 by using, for example, the video camera 101 attached to the HMD 108. Then, the three-dimensional model data of the measurement range of the sensor 301 prepared in advance is read from the virtual space database 106, and the CG image 302 representing the measurement range of the sensor 301 is set as the viewpoint position / posture of the observer calculated by the viewpoint position / posture calculation unit 104. Generate based on. The CG image 302 is combined with a real image captured by the camera 101 and displayed on the HMD 108 to present the measurement range of the sensor 301 to the observer.

  Note that, as described above, for the three-dimensional model data representing the measurement range of the sensor, a measurement range based on the sensor specifications may be used, or a measurement range based on experience may be used. Further, any display method such as wire frame display, display as a solid with a surface, display as an opaque object, display as an object having transparency, and the like can be employed. When there are a plurality of sensors, visually distinguishable attributes such as colors can be displayed so that the measurement ranges of the individual sensors can be identified. In addition, it is possible to display so that it is easy to grasp the overlap of the measurement range, such as displaying the range that can be measured only by each of the two sensors in red and blue and the overlapping measurement range in purple. is there.

  In the present embodiment, the two-dimensional marker 205 is a black square written on a white card, and the pattern of the black square is made different for each marker to facilitate marker identification. Details of the marker will also be described later.

(How to create a 3D model of sensor measurement range)
Various methods can be considered as a method for creating a three-dimensional model of the sensor measurement range that can be used in the present embodiment.
First, a method for creating a three-dimensional model of a measurement range based on sensor specifications will be described.
When the position / orientation sensor is an optical sensor, based on the specifications of the target model as shown in FIG. 9, the dimension of the measurement range is input on the computer graphic modeling software, so that the three-dimensional model of the sensor measurement range is obtained. Can be created.
In the case of a magnetic sensor, create a three-dimensional model of the sensor's measurement range by creating a hemisphere with the radius of the measurement range described in the specifications of the target model using computer graphic modeling software. Can do.

Next, a method for creating a three-dimensional model of the measurement range of the position and orientation sensor based on experience will be described.
If data indicating “outside sensor measurement range” is sent from the sensor when it goes out of the measurement range of the sensor, acquire and store multiple pieces of the immediately preceding coordinate (X, Y, Z) data deep. Then, the measurement range can be modeled by generating a polygon having the stored coordinate point as a vertex. As a method of generating polygons from stored vertices, the Marching Cubes method `` Lorensen, WE and Cline HE, Marching Cubes: A High Resolution 3D Surface Construction Algorithm, ACM computer Graphics, 21 (3), 1987. '' ing.
When data indicating “outside sensor measurement range” is not sent, the operator experiences in advance and creates three-dimensional data indicating the measurement range from the experience result. For example, in the magnetic sensor, the radius of the hemisphere is set from the measurement result, and a three-dimensional model of the hemisphere is created.

(Viewpoint position and orientation calculation unit 104)
Here, the operation of the viewpoint position / orientation calculation unit 104 that performs an important function in the image composition apparatus according to the present embodiment will be described in detail. The basic operation of the viewpoint position and orientation calculation unit 104 is coordinate conversion.

FIG. 4 shows a coordinate system handled by the image composition apparatus of this embodiment.
In the present embodiment, the virtual object 206 is expressed on the marker coordinate system 401.
Based on the relationship between the marker position in the captured image and the marker position in the marker coordinate system, a transformation matrix T _cm from the marker coordinate system to the camera coordinate system is dynamically obtained. Then, based on the conversion matrix T _sc from the camera coordinate system to the HMD screen coordinate system and the conversion matrix T _cm which are obtained in advance, the three-dimensional model of the measurement range of the sensor defined in the marker coordinate system is converted. Then, the virtual object 206 is displayed on the HMD screen.

Below, the calculation method of a conversion matrix is demonstrated concretely.
In the embodiment of FIG. 3, a marker is attached to each of the two sensors to be arranged, and the three-dimensional model of the measurement range of each sensor is managed independently. Therefore, in this embodiment, the conversion matrix shown below is calculated for each marker.
In the camera coordinate system 402, the focal position is the origin, the direction perpendicular to the image plane is the Z axis, and the directions parallel to the x and y axes of the image are the X and Y axes.
An arbitrary point (X _m , Y _m , Z _m ) expressed in the marker coordinate system 401 can be converted into a point on the camera coordinate system 402 by a combination of rotational movement and parallel movement. _Xc , _Yc , _Zc ).

An image plane projected by the perspective transformation model is called an ideal screen coordinate system 403. It is assumed that a point (X _c , Y _c , Z _c ) in the camera coordinate system 402 appears in (x _c , y _c ) in the ideal screen coordinate system 403.
Further, in the mixed reality system, position measurement in a wide range is required, and therefore a wide-angle lens is used for the video camera 101. Therefore, barrel distortion occurs in the photographed image.

First, a coordinate system converted from the ideal screen coordinate system 403 by the image distortion conversion function 404 is set as an observation screen coordinate system 405. Further, the image data actually captured from the video camera 101 is expressed by the observation screen coordinate system 405, and the coordinate value is expressed by (x _d , y _d ).

The HMD 108 and the observer's eyes can be expressed by an ideal perspective transformation model. The image display surface of the HMD 108 is referred to as an HMD screen coordinate system 406, and the coordinate values above it are represented by (x _s , y _s ). A coordinate system having the X and Y axes in the same direction and the Z axis in the vertical direction with respect to the HMD screen coordinate system 406 and having the focal point of the eye as the origin is called a viewpoint coordinate system 407, and the coordinate value is ( _Xe , _Ye , _Ze ).

  This viewpoint coordinate system 407 is easier to set when the line-of-sight direction is the Z-axis, but in order to establish a perspective transformation model with the HMD screen coordinate system 406, depending on the relative relationship with the screen. The coordinate system setting as specified is important. The viewpoint coordinate system 407 and the HMD screen coordinate system 406 need to be set for each of the left and right eyes. Here, for simplicity of explanation, the viewpoint coordinate system 407 and the HMD screen coordinate system 406 will be described as a coordinate system common to both eyes.

Next, an outline of processing in the viewpoint position / orientation calculation unit 104 will be described with reference to a flowchart shown in FIG.
In the pre-processing of S501, binarization with a fixed threshold value and area / circumscribed rectangle calculation for each connected region are performed on the inputted real image. In this embodiment, the speed is increased by using an algorithm that performs these processes in one pass.

In the marker extraction in S502, the giant region and the minute region are excluded from the connected region detected in S501 by the area value, and the connected region that touches the image boundary is also excluded from the rectangle information that circumscribes the connected region.
Contour tracking is performed on the remaining connected regions, and all pixel positions on the contour are stored.

  A polygonal line approximation is performed on the contour line data, and a connected region that can be approximated with sufficient accuracy by four line segments is set as a marker candidate. Then, the coordinate values of the four break points (vertices) of the marker candidate area are stored.

FIG. 6 shows an example of a dimension marker applicable to the present embodiment.
The two-dimensional marker is a square marker in which an identification pattern 501 is provided in a black frame. In this embodiment, since marker identification is performed by template matching, an arbitrary symbol that can recognize the directionality (up / down / left / right) of the marker can be used as the identification pattern in the center.

In the marker identification in S503, the marker is detected and identified by template matching between the photographed image and a previously registered pattern.
In order to perform this template matching, it is necessary to normalize the image.

First, the marker is projected from the marker coordinate system 401 to the ideal screen coordinate system 403 by the perspective transformation model.
Equation 1 shows that a point (X _m , Y _m , 0) in the X _m -Y _m plane of the marker coordinate system 401 is converted into a point (x _c , y _c ) on the ideal screen coordinate system 403 by the perspective transformation model matrix C. Is a conversion formula to convert to.

  Since the physical size of the marker (the length L (mm) of one side in FIG. 6) is known, using the information and the coordinate values of the four vertices of the marker candidate area detected from the image, the simultaneous equations are solved. The value of the perspective transformation model matrix C can be obtained. Therefore, the pattern inside the marker can be normalized by this equation.

  Specifically, the pattern area inside the marker is divided into vertical and horizontal 64 × 64 areas, and pixel values corresponding to the respective areas are extracted from the input image to obtain a 64 × 64 pixel pattern image. This is reduced to an image of 16 × 16 pixels and used for template matching.

  In addition, in order to cope with the rotation of the marker, templates used for template matching are created in a total of four, one each rotated at 0 degrees, 90 degrees, 180 degrees, and 270 degrees.

  Expression 2 is an expression for calculating the similarity between the four template images and the input image.

In Expression 2, x _i represents the i-th element of a 16 × 16 reduced image (image vector). x is an average value of elements, and N is the number of dimensions (in this case, 255). x ^(l) means the l-th (l = 1, 2, 3, 4) template image (image vector), and x _i ^(l) means the i-th pixel of the l-th template image. Expression 2 corresponds to an expression for obtaining the cosine of two image vectors with normalized brightness.
Then, the template type and direction in which the similarity s with the reduced image has the maximum value are regarded as the marker type and direction.

  Subsequently, in the vertex position detection of S504, straight line fitting is performed on the contour line data corresponding to each side of the marker by the least square method, and the intersection of these straight lines is set as the vertex coordinate value.

At the time of this straight line fitting, conversion by the distortion function of Expression 3 is performed, and the vertex coordinate value in the ideal screen coordinate system 403 is obtained.
Here, (x _c , y _c ) is a coordinate value in the ideal screen coordinate system 403, and (x _d , y _d ) is a coordinate value in the observation screen coordinate system 405.

Further, p is a distortion rate, (x _c0 , y _c0 ) is a distortion center coordinate value, and these three parameters are calculated in advance by camera calibration.

  Since the observed square marker image has barrel distortion due to the influence of the wide-angle lens, the side of the marker is observed as a curve. However, the barrel distortion is corrected by the conversion formula 3 from the observation screen coordinate system 405 to the ideal screen coordinate system 403, and the side can be handled as a straight line.

  Finally, in the post-process of S505, suppression of marker misdetection is performed. Specifically, the past marker detection position and size are stored during processing, and if the detected marker position / size is similar to the stored value, the same marker is detected. Is considered. When the similarity is lower than a predetermined value, it is determined that the marker is different or erroneously detected. Thereby, it is possible to suppress marker pattern identification errors.

Next, the marker three-dimensional position estimation process will be described.
First, the conversion matrix T _cm from the marker coordinate system 401 to the camera coordinate system 402 is estimated.
This transformation matrix is composed of a rotational movement component R and a parallel movement component T. The ideal screen coordinate system 403 and the camera coordinate system 402 can be transformed by a perspective transformation model, and the specific transformation matrix P can be calculated in advance by calibration. Since the calculation of the perspective transformation model is a commonly used technique, the description of the specific calculation method is omitted here.

  The following formulas 4 and 5 show the relationship between these coordinate systems.

Expression 1 is a relational expression between the X _m -Y _{m in-} plane coordinate value (two-dimensional coordinate value) of the marker coordinate system 401 and the ideal screen coordinate system 403, but Expressions 4 and 5 are 3 of the marker coordinate system 401. This is a relational expression between the dimension coordinate value and the camera coordinate system 402.

  The coordinate values of the four vertices of the marker in the marker coordinate system 401 are set as shown in FIG. 6, and the coordinate values in the ideal screen coordinate system 403 corresponding to these are also obtained by the processing so far.

FIG. 7 is a flowchart showing a procedure for calculating the transformation matrix T _cm from Equation 4 and Equation 5.
In S701, the rotational movement component R is estimated.

An equation of two straight lines facing from the vertex position of the marker in the ideal screen coordinate system 403 is obtained (Formula 6).
Then, by substituting (x _c , y _c ) in Expression 5 into Expression 7, Expression 7 is obtained.

  Expression 7 is an equation of a plane in the three-dimensional space expressed by the camera coordinate system 402, and means that the side of the marker in the three-dimensional space exists in this plane. Since the two opposite sides of the marker are parallel, their direction vectors coincide, and the in-plane directions of the two planes of Equation 7 are obtained. That is, the vector calculated as the outer product of the normal vectors of the two planes of Expression 7 is the direction vector in the camera coordinate system 402 with two parallel sides.

By performing this calculation on two sets of parallel two sides, the direction vectors U ₁ and U ₂ of two adjacent sides of the marker can be obtained. Since the marker used in this embodiment is a square, ideally, these two vectors are orthogonal, but are not actually orthogonal due to measurement errors. Therefore, two unit vectors V ₁ and V ₂ which are orthogonal to each other in the plane including them are calculated for the two measured direction vectors, and these are used instead of U ₁ and U ₂ .

Further, by calculating the outer product of V ₁ and V ₂ , a unit vector V _{3 in the} direction perpendicular to the marker plane can be obtained. At this time, [V ₁ ^t , V ₃ ^t , V ₃ ^t ] is a rotation conversion component R from the marker coordinate system 401 to the camera coordinate system 402.

  However, it is necessary to determine the correspondence between the two direction vectors and the X and Y axes of the marker coordinate system 401 and the forward and reverse directions from the result of the template matching described above. Since the Z-axis of the marker coordinate system 401 is set downward in the marker plane, it is set in a direction in which the inner product of the direction vector pointing from the origin of the camera coordinate system 402 to the marker diagonal intersection in the ideal screen coordinate system 403 is positive. .

In S702, the translation component T is estimated.
Combining Equations 4 and 5, and substituting the coordinate values in the marker coordinate system 401 and the coordinate values in the ideal screen coordinate system 403 of the vertexes of the markers 4, eight linear equations relating to T ₁ , T ₂ , and T ₃ are obtained. can get. Since the matrices P and R are known, T ₁ , T ₂ , and T ₃ can be calculated from these equations. Finally, in S703, to correct the transformation matrix _{T cm.}

Although T _cm can be obtained in the calculation up to S702, the calculation of the rotation matrix R often involves a large error. Therefore, the rotation matrix R is corrected again using the image information. The rotation matrix is expressed by 9 parameters _R 11 _{to R 33} in Formula 4, three rotation angles which (a: inclination direction of _{Z m} axis, b: angle of inclination of the _{Z m} axis, c: X _m −Y The angle of rotation around the Z _m axis of the _m plane.

This is a modification of the ZYZ Euler angle representation. In the normal Euler angle expression, a slight variation of the vector may greatly change the rotation angle, but such an influence is small in this expression.
Expression 8 is an expression in which a rotation matrix is expressed by a rotation angle.

(Equation 9) can be derived from this equation, and the rotation angles a to c can be obtained from the rotation matrix R.

Thus, the coordinate value in the ideal screen coordinate system 403 can be calculated by substituting the coordinate value of the vertex of the marker 4 into Equations 4 and 5 using the T _cm obtained so far. The values of a, b, and c are corrected so that the square sum of the error between this calculated value and the value actually obtained by image processing is reduced. Specifically, a new rotation matrix R is obtained through 10 iterations using the hill-climbing method. Further, the process of S702 is applied again to update the translation component T.

A transformation matrix T _sc from the camera coordinate system 402 to the HMD screen coordinate system 406 is calculated in advance by a known calibration method or the like. The calculated transformation matrix is used as a projection matrix from the camera coordinate system 402 to the HMD screen coordinate system 406.

In this way, the viewpoint position / orientation calculation unit 104 is expressed in the marker coordinate system 401 by using the conversion matrix T _cm from the marker coordinate system 401 to the camera coordinate system 402 obtained by successive observations and the conversion matrix T _sc. A transformation matrix for projecting the three-dimensional CG model data onto the HMD screen coordinates 406 is obtained. Then, this transformation matrix is output to the image generation unit 105 as viewpoint position and orientation information.

  The image synthesizing unit 105 can generate a CG image using this transformation matrix to generate a CG image correctly projected on the HMD screen coordinate system 406, and a composition in which the real image and the CG image are correctly aligned. An image can be generated. As a result, the observer can view a virtual object expressed with reference to the marker coordinate system as if it were an object existing at the marker position in the real world.

  As the above-described three-dimensional CG model data of the virtual object, three-dimensional CG model data representing the measurement range of the sensor 301 is prepared in advance. The observer 301 can work while confirming the measurement range by sequentially observing the sensor 301 with the two-dimensional marker pasted with the video camera 101 and displaying the sensor 301 in alignment with the real space by the above-described processing. .

  Note that a CG image representing the measurement range of the sensor can be generated and displayed at all times, but it may be a hindrance to the experience of the original mixed reality application. Therefore, for example, it may be configured such that the display can be dynamically switched according to whether the switch operated by the observer is on or off. It is also possible to change the display transparency so that the transparency is lowered when necessary, and the transparency is raised when unnecessary.

In this embodiment, a two-dimensional marker is directly attached to a sensor whose measurement range is to be visualized, and the measurement range is visualized. However, for example, when examining the layout of the sensor, it is not necessary to use a real sensor. . Rather, if the sensor is large or heavy, place the sensor while visualizing the measurement range by attaching a two-dimensional marker to another place that is easy to move when considering the placement, for example, a racket or the tip of a cane. If the actual sensor is installed after the determination, the working efficiency is improved.
In this embodiment, one two-dimensional marker is used for each sensor. However, when the positional relationship between the markers is defined in advance, a plurality of point markers can be used for one sensor. . Similarly, a plurality of two-dimensional markers can be used.

As described above, according to the present embodiment, by using the mixed reality technology and visualizing the measurement range of the sensor as a virtual object, the installation position of the sensor can be promptly and appropriately without depending on intuition. Can be determined.
In addition, during execution of the mixed reality application, the observer can grasp the measurement range of the sensor and can move within the correct range.

<Second Embodiment>
The image composition device according to the second embodiment acquires position and orientation information of the observer's head in the world coordinate system from the measurement values of the actually installed position and orientation sensor after the position and orientation sensor is installed. Then, a position and orientation sensor is prepared by superimposing and displaying on a real image obtained from a camera arranged in the vicinity of the observer's line of sight, which is prepared in advance and defined in the world coordinate system and representing the measurement range of the sensor. It visualizes the measurement range.

  In the present embodiment, the position and orientation of the observer in the world coordinate system can be obtained from the output value of the sensor. Therefore, the three-dimensional CG model data representing the measurement range of the sensor is managed in the world coordinate system. Also in the present embodiment, as in the first embodiment, the 3D CG model data for visualizing the measurement range may be a measurement range based on specifications or a measurement range obtained through experience. . The model data display method is also arbitrary.

FIG. 10 shows a configuration example of an image composition device according to the second embodiment. In FIG. 10, the same components as those in FIG.
Reference numeral 1001 denotes a position and orientation sensor that detects the viewpoint position and orientation of the observer. A viewpoint position / orientation calculation unit 1002 calculates a conversion matrix for converting the world coordinate system to the HMD screen coordinate system 406 based on the output of the position / orientation sensor 1001. The image generation unit 105 generates a virtual object image based on a three-dimensional model defined in the world coordinate system stored in the virtual space database 1003 using the transformation matrix calculated by the viewpoint position / orientation calculation unit 1002. .
However, as is well known in the field of mixed reality technology, the output value of the position / orientation sensor can be corrected by information obtained from the marker.

FIG. 8 is a diagram schematically showing the display of the sensor measurement range realized by the image composition device of the present embodiment.
The position and orientation sensor 1001 detects the viewpoint position and orientation of the observer 201. Then, the three-dimensional CG model data (described in this embodiment in the world coordinate system) representing the measurement range of the sensor 1001 and the three-dimensional CG model data of the CG object 801 to be presented to the observer 201 are prepared. Read from the virtual space database 1003. Next, a three-dimensional CG image 302 projected onto the screen coordinate system 406 of the HMD 108 is generated from the viewpoint position and orientation of the observer measured by the position and orientation sensor 1001 and the transformation matrix from the viewpoint position and orientation calculation unit 1002. The image composition unit 107 synthesizes the CG image 302 from the image generation unit 105 with the actual photograph image from the image input unit 102 and displays it on the HMD 108. In this manner, the observer can visually recognize the measurement range of the sensor 1001.

  Here, when the measurement range of the sensor is simply presented to the observer 201 by the translucent three-dimensional CG image 302, the observer 201 feels like being confined in the CG image representing the measurement range within the measurement range. Will receive. Furthermore, in a mixed reality application, a CG image representing the measurement range of the sensor is overlaid on a target CG object that should be visually recognized, which may cause deterioration in drawing quality or obstructing the visual recognition of the CG object. is there.

  In the present embodiment, it is only necessary that the observer 201 can grasp how far the sensor can be measured in the real space. Therefore, it is not always necessary to draw a CG image representing the measurement range at all times. For example, when the observer 201 approaches the boundary of the measurement range, by adjusting and displaying the translucency of the three-dimensional object indicating the measurement range, the observer 201 is approaching the limit of the measurement range. You may make it inform.

  By applying such a display method, at a position where the position and orientation of the observer 201 can be normally acquired (that is, in a state where the observer exists sufficiently inside the measurement range of the sensor), a CG representing the measurement range. A method of explicitly showing the boundary of the measurement range to the observer 201 can be realized as the image approaches the boundary of the measurement range without displaying an image.

As described above, according to the present embodiment as well, by visualizing the measurement range of the sensor as a virtual object, the installation position of the sensor can be determined promptly and appropriately without depending on intuition.
In addition, during execution of the mixed reality application, the observer can grasp the measurement range of the sensor and can move within the correct range.

(Third embodiment)
In the above-described embodiments, the sensor measurement range is presented to the observer. However, it is possible to present the measurement range of the visualized sensor to the operator of the image composition device. By visualizing the measurement range of the sensor on the operation screen for the operator, the operator can grasp the positional relationship between the observer and the sensor measurement range. Therefore, when the operator monitors the work of the observer, the observer can be guided to an appropriate position (within the sensor measurement range).
In addition, when a CG object as shown in FIG. 8 is dynamically presented to the observer, it can be arranged at a position that is most easily seen by the observer.

  Thus, when presenting the sensor measurement range to the operator, it is preferable to use a visualized image viewed from the viewpoint position and orientation of a third person who can grasp the relationship between the observer and the sensor measurement range. . FIG. 8 shows an example of an image presented to the operator. In this way, the image generation unit 105 generates a measurement range of the observer (registered three-dimensional CG model) and the sensor as a CG image using the third person's viewpoint position / posture that can see the whole, and an operator (not shown) Can be presented on the display device. FIG. 8 shows an example in which the third person's viewpoint position / posture is used as if looking down from the upper side. However, as a viewpoint position / posture of a third person looking directly below, an image that schematically represents the mixed reality space from above may be generated and presented to the operator as shown in an overhead view.

  In addition, the measurement range of the position and orientation sensor placed in the real space is rendered by computer graphics as a virtual object, and the measurement range of the position and orientation sensor is visualized by aligning and synthesizing the real space with the captured real image. If possible, the present invention can be applied to an image composition apparatus having an arbitrary configuration other than the above-described embodiment.

(Fourth embodiment)
The image synthesizing apparatus according to the fourth embodiment further uses an objective camera (a camera that captures the viewpoint position and orientation of a third party) in addition to the position and orientation sensor in order to acquire the position and orientation information of the observer. The position and orientation information of the observer is acquired with high accuracy. Moreover, both the measurement range of the sensor and the measurement range of the objective camera are visualized and presented.

  When acquiring position / orientation information using an objective camera, it is possible to determine from the actual video of the objective camera how much the objective camera covers and can contribute to acquisition of position / orientation information. Have difficulty. In addition, in order to improve the accuracy of position and orientation information, measurement may be performed with a plurality of objective cameras arranged in a scene. In this case, information such as which objective camera overlaps with the measurement range of which objective camera is necessary, but it is difficult to make a judgment only from the actual captured video of the objective camera.

  That is, when the objective camera is arranged in the scene, the same problem as the above-described sensor arrangement problem occurs. Therefore, in order to solve this problem, in the present embodiment, the visual field range (measurement range) of the objective camera is visualized and presented to the operator and the observer in the same manner as the visualization of the measurement range of the sensor described above. .

FIG. 11 shows a configuration example of an image composition device according to the fourth embodiment. In FIG. 11, the same components as those in FIG. 1 or FIG.
Reference numeral 1101 denotes an objective camera arranged to detect the viewpoint position and orientation of the observer 201 with high accuracy. The position and orientation of the objective camera 1201 are fixed and known. As a guide for detecting the viewpoint position and orientation of the observer 201, at least one point, preferably a plurality of markers, is attached to the head of the observer using, for example, an HMD. When attaching a plurality of markers, the positional orientation relationship between the markers is obtained in advance. The video captured by the objective camera 1201 is input through the image input unit 1102 similar to the image input unit 102.

  The marker detection unit 1103 detects a marker, for example, a two-dimensional marker, mounted on the observer's head from the image supplied from the image input unit 1102. This two-dimensional marker is an index having features (color, shape, pattern, etc.) that can be extracted from an image. Therefore, the marker detection unit 1103 detects the two-dimensional marker by searching an area that matches the characteristics of the two-dimensional marker from the image.

  The viewpoint position / orientation calculation unit 1002 calculates the three-dimensional position and orientation of the marker mounted on the observer's head included in the image captured by the objective camera 1201 whose position and orientation are known. For this calculation, marker information (position in the image, direction of the marker, area, etc.) detected by the marker detection unit 1103, camera parameters of the objective camera 1201, and the like are used. Then, the viewpoint position and orientation of the observer is calculated from the calculated position and orientation of the marker.

  The viewpoint position / orientation calculation unit 1002 corrects the output of the position / orientation sensor 1001 using the viewpoint position / orientation calculated based on the video captured by the objective camera 1201. Based on the corrected viewpoint position / orientation value, a conversion matrix for converting the world coordinate system to the HMD screen coordinate system 406 is calculated. The image generation unit 105 generates a virtual object image based on the three-dimensional model defined in the world coordinate system stored in the virtual space database 1003 using the transformation matrix calculated by the viewpoint position / orientation calculation unit 1002. To do. A three-dimensional model representing the measurement range of the sensor 301 and the visual field range 1202 of the objective camera 1201 can also be stored in advance in the virtual space database 1003 and drawn and displayed as a virtual object image.

  FIG. 12 is a diagram schematically showing the display of the sensor measurement range and the visual field range of the objective camera, which is realized by the image composition device of the present embodiment. Here, FIG. 12 shows an example in which one objective camera is used for ease of explanation and understanding. However, a plurality of objective cameras may be used.

  The viewpoint position / orientation calculation unit 1002 detects the viewpoint position / orientation of the observer 201 from the measurement result of the position / orientation sensor 1001. Then, using the viewpoint position and orientation of the observer 201 detected from the image captured by the objective camera 1201, the measurement value of the position and orientation sensor 1001 is corrected. The viewpoint position / orientation calculation unit 1002 calculates a transformation matrix using the corrected viewpoint position / orientation.

  Then, the image generation unit 1005 reads, from the virtual space database 1003, three-dimensional CG model data (described in the world coordinate system in this embodiment) representing the measurement ranges 1201 and 302 of the objective camera 1201 and the sensor 1001 prepared in advance. . Also, the three-dimensional CG model data of the CG object 801 to be presented to the observer 201 is also read from the virtual space database 1003. The image composition unit 105 generates a three-dimensional CG image projected on the screen coordinate system 406 of the HMD 108 from the three-dimensional CG model data and the transformation matrix calculated by the viewpoint position / orientation calculation unit 1002. The image composition unit 107 synthesizes the CG image from the image generation unit 105 with the photographed image from the image input unit 102 and displays it on the HMD 108. In this way, the observer 201 can visually recognize the measurement ranges 1202 and 302 of the objective camera 1201 and the position and orientation sensor 1001.

Further, as described in the third embodiment, the measurement range 1202 of the objective camera 1201 and the measurement range 302 of the position and orientation sensor 1001 can of course be configured to be presented to the operator of the image composition apparatus. is there.
In this case, assuming that the viewpoint position and orientation of a third party is assumed as a virtual objective camera for generating an operator image (objective image) that is different from the objective camera 1201, an image to be presented to the operator is generated and displayed. Good.

  The depth of the CG object 1202 indicating the visual field range (measurement range) of the objective camera can be determined by using a marker from the image captured by the objective camera 1201 to the actual distance (detection of the viewpoint position and orientation). Decide based on what you can observe (by size). Since this depends on the performance of the objective camera 1201 actually used, the size of the marker, and the like, it can be determined in advance by measurement.

  In this embodiment, the viewpoint position and orientation of the observer is calculated by a combination of the position and orientation sensor and the objective camera, and the measurement range is visualized. However, it is not always necessary to combine the objective camera with the position and orientation sensor. For example, an image composition device that detects the viewpoint position and orientation of an observer by combining an orientation sensor such as a gyro and an objective camera and constructs a mixed reality space may be used.

(Other embodiments)
In the above-described embodiment, the following method can be exemplified as a method for more effectively presenting the measurement range of the sensor or objective camera to the observer and the operator. For example, the measurement range is color-coded according to the accuracy of position and orientation information that can be acquired, and a gradation display is performed. In addition, there is a method in which a visual field range (measurement range) is drawn as a semi-transparent CG object and a portion where the visual field ranges overlap is displayed so as to be grasped by mixing the colors of the CG object. Furthermore, there is a method of presenting the number of the objective camera or the like existing in the scene to the observer and the operator by superimposing and displaying it on the image of the objective camera as annotation information.

  In the fourth embodiment, the measurement ranges of both the position and orientation sensor and the objective camera are visualized. However, only the measurement range of the objective camera may be visualized. Of course, it is also possible to switch and display only the measurement range of the position and orientation sensor, only the measurement range of the objective camera, or both measurement ranges.

Further, a function equivalent to that of the above-described image composition device may be realized by a system constituted by a plurality of devices.
A software program that realizes the functions of the above-described embodiments is supplied directly from a recording medium or to a system or apparatus having a computer that can execute the program using wired / wireless communication. The present invention includes a case where an equivalent function is achieved by a computer executing the supplied program.

  Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.

  As a recording medium for supplying the program, for example, a magnetic recording medium such as a flexible disk, a hard disk, a magnetic tape, MO, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R, DVD- There are optical / magneto-optical storage media such as RW, and non-volatile semiconductor memory.

  As a program supply method using wired / wireless communication, a computer program forming the present invention on a server on a computer network, or a computer forming the present invention on a client computer such as a compressed file including an automatic installation function A method of storing a data file (program data file) that can be a program and downloading the program data file to a connected client computer can be used. In this case, the program data file can be divided into a plurality of segment files, and the segment files can be arranged on different servers.

  That is, the present invention includes a server device that allows a plurality of users to download a program data file for realizing the functional processing of the present invention on a computer.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to the user, and key information for decrypting the encryption for a user who satisfies a predetermined condition is provided via a homepage via the Internet, for example. It is also possible to realize the program by downloading it from the computer and executing the encrypted program using the key information and installing it on the computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.

  Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU of the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

It is a block diagram which shows the structural example of the image synthesizing | combining apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows typically the specific apparatus arrangement | positioning at the time of applying the image composition apparatus of FIG. 1 to a mixed reality application, and the mixed reality which an observer senses. It is a figure which shows typically the measurement range visualization of the position and orientation sensor in this embodiment. It is a figure which shows the coordinate system which the image composition apparatus of this embodiment handles. 6 is a flowchart for explaining an overview of viewpoint position / orientation calculation processing in the image composition apparatus according to the embodiment. It is a figure which shows the example of the two-dimensional marker which can be used by this embodiment. It is a flowchart illustrating a procedure for calculating the transformation matrix T _m. It is a figure which represents typically the display of the sensor measurement range implement | achieved by the image synthesizing | combining apparatus of the 2nd Embodiment of this invention. It is a figure which represents typically the example of a specification of a sensor measurement range. It is a block diagram which shows the structural example of the image composition apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structural example of the image synthesizing | combining apparatus which concerns on the 4th Embodiment of this invention. It is a figure which represents typically the display of the sensor measurement range implement | achieved by the image synthesizing | combining apparatus which concerns on the 4th Embodiment of this invention, and the visual field range of an objective camera.

Claims (10)

In an image composition method for aligning and synthesizing a computer graphics image with a real image,
A first setting step of setting a three-dimensional computer graphics model representing a measurement range of an objective camera whose position and orientation are known;
A viewpoint position and orientation information acquisition step for acquiring viewpoint position and orientation information;
A CG image generation step of generating a computer graphics image representing the measurement range of the objective camera using the viewpoint position and orientation information and the three-dimensional computer graphics model;
An image synthesizing method comprising an image synthesizing step of synthesizing the photographed image and the computer graphics image.   Viewpoint position / orientation information detected by the viewpoint position / orientation information acquisition step using the position / orientation sensor or detection information of the attitude sensor, the position / orientation of the objective camera, and the image of the marker included in the image captured by the objective camera. The image synthesizing method according to claim 1, wherein the viewpoint position / orientation is acquired. A second setting step for setting a position and orientation sensor or a three-dimensional computer graphics model representing a measurement range of the orientation sensor;
A sensor position information acquisition step of acquiring the position and orientation sensor or position information of the orientation sensor;
The CG image generation step uses the position information and the three-dimensional computer graphics model set in the first and second setting steps, and uses the measurement range of the objective camera and the position and orientation sensor or the orientation sensor. The image composition method according to claim 2, wherein a computer graphics image representing a measurement range is generated. In an image composition method that uses position and orientation information measured using a sensor and an objective camera, each having a predetermined measurement range, and aligns and synthesizes a computer graphics image with a real image,
A setting step of setting a three-dimensional computer graphics model representing the predetermined measurement range of the sensor and the objective camera;
A camera position / orientation information acquisition step of acquiring position / orientation information of a camera that captures the live-action image from detection information of the sensor and an image captured by the objective camera;
A CG image generation step of generating a computer graphics image representing the predetermined measurement range of the sensor and the objective camera using the camera position and orientation information and the three-dimensional computer graphics model;
An image synthesizing method comprising an image synthesizing step of synthesizing the photographed image and the computer graphics image. The virtual object generated by using a three-dimensional computer graphics model of the virtual object prepared in advance and the position and orientation information of the camera on a real image captured by a camera provided in a head-mounted display device worn by an observer In an image synthesis method for synthesizing images of the above and displaying the synthesized image on the head-mounted display device,
A setting step of setting a three-dimensional computer graphics model representing the measurement ranges of the sensor and the objective camera arranged in the space where the observer exists;
An image composition method comprising an image generation step of generating the measurement range as an image of the virtual object. In an image synthesizer that positions and synthesizes a computer graphics image with a live-action image,
First setting means for setting a three-dimensional computer graphics model representing a measurement range of an objective camera whose position and orientation are known;
Viewpoint position and orientation information acquisition means for acquiring viewpoint position and orientation information;
CG image generation means for generating a computer graphics image representing the measurement range of the objective camera using the viewpoint position and orientation information and the three-dimensional computer graphics model;
An image composition device comprising image composition means for compositing the photographed image and the computer graphics image. In an image composition device that uses position and orientation information measured using a sensor and an objective camera, each having a predetermined measurement range, and aligns and synthesizes a computer graphics image with a real image,
Setting means for setting a three-dimensional computer graphics model representing the predetermined measurement range of the sensor and the objective camera;
Camera position and orientation information acquisition means for acquiring position and orientation information of a camera that captures the live-action image from detection information of the sensor and an image captured by the objective camera;
CG image generation means for generating a computer graphics image representing the predetermined measurement range of the sensor and the objective camera using the camera position and orientation information and the three-dimensional computer graphics model;
An image composition device comprising image composition means for compositing the photographed image and the computer graphics image. The virtual object generated by using a three-dimensional computer graphics model of the virtual object prepared in advance and the position and orientation information of the camera on a real image captured by a camera provided in a head-mounted display device worn by an observer In the image composition device that synthesizes the images and displays the composite image on the head-mounted display device,
Setting means for setting a three-dimensional computer graphics model representing a measurement range of each of a sensor and an objective camera arranged in a space where the observer exists;
An image composition device comprising image generation means for generating the measurement range as an image of the virtual object.   A program for causing a computer to execute the image composition method according to any one of claims 1 to 5.   A computer-readable recording medium storing the program according to claim 9.

Images