JP2005251118A - Method and device for image processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique capable of intuitively operating a virtual object arranged in a compound real space to an observer who observes a compound real space that is compounded of a real space and a virtual space. <P>SOLUTION: A position attitude of a handy sensor mounted on a hand of the observer is set up as the position attitude of the target virtual object observed by the observer (S206), an image that observes the virtual object establishing its position attitude from a view point of the observer is produced (S208), and the image is provided to the observer by superposing it on the real space (S211). <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for providing an observer with a virtual object superimposed in a real space.

  The MR and AR techniques are techniques for synthesizing a live-action video and CG (Computer Graphics) and making it appear as if a virtual CG object exists in front of the user.

  In a typical application of MR technology, a user wears a video see-through type HMD. On the display in the HMD, the video from the camera in front of the HMD and the CG video created by the computer are combined. Since the CG image is drawn according to the position and orientation of the user's viewpoint measured by the position sensor, it appears to the user as if a virtual CG exists in the real space.

  As an actual application of MR technology, verification of industrial design as shown in FIG. 1 can be mentioned. FIG. 1 is a diagram showing a state in which a CAD model (CAD model of a printer toner cartridge) 100 is displayed as the CG object and used for previewing the CAD model. At this time, as described above, the user who previews the CAD model needs to wear the HMD (Head Mount Display) on his / her head.

  FIG. 2 is a diagram illustrating the configuration of the HMD.

  The HMD shown in the figure is a video see-through HMD. The HMD includes a right eye camera 1110, a left eye camera 1111, an HMD built-in position sensor 1120, a right eye LCD 1130, and a left eye LCD 1131.

  The user wears the HMD on his / her head and previews the CAD model. At this time, as shown in FIG. 3, on the LCDs 1130 and 1131 in the HMD, the live-action video photographed by the cameras 1110 and 1111 and the CAD model 300 drawn as a CG image are combined and displayed.

  FIG. 4 is a diagram showing a device configuration of a CAD viewer using MR technology that can be observed by a plurality of people. Here, the CAD model 4000 is previewed simultaneously by three people (4a to 4c), but the configuration of the equipment for each of the three people is almost the same.

  Taking the user 4b as an example, a live-action video shot by a camera provided in the HMD 1100 is captured by the computer 1300 and is synthesized with the CG of the CAD model 4000 inside. The synthesized image is sent to the HMD 1100 as a video signal.

  The computer 1300 can know the position of the HMD 1100 in three dimensions from the position sensor 1200. The computer 1300 determines how to draw the CG of the CAD model 4000 on the screen from this position information.

By synthesizing the CAD model 4000 on the live-action image, the user can observe the CAD model 4000 as if it were on the spot.
JP 2002-271893 A JP 2003-303356 A

  However, in the CAD model preview using the MR technique described above, it is difficult to intuitively move the displayed CAD model 4000.

  As described above, the computer 1300 determines the drawing position of the CAD model from the position information of the HMD. That is, in the example of FIG. 4, when the HMD 1100 is moved, the position of the CAD model 4000 on the screens of the LCDs 1130 and 1131 changes, but the CAD model 4000 is always drawn on a desk set in advance.

  However, in order to view the CAD model 4000 from various directions, it is inconvenient if the CAD model 4000 is always drawn on the desk. For example, when it is desired to view the bottom surface of the CAD model 4000, the drawing position of the CAD model 4000 must be changed by some method.

  Conventionally, to change the drawing position of the CAD model 4000, it is necessary to input a direction in which the CAD model 4000 is to be directed to the computer 1300. For that purpose, it is necessary to input the vector and angle of the direction you want to turn in numerical values, and it was hard to say that the model can be moved intuitively.

  This was particularly a problem when the CAD model 4000 was viewed by a large number of people. For example, when all users want to see a specific part as a CAD model, it is not easy to use the computer 1300 by consulting how to place the model.

  Another problem is that when a new part is added to or deleted from the CAD model 4000, it is impossible to know where the added or deleted part is. When the added and deleted parts are on the back side of the CAD model 4000 from the viewpoint of the user, it is difficult to grasp what changes have been made to the CAD model 4000.

  The present invention has been made in view of the above problems, and to a viewer observing a mixed reality space obtained by synthesizing a real space and a virtual space, a virtual object placed in the mixed reality space is more An object is to provide a technique for enabling intuitive operation.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, an image processing method for providing an observer with a virtual object superimposed in real space,
A first calculation step of calculating a position and orientation of the viewpoint based on a measurement result of a sensor for measuring the position and orientation of the observer's viewpoint;
A second calculation step for determining a position and orientation of the handheld sensor based on a result measured by the handheld sensor attached to the observer's hand;
A setting step for setting the position and orientation calculated in the second calculation step as a position and orientation of a virtual object to be observed by the observer;
Provided to the observer by generating an image when the virtual object having the position and orientation set in the setting step is viewed from the viewpoint of the position and orientation obtained in the first calculation step and superimposing it on a real space A process,
Furthermore, in the providing step,
When a part of the virtual object is instructed, the virtual object whose orientation has been changed so that the part can be seen from the position and orientation of the viewpoint is determined from the viewpoint of the position and orientation obtained in the first calculation step. An image when viewed is generated and superimposed on a real space and provided to the observer.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, an image processing apparatus that provides a viewer with a virtual object superimposed in real space,
First calculation means for calculating the position and orientation of the viewpoint based on a measurement result of a sensor for measuring the position and orientation of the observer's viewpoint;
Second calculation means for obtaining a position and orientation of the handheld sensor based on a result measured by the handheld sensor attached to the observer's hand;
Setting means for setting the position and orientation calculated by the second calculation means as the position and orientation of a virtual object to be observed by the observer;
Provided to the observer by generating an image when the virtual object having the position and orientation set by the setting means is viewed from the viewpoint of the position and orientation obtained by the first calculation means and superimposing it on a real space Means and
Further, the providing means includes
When a part of the virtual object is designated, the virtual object in which the direction of the virtual object is changed so that the part can be seen from the position and orientation of the viewpoint is determined from the viewpoint of the position and orientation obtained by the first calculation unit. An image when viewed is generated and superimposed on a real space and provided to the observer.

  According to the configuration of the present invention, an observer observing a mixed reality space obtained by synthesizing a real space and a virtual space can more intuitively operate a virtual object placed in the mixed reality space.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

  FIG. 8 is an external view of a system according to the present embodiment. In the figure, an observer who uses this system is shown together with the system. In the figure, the number of observers is three, but the following description is not limited to this.

  The system according to the present embodiment uses a well-known MR technology that provides a virtual object (CG model) to a viewer by superimposing a virtual object in a real space. However, this is for making various operations such as changing the position and posture easier. Hereinafter, the system according to the present embodiment will be described in detail.

  In the figure, reference numerals 1000, 2000, and 3000 denote observers, respectively, that use this system to observe a space (a mixed reality space) in which a virtual object is superimposed on a real space. The well-known HMDs 1100, 2100, and 3100 are attached to the heads of the respective observers, and each observer can see an image of the mixed reality space by the HMD attached to the own head.

  An image of the mixed reality space provided to each observer is generated by computers 1300, 2300, and 3300 connected to the respective HMDs.

  Here, since the configuration of the apparatus for generating and providing an image of the mixed reality space for each observer is practically the same, in the following description, the mixed reality space for the observer 1000 is described. The configuration of the apparatus that provides the image will be described, but the same description is similarly applicable to other observers 2000 and 3000 unless otherwise specified.

  The observer 1000 wears the HMD 1100 on his / her head, and as is well known in the MR technology, the observer 1000 can see an image of the mixed reality space according to the position / posture of his / her head using the HMD. .

  In addition, the observer 1000 has a hand-held sensor 5000 in his / her hand. In this embodiment, the hand-held sensor 5000 only needs to be held by any one of the observers 1000, 2000, and 3000, but in this embodiment, only the observer 1000 has it.

  Note that the hand-held sensor 5000 does not necessarily need to be “held” in the hand, but may be “attached” to the hand. The hand-held sensor 5000 measures the relative position and orientation with a three-dimensional sensor fixed station 1210 described later, and outputs a signal indicating the measurement result to the three-dimensional sensor main body 1200 described later.

  In the case of a magnetic sensor, the three-dimensional sensor fixed station 1210 generates a magnetic field that is detected by a later-described sensor and a hand-held sensor 5000 provided in the HMD 1100 to measure each position and orientation. In the following description, for the sake of convenience, all sensors are magnetic sensors, but the following description is substantially the same even when other sensors such as an ultrasonic sensor are used.

  The three-dimensional sensor main body 1200 controls a sensor (to be described later) provided in the HMD 1100, a three-dimensional sensor fixed station 1210, and a handheld sensor 5000, and calculates a result obtained by measuring the sensor and handheld sensor 5000 (to be described later) provided in the HMD1100 as data. 1300 is output.

  The computer 1300 includes, for example, a PC (personal computer), WS (workstation), and the like, and performs various processes for providing the observer 1000 with an image of the mixed reality space according to the position and orientation of his / her head. Do. Further, the computer 1300 is connected to the network hub 6000 and can perform data communication with other computers 2300 and 3300. For example, the result measured by the handheld sensor 5000 is transmitted from the computer 1300 to the computers 2300 and 3300. be able to.

  FIG. 9 is a diagram showing a basic configuration of an apparatus for providing an image of the mixed reality space to the observer 1000.

  The HMD 1100 includes an HMD built-in sensor 1120, a right eye camera 1110, a left eye camera 1111, a right eye LCD 1130, and a left eye LCD 1131. The HMD built-in sensor 1120 detects a magnetic field emitted from the three-dimensional sensor fixed station 1210, and outputs a signal indicating the intensity of the detected magnetic field to the three-dimensional sensor main body 1200.

  As described above, the hand-held sensor 5000 is attached to the hand of the observer 1000, and the position and orientation thereof changes as the observer's 1000 hand moves. The handheld sensor 5000 detects a magnetic field emitted from the three-dimensional sensor fixed station 1210 and outputs a signal indicating the intensity of the detected magnetic field to the three-dimensional sensor main body 1200.

  The three-dimensional sensor main body 1200 has a sensor coordinate system (world coordinate system (one point in the real space as an origin, and three orthogonal to each other at this origin based on the strength of signals received from the HMD built-in sensor 1120 and the handheld sensor 5000). Built-in HMD in a coordinate system in which the axes are x, y, and z axes, and the origin is a predetermined position where the position is known, and the three axes orthogonal to each other at the origin are x, y, and z axes, respectively. Data indicating the position and orientation of the sensor 1120 and the handheld sensor 5000 is obtained and output to the computer 1300.

  The right-eye camera 1110 and the left-eye camera 1111 capture moving images in real space that can be seen from the right and left eyes of an observer 1000 who wears the HMD 1100 on the head. Image data of each frame constituting a moving image captured by each of the right eye camera 1110 and the left eye camera 1111 is output to the computer 1300.

  The right-eye LCD 1130 and the left-eye LCD 1131 display images (mixed reality space images) obtained by superimposing virtual object images generated by the computer 1300 on the real spaces captured by the right-eye camera 1110 and the left-eye camera 1111, respectively. Thus, the observer 1000 wearing the HMD 1100 on the head can see an image of the mixed reality space corresponding to his / her right eye and left eye in front of his / her eyes.

  Next, the computer 1300 will be described.

  The serial I / O 1310 functions as an interface for receiving data output from the three-dimensional sensor main body 1200, and the received data is output to the memory 1302. This data is data indicating the position and orientation of the HMD built-in sensor 1120 and the handheld sensor 5000 in the sensor coordinate system. In the present embodiment, this position and orientation is composed of data indicating the position (x, y, z) and data indicating the orientation (roll, pitch, yaw). That is, the HMD built-in sensor 1120 and the handheld sensor 5000 are measurement sensors with six degrees of freedom.

  Note that there are other methods for expressing the position and orientation, and the present invention is not limited to this.

  Here, the memory 1302 stores data (offset data) indicating the position and orientation of the three-dimensional sensor fixed station 1210 measured in advance in the world coordinate system. Therefore, as is well known using this offset data, any position and orientation in the sensor coordinate system can be converted into a position and orientation in the world coordinate system, so the CPU 1301 can send the offset data and the serial I / O 1310 via the offset data. By using “data indicating the position and orientation of the HMD built-in sensor 1120 in the sensor coordinate system” input to the memory 1302, “position and orientation of the HMD built-in sensor 1120 in the world coordinate system” can be obtained, and offset data and Using the “data indicating the position and orientation of the handheld sensor 5000 in the sensor coordinate system” input to the memory 1302 via the serial I / O 1310, the “position and orientation of the handheld sensor 5000 in the world coordinate system” can be obtained. it can.

  Since the position / orientation calculation technique based on such coordinate transformation is a well-known technique, further explanation is omitted.

  Each of the video capture cards 1320 and 1321 functions as an interface for receiving an image of each frame input from the right eye camera 1110 and the left eye camera 1111, and the received images are sequentially output to the memory 1302. In the present embodiment, the image of each frame is assumed to conform to NTSC, but the present invention is not limited to this.

  Each of the video cards 1330 and 1331 functions as an interface for outputting the mixed reality space image corresponding to the right eye and left eye generated by the processing described later by the CPU 1301 to the right eye LCD 1130 and the left eye LCD 1131. The corresponding mixed reality space images are output to the right eye LCD 1130 and the left eye LCD 1131 via the video cards 1330 and 1331 as VGA-compliant image signals, respectively.

  The CPU 1301 controls the entire computer 1300, controls data communication with other computers 2300 and 3300, and executes processing described later that should be performed by the computer 1300. Execution of such processing is performed by executing various programs and data stored in the memory 1302.

  The memory 1302 is a work area used by the CPU 1301 to execute various processes, an area for storing programs and data for causing the CPU 1301 to execute various processes, a serial I / O 1310, and a video capture card. An area for temporarily storing various data input via 1320 and 1321 is provided.

  The network I / F (interface) 1390 functions as an interface for connecting the computer 1300 to the network hub 6000 via a communication path. The computer 1300 is connected to the other computer 2300 via the network I / F 1390. Data communication with 3300 can be performed.

  The PCI bridge 1303 is for electrically connecting a bus connecting the CPU 1301, the memory 1302, and the network I / F 1390 to a bus connecting the serial I / O 1310, the video capture cards 1320 and 1321, and the video cards 1330 and 1331. It is.

  Next, processing for presenting an image of the mixed reality space to the observer 1000 by each apparatus having the above configuration will be described. Further, “virtual object” in the following description is a virtual object to be observed by the observer 1000. That is, it is a virtual object for the observer 1000 to easily change the position and orientation in order to observe well.

  The three-dimensional sensor main body 1200 always receives signals output from the handheld sensor 5000 and the HMD built-in sensor 1120, converts the data into data corresponding to the received signals, that is, data indicating the position and orientation in the sensor coordinate system, and outputs the data to the computer 1300. To do. The CPU 1301 of the computer 1300 uses the offset data and the data input to the memory 1302 via the serial I / O 1310 to determine the position and orientation of the handheld sensor 5000 in the world coordinate system and the position of the HMD built-in sensor 1120 in the world coordinate system. Calculate posture. Data indicating the calculation result is temporarily stored in the memory 1302 for use in processing to be described later.

  FIG. 5 is a diagram illustrating a configuration example of the handheld sensor 5000. The hand-held sensor 5000 includes a sensor unit 5001 that detects a magnetic field from the three-dimensional sensor fixed station 1210, a sensor cover 5002 that functions as a cover for the sensor unit 5001, and a hand-held unit 5003 that the observer 1000 holds in the hand. Become.

  On the other hand, the right-eye camera 1110 and the left-eye camera 1111 always capture a moving image of the real space that is visible according to the position and orientation of the HMD 1100 (in other words, the position and orientation of the head of the observer 1000). The image of each frame is sequentially input to the memory 1302 through the video capture cards 1320 and 1321 as data.

  The CPU 1301 calculates the displacement of the position and orientation between the HMD built-in sensor 1120 and the left-eye camera 1111 in advance based on the previously calculated position and orientation in the world coordinate system of the HMD built-in sensor 1120 (data indicating the measurement result is stored in the memory 1302). The position and orientation of the left-eye camera 1111 in the world coordinate system are obtained, and an image of a virtual object that can be seen from the obtained position and orientation is generated.

  Here, the position and orientation of the hand-held sensor 5000 in the world coordinate system is set as the position and orientation of the virtual object. That is, the observer 1000 can arbitrarily set the position and orientation of the virtual object by moving the handheld sensor 5000 to an arbitrary position and orientation.

  Thereby, a virtual object can be arranged at the position and orientation of the hand-held sensor 5000 in the world coordinate system. The details of the process of determining the position and orientation of the virtual object will be described later.

  The CPU 1301 then generates an image of the virtual object that can be seen from the position and orientation of the left-eye camera 1111 in the world coordinate system after the placement of the virtual object. Further, when generating an image of a virtual object, data related to the virtual object (for example, data related to the shape of the virtual object, data related to color and texture, etc.) is used, and this data is stored in the memory 1302. .

  Then, the CPU 1301 superimposes the virtual object image generated by the above processing on the “real space image that can be seen from the left eye of the observer” input from the video capture card 1321 to the memory 1302 previously. As a result, the mixed reality space image in which the virtual object viewed from the left eye of the observer 1000 is superimposed on the real space viewed from the left eye of the observer 1000 can be generated. The image is output to the left-eye LCD 1131 of the HMD 1100 via the video card 1331. As a result, an image of the mixed reality space viewed from the left eye of the observer 1000 is displayed on the left eye LCD 1131 of the HMD 1100.

  Note that the process of displaying the mixed reality space image viewed from the right eye of the viewer 1000 on the right eye LCD 1130 is performed in the same manner as in the case of the left eye. That is, the CPU 1301 adds the position and orientation of the HMD built-in sensor 1120 and the right-eye camera 1110 to the position and orientation calculated in advance in the world coordinate system (measured in advance, and data indicating the measurement result is stored in the memory 1302. Is stored to obtain the position and orientation of the right-eye camera 1110 in the world coordinate system, and an image of a virtual object that can be seen from the obtained position and orientation is generated.

  Then, the generated image of the virtual object is superimposed on the “real space image that can be seen from the right eye of the observer” input from the video capture card 1320 to the memory 1302 first. As a result, the mixed reality space image in which the virtual object viewed from the right eye of the observer 1000 is superimposed on the real space viewed from the right eye of the observer 1000 can be generated. The image is output to the right eye LCD 1130 of the HMD 1100 via the video card 1330. Thereby, an image of the mixed reality space viewed from the right eye of the observer 1000 is displayed on the right eye LCD 1130 of the HMD 1100.

  Through the above processing, the right-eye LCD 1130 and the left-eye LCD 1131 display the mixed reality space images that can be seen from the right and left eyes of the observer 1000, respectively. Therefore, the observer 1000 uses the HMD 1100 according to the position and orientation of his / her head. You can see an image of the mixed reality space in front of you.

  FIG. 6 is a diagram illustrating a display example of a virtual object in which the position and orientation of the handheld sensor 5000 is set. The left side of the figure shows a state where no virtual object is displayed, and the right side of the figure shows a display example of the virtual object 4000 in which the position and orientation of the handheld sensor 5000 is set. In the figure, the position of the center of gravity of the virtual object 4000 (predetermined and included in the data related to the virtual object) is matched with the position of the hand-held sensor 5000.

  With the above processing, the observer 1000 can change the position and orientation of the virtual object to be observed simply by moving the hand holding the handheld sensor 5000, so the position and orientation of the virtual object can be changed more intuitively. The virtual object can be observed well.

  FIG. 10 is a flowchart of processing performed by the CPU 1301 when the computer 1300 is activated.

  First, a projection matrix is created from the camera parameters (view angle, focal length, etc.) of the right-eye camera 1110 and left-eye camera 1111 of the HMD 1100, and stored as data in the memory 1302 (step S100). Here, the projection matrix is a projection matrix for generating an image by projecting an object in the virtual space onto a display surface, as is well known. By generating this projection matrix, when a virtual object image generated by using this projection matrix and a real space image captured by the right-eye camera 1110 and the left-eye camera 1111 are combined, the image between the images is displayed. This ensures consistency of the angle of view and focal length.

  Next, the local coordinates of the virtual object (specified by setting the origin set for the virtual object when the virtual object is created by CG software and the axes orthogonal to each other at the origin as x, y, z axes, respectively. Centroid (gx, gy, gz) in the coordinate system). Then, a transformation matrix MG that translates by (−gx, −gy, −gz) is created (step S101). If this transformation matrix MG is applied to a virtual object, the center of gravity of the virtual object can be used as the origin of local coordinates.

  Next, a unit matrix is substituted into the direction correction matrix MD of the virtual object (step S102). The virtual object direction correction matrix MD indicates in what direction the virtual object is facing the handheld sensor 5000. Since MD has a unit matrix in the initial state, the hand-held sensor 5000 and the virtual object face the same direction.

  By setting a matrix representing rotation in the MD, the designated component can be directed toward the user as shown in FIG.

  This MD is a matrix held only by the computer 1300 (more specifically, only held by the memory 1302 of the computer 1300). This matrix is a handheld in the world coordinate system obtained by the CPU 1301 of the computer 1300. Along with the position and orientation of the sensor 5000, it is transmitted to the computers 2300 and 3300.

  Further, the moving flag is set to FALSE (step S102). The moving flag is a flag for indicating whether or not the virtual object is rotating as shown in FIG. Although the details will be described later, the usage is used to determine whether or not to execute an animation operation in which a designated part is directed toward the observer.

  The above processing is executed when the computer 1300 is activated.

  FIG. 11 is a flowchart of a process for the CPU 1301 and other units to generate an image of the mixed reality space and present it to the viewer 1000 after the above process. As described above, the processing for displaying the mixed reality space image on the right-eye LCD 1130 and the processing for displaying the mixed reality space image on the left-eye LCD 1131 are both substantially the same. The flowchart shown in FIG.

  Therefore, in the following description, a process for displaying an image of the mixed reality space on the right eye LCD 1130 will be described as an example. However, the same process is performed when an image of the mixed reality space is displayed on the left eye LCD 1131.

  Since each frame image (video image) constituting the moving image in the real space captured by the right-eye camera 1110 is input to the computer 1300 via the video capture card 1320, the CPU 1301 sequentially inputs this to the memory 1302. (Step S200), the data is written in the video buffer provided on the memory 1302 (Step S201). In this embodiment, the image of each frame is input from the right-eye camera 1110 with 640 × 480 pixels for one frame (two fields), but is not limited to this.

  On the other hand, since data indicating the position and orientation measured by the HMD built-in sensor 1120 is input from the three-dimensional sensor main body 1200 via the serial I / O 1310, the CPU 1301 uses this data and the offset data to perform a well-known calculation. Thus, the position and orientation of the HMD built-in sensor 1120 in the world coordinate system is obtained, and the position and orientation deviation data between the HMD built-in sensor 1120 and the right-eye camera 1110 is added to the obtained position and orientation, and the right-eye camera 1110 in the world coordinate system. Is obtained (step S202).

  Next, a known viewing matrix is generated according to the projection matrix, the position and orientation of the right-eye camera 1110, and the generated viewing matrix data is recorded in the memory 1302 (step S203). As is well known, this viewing matrix is for matching the position and orientation of the virtual camera viewing the virtual object with the position and orientation of the right-eye camera 1110.

  Further, since data indicating the position and orientation measured by the handheld sensor 5000 is input from the three-dimensional sensor main body 1200 via the serial I / O 1310, the CPU 1301 performs a known calculation using this data and the offset data. Then, the position and orientation of the hand-held sensor 5000 in the world coordinate system is obtained (step S204). More specifically, processing for obtaining a conversion matrix ST from the origin in the world coordinate system to the position and orientation of the hand-held sensor 5000 in the world coordinate system is performed. The obtained transformation matrix ST data is recorded in the memory 1302.

  The data of the transformation matrix ST is output to the other computers 2300 and 3300 via the network I / F 1390 and the network hub 6000.

  Next, the direction adjustment matrix MD of the virtual object is updated (step S205). This is a process for realizing an animation in which the designated part is directed toward the observer 1000 as shown in FIG.

  FIG. 15 is a flowchart showing details of the processing in step S205.

  First, the CPU 1301 refers to a flag indicating whether or not the virtual object is rotating, and determines whether or not this flag is TRUE (step S400). When this flag is FALSE, the virtual object is not rotated, that is, it is not necessary to rotate, so the process according to the flowchart of FIG. 11 is terminated, and the process returns to step S206 of FIG.

  On the other hand, if this flag is TRUE, the process proceeds to step S401, and the difference between the angle (direction correction target value) RA at which the virtual object is to be rotated and the current virtual object angle (direction correction current value) RC. Is less than the threshold θ (step S401). In the present embodiment, the threshold θ is “3 degrees”, but is not limited to this. The angle RA will be described later.

  If the difference is smaller than θ, the process proceeds to step S402, it is determined that the rotation is completed, the flag value is updated to FALSE (step S402), and the process according to the flowchart of FIG. Then, the process returns to step S206 in FIG.

  On the other hand, if the difference is equal to or larger than θ, the process proceeds to step S403, and the process of updating by adding θ to the current virtual object angle RC is performed (step S403). Then, a process for updating the matrix MD is performed by multiplying the direction adjustment matrix MD by the “transformation matrix that rotates by θ around the vector RV indicating the rotation axis of the virtual object” (step S404).

  Therefore, when the virtual object is rotated around the vector RV and the desired part of the virtual object (for example, a part selected by the observer 1000 in the virtual object) is seen by the observer 1000, this angle is This corresponds to the angle RA.

  By repeatedly performing the process in step S205 described above, an animation in which a desired portion of the virtual object is directed to the observer 1000 can be realized.

  Returning to FIG. 11, a world matrix which is a matrix indicating the position and orientation of the virtual object is obtained according to the following equation (step S206).

MD / MG / ST
Here, MD is a virtual object direction correction matrix, and MG is a transformation matrix that moves the center of gravity of the virtual object to the origin of the local coordinates. The matrix world matrix thus obtained is for making the position of the center of gravity of the virtual object coincide with the position and orientation of the handheld sensor 5000.

  Next, it is determined whether or not the flag is TRUE (step S207). If the flag is FALSE, the process proceeds to step S208, and the CPU 1301 generates a virtual object image using a known technique using a world matrix or the like (step S208). As described above, this image is an image of a virtual object that can be seen according to the position and orientation of the right-eye camera 1110 in the world coordinate system. The generated virtual object image is rendered on the memory 1302.

  On the other hand, if the flag is TRUE in step S207, the process proceeds to step S209. In steps S209 and S210, an image of a desired portion of the virtual object and an image of a portion that is not the desired portion are rendered in the same manner as in step S208. However, in rendering in step S210, the image to be rendered is half-finished. Render with transparency. Since the technology for rendering the rendering target image in a translucent format is well known, a description thereof is omitted here.

The virtual object image is rendered on the memory 1302 regardless of the above-described processing of step S208 or steps S209 and S210, but the real space image is stored in the memory 1302 first. In step S208, or in any of steps S209 and S210, a virtual object image is rendered on the real space image. As a result, an image in which the virtual object image is superimposed on the real space image, that is, an image of the mixed reality space is generated on the memory 1302, so that the CPU 1301 stores the data of the mixed reality space image. Is output to the frame buffer of the video card 1330 (step S211). As a result, the image in the frame buffer of the video card 1330 is output to the right-eye LCD 1130. By repeating the above processing, the observer 1000 determines the position and orientation of the virtual object to be observed by the hand holding the handheld sensor 5000. The position and orientation of the virtual object can be changed more intuitively and the virtual object can be observed well.

  Further, as described above, the computer 2300 and 3300 are the same as the computer 1300 by transmitting the position and orientation of the hand-held sensor 5000 in the world coordinate system, that is, the data of the matrix ST, to the other computers 2300 and 3300, as determined by the computer 1300. (The process of receiving the data of the matrix ST from the computer 1300 is performed instead of the process of obtaining the matrix ST in the above process), so that the observer 1000 can use the handheld sensor 5000 for the observers 2000 and 3000. Therefore, it is possible to show a virtual object that is manipulating the position and posture.

  Next, a process for rotating the virtual object so that the viewer 1000 can see the selected part when the observer instructs (selects) a part of the virtual object will be described.

  The part of the virtual object is, for example, a part constituting the virtual object. The instruction (selection) means may be selected in advance, for example, or an operator of the computer 1300 may instruct using a keyboard or mouse provided in the computer 1300, The instruction method is not particularly limited. Further, the part of the virtual object is not limited to that instructed by the observer 1000, and may be a part added to the virtual object or a deleted part. That is, the part of the virtual object is a part to be observed by the observer 1000, and what this is is not particularly limited.

  For example, as shown in FIG. 14, the virtual model is composed of a plurality of parts (corresponding to each cube in the figure), and the observer 1000 observes this virtual object from the front of the page. When the part to be shown is selected, the observer 1000 wants to observe the part 4100. In order to make the part 4100 easier to see, the virtual object is rotated so that the part 4100 comes to the front of the page. In the following description, the virtual object shown in the figure will be described as an example, but it is clear that the essence of the following description is not limited to this.

  FIG. 12 is a flowchart of processing for rotating a virtual object so that the virtual object is directed to the observer when a part of the virtual object is instructed. The process according to the flowchart of FIG. 15 is performed before step S400 in the flowchart of FIG. 15 when a part of the virtual object is selected.

  First, the CPU 1301 obtains the position of the center of gravity of the selected component 4100 (the position of the component 4100) (step S300). This centroid position is approximated by the average value of the vertex positions of each polygon, for example, when the component 4100 is formed of polygons.

  Next, the CPU 1301 obtains a vector PD that connects the centroid position obtained in step S300 and the centroid position of the entire virtual object (step S301). The vector PD indicates which direction the selected component 4100 is currently facing.

  Next, the CPU 1301 calculates the position of the HMD built-in sensor 1120 in the world coordinate system (for example, based on this position, the position of the right eye camera 1110 in the world coordinate system or the position of the left eye camera 1111 in the world coordinate system) May be used, but it is preferable that the position is the “viewpoint position” of the observer 1000) and the center of gravity position of the virtual object is obtained (step S302). In the case of the present embodiment, since there are three observers, the position in the world coordinate system of the HMD built-in sensor provided in the HMD attached to the head of each observer is acquired and acquired. A position obtained by averaging the three positions is obtained, and a vector connecting the obtained position and the position of the center of gravity of the virtual object is obtained as the vector UD. The vector UD indicates the direction in which the part 4100 should be directed.

  Next, the CPU 1301 obtains an angle RA formed by the vector PD and the vector UD by calculating the inner product of the vectors (step S303). RA is an angle (direction correction target value) for rotating the virtual object. Next, the CPU 1301 calculates the outer product of the vector PD and the vector UD, and obtains a vector RV perpendicular to each vector (step S304). The vector RV becomes a rotation axis when rotating the virtual object. FIG. 13 shows the vectors RV, UD, and PD.

  Next, when the angle RA is greater than 180 degrees (step S305), the CPU 1301 reverses the direction of the rotation axis RV and reverses the angle RA to “180 degrees−RA” in order to reverse the rotation direction. Update (step S306).

  Next, an angle RC (direction correction current value) indicating how many times the virtual object has been rotated is set to 0 degrees, and a flag indicating that the virtual object is being rotated is set to TRUE (step S307).

  Then, the process proceeds to step S400 in FIG.

  That is, in this process, the virtual object is rotated so that the vector connecting the center of gravity of the virtual object and the position of the part 4100 substantially matches the vector connecting the center of gravity of the virtual object and the viewpoint position of the observer. Yes.

  Through the above processing, the component 4100 can be rotated toward the observer 1000 by rotating the center of the center of gravity of the virtual object at the position of the handheld sensor 5000.

  As described above, according to the present embodiment, the virtual object can be viewed from a free angle by changing the position or direction of the handheld sensor.

  In addition, the position of the handheld sensor coincides with the center of gravity of the virtual object. Since the centers of gravity coincide with each other, the observer can operate the virtual object with a sense that “the handheld sensor has a virtual model”.

  When a part (part) constituting the virtual model is selected, the virtual object can be rotated around the center of gravity so that the part faces the viewer. Further, when there are a plurality of observers, the virtual object can be rotated in a direction that can be seen by the most observers. Thereby, a virtual object can be examined smoothly.

  Further, while the virtual object is rotated in an easy-to-view direction, the parts other than the selected part of the virtual object are displayed translucently. As a result, the selected part can be presented to the observer in an easy-to-understand manner.

  In the present embodiment, the position and orientation indicated by the world matrix is set as the position and orientation of the virtual object. However, the position and orientation of the handheld sensor 5000 may be set as it is. Further, in the present embodiment, images other than the selected component image are displayed in a semi-transparent state. However, it is possible to switch whether to display other than the selected component in a semi-transparent state. The display may be displayed in a predetermined color, and the display form of how to display the selected component and other than the selected component is not particularly limited.

  In addition, the method of obtaining the position and orientation of the HMD is not limited to the above method, and the HMD can be obtained by a known technique from a marker or the like whose position in the world coordinate system in the image captured by the right eye camera 1110 and the left eye camera 1111 is known. The posture may be estimated.

  In this embodiment, the virtual object is rotated around the center of gravity. However, the center position of the rotation is not limited to the center of gravity, and may be any one point in the virtual object. The setting is not particularly limited.

  Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or CPU) of the system or apparatus. Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

FIG. 3 is a diagram illustrating a state in which a CAD model (CAD model of a toner cartridge of a printer) 100 is displayed as a CG object and used for a CAD model preview. It is a figure which shows the structure of HMD. It is a figure which shows the image by which the CAD model 300 drawn as a CG image and the real image image | photographed with each camera 1110, 1111 displayed on each LCD1130,1311 in HMD was synthesize | combined. It is a figure which shows the apparatus structure of the CAD viewer using MR technique which can be observed by several persons. It is a figure which shows the structural example of the handheld sensor 5000. FIG. It is a figure which shows the example of a display of the virtual object to which the position and orientation of the handheld sensor 5000 was set. It is a figure which shows a mode that the designated components are orient | assigned to a user's direction. 1 is an external view of a system according to an embodiment of the present invention. It is a figure which shows the basic composition of the apparatus for providing the image of mixed reality space with respect to the observer 1000. FIG. 10 is a flowchart of processing performed by a CPU 1301 when a computer 1300 is activated. 6 is a flowchart of processing for generating an image of a mixed reality space and presenting it to an observer 1000 by the CPU 1301 and other units. When a part of a virtual object is instructed, it is a flowchart of a process of rotating the virtual object so that the part is directed toward the observer. It is a figure which shows each vector RV, UD, PD. It is a figure which shows the virtual object comprised by several components (equivalent to each cube in the same figure). It is a flowchart which shows the detail of the process in step S205.

Claims (7)

An image processing method for providing an observer with a virtual object superimposed in a real space,
A first calculation step of calculating a position and orientation of the viewpoint based on a measurement result of a sensor for measuring the position and orientation of the observer's viewpoint;
A second calculation step for determining a position and orientation of the handheld sensor based on a result measured by the handheld sensor attached to the observer's hand;
A setting step for setting the position and orientation calculated in the second calculation step as a position and orientation of a virtual object to be observed by the observer;
Provided to the observer by generating an image when the virtual object having the position and orientation set in the setting step is viewed from the viewpoint of the position and orientation obtained in the first calculation step and superimposing it on a real space A process,
Furthermore, in the providing step,
When a part of the virtual object is instructed, the virtual object whose orientation has been changed so that the part can be seen from the position and orientation of the viewpoint is determined from the viewpoint of the position and orientation obtained in the first calculation step. An image processing method characterized in that an image when viewed is generated and provided to the observer by superimposing it on a real space.   2. The image processing according to claim 1, wherein in the setting step, the position and orientation calculated in the second calculation step is set as a position and orientation of a center of gravity of a virtual object to be observed by the observer. Method.   In the providing step, the virtual object is set such that a vector connecting the center of gravity position of the virtual object and the position of the part substantially matches a vector connecting the center of gravity position of the virtual object and the viewpoint position of the observer. The image processing method according to claim 1, wherein the image processing method is rotated.   The image processing method according to claim 3, wherein, in the providing step, when generating the image of the virtual object, an image in which a part other than the part is made translucent is generated. An image processing apparatus that provides a viewer with a virtual object superimposed in a real space,
First calculation means for calculating the position and orientation of the viewpoint based on a measurement result of a sensor for measuring the position and orientation of the observer's viewpoint;
Second calculation means for obtaining a position and orientation of the handheld sensor based on a result measured by the handheld sensor attached to the observer's hand;
Setting means for setting the position and orientation calculated by the second calculation means as the position and orientation of a virtual object to be observed by the observer;
Provided to the observer by generating an image when the virtual object having the position and orientation set by the setting means is viewed from the viewpoint of the position and orientation obtained by the first calculation means and superimposing it on a real space Means and
Further, the providing means includes
When a part of the virtual object is designated, the virtual object in which the direction of the virtual object is changed so that the part can be seen from the position and orientation of the viewpoint is determined from the viewpoint of the position and orientation obtained by the first calculation unit. An image processing apparatus, characterized in that an image when viewed is generated and provided to the observer by being superimposed on a real space.   A program causing a computer to execute the image processing method according to any one of claims 1 to 4.   A computer-readable storage medium storing the program according to claim 6.

Images