JP2007004714A - Information processing method and information processing unit - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for synchronizing timing for acquiring a video image of a real space and timing for acquiring position/attitude information. <P>SOLUTION: Timing when the moving amount of a sensor 5000 on hand is minimum is detected on the basis of input position/attitude information. The timing when the moving amount of the sensor 5000 on hand is minimum is detected on the basis of the video image of the real space. Also, the difference between the timing for acquiring the video image of the real space and the timing for acquiring the position/attitude information is calculated on the basis of the respective timing. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for providing an image of a mixed reality space obtained by synthesizing a real space and a virtual space.

  The MR and AR techniques are techniques for synthesizing a live-action video and CG (Computer Graphics) and making it appear as if a virtual CG object exists in front of the user.

  In a typical application of MR technology, a user wears a video see-through type HMD. On the display in the HMD, the video from the camera in front of the HMD and the CG video created by the computer are combined. Since the CG image is drawn according to the position and orientation of the user's viewpoint measured by the position sensor, it appears to the user as if a virtual CG exists in the real space.

  As an actual application of MR technology, verification of industrial design as shown in FIG. 1 can be mentioned. FIG. 1 is a diagram showing a state in which a CAD model (CAD model of a printer toner cartridge) 100 is displayed as the CG object and used for previewing the CAD model. At this time, as described above, the user who previews the CAD model needs to wear the HMD (Head Mount Display) on his / her head.

  FIG. 2 is a diagram illustrating the configuration of the HMD.

  The HMD shown in the figure is a video see-through HMD. The HMD includes a right eye camera 1110, a left eye camera 1111, an HMD built-in position sensor 1120, a right eye LCD 1130, and a left eye LCD 1131.

  The user wears the HMD on his / her head and previews the CAD model. At this time, as shown in FIG. 3, on the LCDs 1130 and 1131 in the HMD, the live-action video photographed by the cameras 1110 and 1111 and the CAD model 300 drawn as a CG image are combined and displayed.

  FIG. 4 is a diagram showing a device configuration of a CAD viewer using MR technology. A live-action image taken by a camera provided in the HMD 1100 is captured by a computer 1300 and synthesized with a CAD model CG inside. The CAD model is arranged at the position of the handheld sensor 5000 with the posture of the handheld sensor 5000. The position and orientation of the handheld sensor 5000 is measured by the sensor devices 1200 and 1210. The synthesized image is sent to the HMD 1100 as a video signal.

By synthesizing the CAD model on the live-action video, the user can observe the CAD model as if it were on the spot.
JP 2002-271893 A JP 2003-303356 A

  However, the above-described system has a problem that the captured real image and the CAD model do not completely match.

  FIG. 5 is a diagram illustrating a position and orientation relationship between the hand-held sensor 5000 and the CAD model 4000. As shown in the figure, when the hand-held sensor 5000 is stationary, even if the CAD model 4000 is displayed on the hand-held sensor 5000, if the hand-held sensor 5000 is moved, the CAD model 4000 moves behind the movement of the hand-held sensor 5000. Such symptoms are observed.

  This is because the capture time of the actual image and the measurement time of the position information are different. In the case of FIG. 5, the capture time of the real image is earlier than the measurement time of the position information. In other words, it takes time to capture a real image, and the capture is delayed as compared to the position sensor.

  In the case of a live-action video, since the video from the HMD 1100 is buffered for one frame in the video capture card in the computer 1300, it takes 1/30 seconds or more. In addition, it takes time for processing by the OS or the like before the video is processed in the computer 1300.

  In the case of position information, it takes time for the controller in the position sensor 1200 to process the signal from the magnetic sensor 1210 and calculate the position of the sensor. Further, it takes time for the position sensor 1200 and the computer 1300 to communicate, and processing time by the OS of the computer 1300 and the like.

  As described above, there are delay factors in the acquisition of the actual image and position information, and the delay times of the two are different. Therefore, even if the computer 1300 simply acquires the actual image and position information and synthesizes the actual image and the CAD model 4000, the positions of the actual image and the CAD model 4000 are shifted as shown in FIG.

  In addition, there is also a demand for knowing the absolute amount of delays in live-action video and position information in order to analyze such delays and determine in which application fields the MR system can be used. For example, I would like to make a judgment that there is room for improvement because there is a delay of about 0.3 seconds, or that it is difficult to catch a ball with HMD because there is a delay of about 0.3 seconds.

  However, the absolute amount of such delays could not be known by conventional systems.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for synchronizing the timing of acquiring a real space image and the timing of acquiring position and orientation information.

  Moreover, it aims at applying to the system which needs to synchronize each acquisition timing.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, a first input step for continuously inputting the position and orientation information of the pointing tool;
A second input step of continuously inputting an image of the real space including the pointing device;
Based on the position and orientation information input in the first input step, a first detection step of detecting a timing at which the amount of movement of the pointing tool is minimized;
A second detection step of detecting a timing at which the amount of movement of the pointing tool is minimized based on the video input in the second input step;
A calculation step for obtaining a difference between the acquisition timing of the image in the real space and the acquisition timing of the position and orientation information based on the timing detected in the first detection step and the timing detected in the second detection step; ,
And a first output step for outputting the deviation obtained in the calculation step.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, a control step of controlling a device that can be controlled to take either the first state or the second state;
A first detection step of detecting the timing when the device is in the first state according to the control in the control step;
A second detection step of detecting a timing when the device is in the first state based on an image of the real space including the pointing tool;
An output step of outputting a difference between the timing detected in the first detection step and the timing detected in the second detection step is provided.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, a first input means for continuously inputting the position and orientation information of the pointing tool;
Second input means for continuously inputting an image of a real space including the pointing device;
First detection means for detecting a timing at which the amount of movement of the pointing tool is minimized based on the position and orientation information input by the first input means;
Second detection means for detecting a timing at which the amount of movement of the pointing tool is minimized based on the video input by the second input means;
Calculation means for obtaining a difference between the acquisition timing of the image of the real space and the acquisition timing of the position and orientation information based on the timing detected by the first detection means and the timing detected by the second detection means; ,
And a first output means for outputting the deviation obtained by the calculation means.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, control means for controlling a device that can be controlled to take either the first state or the second state;
First detection means for detecting timing when the device is in a first state in accordance with control by the control means;
Second detection means for detecting timing when the device is in the first state based on an image of the real space including the pointing tool;
An output means for outputting a difference between the timing detected by the first detection means and the timing detected by the second detection means.

  With the configuration of the present invention, it is possible to synchronize the timing for acquiring the video in the real space and the timing for acquiring the position and orientation information. Further, the present invention can be applied to a system that needs to synchronize each acquisition timing.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
Since the conventional appearance of the system according to this embodiment for providing a user with a space (hereinafter referred to as a mixed reality space) obtained by combining the real space and the virtual space is used, the appearance is shown in FIG. It will be shown.

  Again, each part shown in FIG. 4 is demonstrated. The system according to the present embodiment uses the well-known MR technology that provides a user with a virtual object (CG model) superimposed on the real space, and the user positions the virtual object that is the observation target of the user. This is to make various operations such as changing the position and posture easier. Hereinafter, the system according to the present embodiment will be described in detail.

  In the figure, reference numeral 1000 denotes a user, who observes a space (a mixed reality space) in which a virtual object is superimposed in a real space using the present system. A well-known HMD 1100 is attached to the head of the user 1000, and the user 1000 can view an image of the mixed reality space with the HMD 1100 attached to his / her head.

  The video of the mixed reality space provided to the user 1000 is generated by a computer 1300 connected to the HMD 1100.

  The user 1000 holds the handheld sensor 5000 in his / her hand. Note that the hand-held sensor 5000 does not necessarily need to be “held” in the hand, but may be “attached” to the hand. The hand-held sensor 5000 measures the relative position and orientation with a three-dimensional sensor fixed station 1210 described later, and outputs a signal indicating the measurement result to the three-dimensional sensor main body 1200 described later.

  In the case of a magnetic sensor, the three-dimensional sensor fixed station 1210 generates a magnetic field that is detected by a later-described sensor and a hand-held sensor 5000 provided in the HMD 1100 to measure each position and orientation. In the following description, for the sake of convenience, all sensors are magnetic sensors, but the following description is substantially the same even when other sensors such as an ultrasonic sensor are used.

  The three-dimensional sensor main body 1200 controls a later-described sensor provided in the HMD 1100, a three-dimensional sensor fixed station 1210, and a hand-held sensor 5000, and uses a result measured by the later-described sensor provided in the HMD 1100 and the hand-held sensor 5000 as data. 1300 is output.

  The computer 1300 is configured by, for example, a PC (personal computer), WS (workstation), or the like, and performs various processes for providing the user 1000 with an image of the mixed reality space according to the position and orientation of his / her head. .

  FIG. 7 is a block diagram showing the basic configuration of the system including the computer 1300.

  The HMD 1100 includes an HMD built-in sensor 1120, a right eye camera 1110, a left eye camera 1111, a right eye LCD 1130, and a left eye LCD 1131. The HMD built-in sensor 1120 detects a magnetic field emitted from the three-dimensional sensor fixed station 1210, and outputs a signal indicating the intensity of the detected magnetic field to the three-dimensional sensor main body 1200.

  As described above, the hand-held sensor 5000 is attached to the hand of the observer 1000, and the position and orientation thereof changes as the observer's 1000 hand moves. The handheld sensor 5000 detects a magnetic field emitted from the three-dimensional sensor fixed station 1210 and outputs a signal indicating the intensity of the detected magnetic field to the three-dimensional sensor main body 1200.

  The three-dimensional sensor main body 1200 is based on the intensity of signals received from the HMD built-in sensor 1120 and the hand-held sensor 5000, with the origin of the position of the three-dimensional sensor fixed station 1210 and three axes orthogonal to each other at this origin. Data indicating the positions and orientations of the HMD built-in sensor 1120 and the handheld sensor 5000 in the coordinate system (x, y, z axes) are obtained and output to the computer 1300.

  The right-eye camera 1110 and the left-eye camera 1111 capture moving images in real space that can be seen from the right and left eyes of an observer 1000 who wears the HMD 1100 on the head. Image data of each frame constituting a moving image captured by each of the right eye camera 1110 and the left eye camera 1111 is output to the computer 1300.

  The right-eye LCD 1130 and the left-eye LCD 1131 display images (mixed reality space images) obtained by superimposing virtual object images generated by the computer 1300 on the real spaces captured by the right-eye camera 1110 and the left-eye camera 1111, respectively. Thus, the observer 1000 wearing the HMD 1100 on the head can see an image of the mixed reality space corresponding to his / her right eye and left eye in front of his / her eyes.

  FIG. 6 is a diagram illustrating an external appearance and an internal configuration of the handheld sensor 5000. FIG. 6A is a diagram showing the appearance of the handheld sensor 5000, 5004 is a marker having a predetermined color, and 5005 is an LED that is turned on / off under the control of the computer 1300. FIG. 6B is a diagram showing the internal configuration of the handheld sensor 5000, 5001 is a sensor unit for detecting a magnetic field from the three-dimensional sensor fixed station 1210, 5002 is a sensor cover that functions as a cover for the sensor unit 5001, and 5003. Is a hand-held portion that is held when the user 1000 holds the hand-held sensor 5000 in the hand. Note that the shape and configuration of the handheld sensor 5000 are not limited to this, but may be provided with a sensor unit, a marker, and an LED.

  Next, the computer 1300 will be described.

  The serial I / O 1310 functions as an interface for receiving data output from the three-dimensional sensor main body 1200, and the received data is output to the memory 1302. This data is data indicating the position and orientation of the HMD built-in sensor 1120 and the handheld sensor 5000 in the sensor coordinate system. In the present embodiment, this position and orientation is composed of data indicating the position (x, y, z) and data indicating the orientation (roll, pitch, yaw). That is, the HMD built-in sensor 1120 and the handheld sensor 5000 are measurement sensors with six degrees of freedom.

  Note that there are other methods for expressing the position and orientation, and the present invention is not limited to this.

  Here, the memory 1302 stores data (offset data) indicating the position and orientation of the three-dimensional sensor fixed station 1210 measured in advance in the world coordinate system. Therefore, as is well known using this offset data, any position and orientation in the sensor coordinate system can be converted into a position and orientation in the world coordinate system, so the CPU 1301 can send the offset data and the serial I / O 1310 via the offset data. By using “data indicating the position and orientation of the HMD built-in sensor 1120 in the sensor coordinate system” input to the memory 1302, “position and orientation of the HMD built-in sensor 1120 in the world coordinate system” can be obtained, and offset data and Using the “data indicating the position and orientation of the handheld sensor 5000 in the sensor coordinate system” input to the memory 1302 via the serial I / O 1310, the “position and orientation of the handheld sensor 5000 in the world coordinate system” can be obtained. it can.

  Since the position / orientation calculation technique based on such coordinate transformation is a well-known technique, further explanation is omitted.

  The parallel I / O 1311 functions as an interface for sending a control signal for controlling ON / OFF of the LED 5005 to the LED 5005. Such a control signal is issued under the control of the CPU 1301.

  Each of the video capture cards 1320 and 1321 functions as an interface for receiving an image of each frame input from the right eye camera 1110 and the left eye camera 1111, and the received images are sequentially output to the memory 1302. In the present embodiment, the image of each frame is assumed to conform to NTSC, but the present invention is not limited to this.

  Each of the video cards 1330 and 1331 functions as an interface for outputting the mixed reality space image corresponding to the right eye and left eye generated by the processing described later by the CPU 1301 to the right eye LCD 1130 and the left eye LCD 1131. The corresponding mixed reality space images are output to the right eye LCD 1130 and the left eye LCD 1131 via the video cards 1330 and 1331 as VGA-compliant image signals, respectively. Note that the image signal to be output is not limited to that based on VGA.

  The CPU 1301 controls the entire computer 1300 and executes each process described later that should be performed by the computer 1300. Execution of such processing is performed by executing various programs and data stored in the memory 1302.

  The memory 1302 is a work area used by the CPU 1301 to execute various processes, an area for storing programs and data for causing the CPU 1301 to execute various processes, a serial I / O 1310, and a video capture card. An area for temporarily storing various data input via 1320 and 1321 is provided.

  The PCI bridge 1303 is for electrically connecting a bus connecting the CPU 1301 and the memory 1302 and a bus connecting the serial I / O 1310, the parallel I / O 1311, the video capture cards 1320 and 1321, and the video cards 1330 and 1331. It is.

  Next, processing for providing an image of the mixed reality space to the user 1000 by the system having the above configuration will be described.

  FIG. 8 is a flowchart of processing performed by the computer 1300 to provide the user 1000 with the mixed reality space image. Note that, as described above, the position and orientation measured by the handheld sensor 5000 and the HMD built-in sensor 1120 and the camera (hereinafter, “camera” in the description common to both the right eye camera 1110 and the left eye 1111 are simply “camera”). It is not possible to acquire the image of the real space from (referred to) at the same time, and the respective acquisition timings are “displaced” due to various factors.

  Therefore, in the present embodiment, by reducing this “shift”, the shift between the real space and the virtual space is reduced when providing the user 1000 with an image of the mixed reality space.

  In the following description, this deviation (delay time) is assumed to be n / 60 seconds. That is, it is assumed that the acquisition timing of the real space image from the camera is delayed by n / 60 seconds from the acquisition timing of the position and orientation measured by the handheld sensor 5000 and the HMD built-in sensor 1120. Therefore, when n> 0, the acquisition timing of the real space image from the camera is delayed from the acquisition timing of the position and orientation measured by the handheld sensor 5000 and the HMD built-in sensor 1120, and n <0. In this case, the acquisition timing of the real space image from the camera is ahead of the acquisition timing of the position and orientation measured by the handheld sensor 5000 and the HMD built-in sensor 1120, and when n = 0, the respective timings are acquired. Will match. This n is determined in advance, and the determination of n will be described later.

  Further, a program and data for causing the CPU 1301 to execute processing according to the flowchart shown in FIG. 8 are stored in the memory 1302, and the CPU 1301 performs processing using this, so that the computer 1300 performs each of the following descriptions. Processing will be performed.

  First, since a real space image (video image) captured by the camera is input to the computer 1300 via the video capture cards 1320 and 1321, the CPU 1301 detects this and sequentially stores it in the video buffer of the memory 1302 ( (Step S100).

  Next, the position and orientation information in the sensor coordinate system of the HMD built-in sensor 1120 and the position and orientation information in the sensor coordinate system of the handheld sensor 5000 (the sensor unit 5001) are transferred from the three-dimensional sensor main body 1200 to the computer 1300 via the serial I / O 1310. Since it is input, the CPU 1301 detects this and stores it in the memory 1302 (step S101). When storing, it is converted into a position and orientation in the world coordinate system and stored. The technique for converting to a position and orientation in the world coordinate system is as described above.

  Next, it is checked whether n = 0 (step S102). If n = 0, the process proceeds to step S111 to perform the subsequent processes. On the other hand, if n ≠ 0, the process proceeds to step S103 to check whether n> 0 is satisfied (step S103). If n> 0, the process proceeds to step S108. First, the first FIFO in which the position / orientation information (in the world coordinate system) of the handheld sensor 5000 stored in the memory 1302 in step S101 is provided in the memory 1302. Is stored (PUSH), and the position and orientation information (in the world coordinate system) of the HMD built-in sensor 1120 is stored (PUSH) in the second FIFO provided in the memory 1302 (step S108).

  Each time an image of the real space is stored in the memory 1302, the position / orientation information of the handheld sensor 5000 and the position / orientation information of the HMD built-in sensor 1120 are taken out (POP) from the FIFO and used in the subsequent processing. Since n data can be stored in the first and second FIFOs, and an image in the real space is stored in the memory 1302 every 1/60 seconds, 1 from the first and second FIFOs. / Position and orientation information is read every 60 seconds.

  Therefore, in the case of n> 0, the position / orientation information of the handheld sensor 5000 and the position / orientation information of the HMD built-in sensor 1120 used in the subsequent processes are stored in the FIFO (after being acquired) after n / 60. I will use something.

  In any case, since the CPU 1301 obtains the position / orientation information in the memory 1302 and then obtains the real space image in the memory 1302, an n / 60 second “displacement” has occurred. When a spatial image is handled in the subsequent processing, the position / orientation information that should be acquired at the same time as this is originally n / 60 seconds ago, so that it can be extracted from the FIFO (that is, n / 60 seconds ago). Used in the memory 1302).

  Accordingly, returning to FIG. 8, the position and orientation information of the handheld sensor 5000 is extracted (POP) from the first FIFO, and the position and orientation information of the HMD built-in sensor 1120 is extracted (POP) from the second FIFO (step S109). ).

  On the other hand, if n <0, the process proceeds to step S106. First, the image stored in the video buffer of the memory 1302 in step S100 is stored (PUSH) in a third FIFO provided in the memory 1302 (PUSH). Step S106).

  Each time the position / orientation information of the handheld sensor 5000 and the position / orientation information of the HMD built-in sensor 1120 are stored in the memory 1302, an image of the real space is taken out (POP) from the FIFO and used in the subsequent processing. Since n data can be stored in the third FIFO, and the position / orientation information is stored in the memory 1302 every 1/60 seconds, the third FIFO receives every 1/60 seconds. The real space image is read out.

  Therefore, in the case of n <0, the real space image used in the subsequent processing is the one after n / 60 after being stored (acquired) in the FIFO.

  Then, an image is taken out (POP) from the third FIFO (step S107). The extracted image is stored in the video buffer.

  Next, using the position and orientation information of the HMD built-in sensor 1120 extracted from the second FIFO or the position and orientation information of the HMD built-in sensor 1120 acquired in step S101, a ViewMatrix indicating the position and orientation of the camera is generated (step S111). ). As for the position and orientation of the camera, for example, the position and orientation relationship between the HMD built-in sensor 1120 and the camera is measured in advance as an offset and stored in the memory 1302 as data. In step S111, the position and orientation of the camera in the world coordinate system can be obtained by adding this offset to the position and orientation information of the HMD built-in sensor 1120. If it is expressed in the form of a matrix, it becomes ViewMatrix.

  Next, using the position and orientation information of the handheld sensor 5000 taken out from the first FIFO or the position and orientation information of the handheld sensor 5000 acquired in step S101, a Modeling Matrix indicating the position and orientation of the handheld sensor 5000 is generated (step S112). ). This indicates the position and orientation of the hand-held sensor 5000 in the world coordinate system. Therefore, next, a virtual object (here, a CAD model) is placed at the position and orientation of the hand-held sensor 5000, and an image of the virtual space including the CAD model that can be seen from the camera is generated using the ViewMatrix, and the video in the memory 1302 is generated. Drawing in the buffer (step S113). Since the real space image is previously drawn in step S100 or step S107 in this video buffer, as a result, the video buffer is an image in which the virtual space image is superimposed on the real space image, that is, Since the mixed reality space image has been generated, the CPU 1301 transfers it to the video cards 1330 and 1331 (step S114).

  Since the mixed reality space image transferred to the video cards 1330 and 1331 is transferred to the right eye LCD 1130 and the left eye LCD 1131, a composite image corresponding to the position and orientation of each eye is placed in front of the right eye and left eye of the user 1000. A real space image is displayed.

  Through the above processing, the mixed reality space image can be generated using the real space image and the position / orientation information that should be acquired at almost the same time. For example, the user 1000 moves and moves the handheld sensor 5000. However, the CAD model can be moved following the sensor position without shifting from the position of the sensor, and the position shift between the real space and the virtual space can be reduced.

  Next, a process for determining the number of data n that can be held by the FIFO (in other words, a parameter for determining the shift time) will be described with reference to FIG. 9 showing a flowchart of the process. This process should be performed before performing the process according to the flowchart shown in FIG. Further, a program and data for causing the CPU 1301 to execute the processing according to the flowchart shown in FIG. 9 are stored in the memory 1302, and the computer 1300 performs processing using the CPU 1301. Processing will be performed.

  Here, the user 1000 swings the hand-held sensor 5000 held by the hand to the left and right. And that is seen through the HMD 1100. That is, such a thing is imaged with a camera.

  In that case, the computer 1300 performs the following processing.

  First, the above image captured by the camera is sequentially stored in the memory 1302 via the video capture cards 1320 and 1321 as an image of the real space (step S200). Next, since the position and orientation information of the handheld sensor 5000 measured by the sensor unit 5001 of the handheld sensor 5000 is input to the computer 1300 via the three-dimensional sensor main body 1200 and the serial I / O 1310, the CPU 1301 detects this. After being converted into position and orientation information in the world coordinate system, it is stored in the memory 1302 (step S201). Note that the memory 1302 also holds the position and orientation information stored before that. That is, it is assumed that a predetermined amount of past position and orientation information is stored from what is currently stored.

  The time when the position / orientation information is stored in step S201 is stored in the memory 1302 as the “capture time” (step S202). Since the CPU 1301 always keeps track of the time, the CPU 1301 performs the processing in step S201 and stores the time at this time in the memory 1302.

  Next, the CPU 1301 detects the area of the marker 5004 from the real space image stored in the memory 1302 in step S200, and obtains the detected position (step S203). As the processing in step S203, for example, by referring to the pixel value of each pixel constituting the image, one or more regions by the pixel group having the color of the marker 5004 are detected, and the region having the largest area (configuration) is detected. The area having the largest number of pixels to be identified), and the position of the center of gravity of the identified area is set as the position in the image of the marker 5004.

  In this manner, the position of the marker 5004 in the image is obtained and stored in the memory 1302, and it is assumed that the memory 1302 also holds data on the center of gravity obtained in the past.

  Then, the difference between the position of the handheld sensor 5000 obtained in the previous step S201 and the position of the handheld sensor 5000 obtained in the current step S201 is calculated, and whether or not the difference value is equal to or less than a predetermined amount. (Step S204), if it is equal to or smaller than the predetermined amount, that is, if the handheld sensor 5000 is almost stationary, the process proceeds to step S205, and the time (timing) at that time is stored in the memory 1302 (step S205). ).

  Note that the processing in steps S204 and S205 is generally processing for obtaining a time at which the amount of movement is minimized using the position information of the handheld sensor 5000 and storing the obtained time in the memory 1302, which has been described here. The processing in steps S204 and S205 is an example.

  Also, the difference between the position in the image of the marker 5004 obtained in the previous step S203 and the position in the image of the marker 5004 obtained in the current step S203 is calculated, and whether or not the difference value is equal to or less than a predetermined amount. (Step S206), if it is equal to or less than the predetermined amount, that is, if the handheld sensor 5000 is almost stationary, the process proceeds to step S207, and the time (timing) at that time is stored in the memory 1302 (step S207). ).

  Note that the processing in steps S206 and S207 is generally processing for obtaining a time at which the amount of movement of the handheld sensor 5000 is minimized using the position of the marker 5004 on the image, and storing the obtained time in the memory 1302. The processing in steps S206 and S207 described here is an example.

  For example, when the hand-held sensor 5000 is moved so as to trace a triangle, the timing at which the acceleration takes the maximum value may be stored in the memory 1302 in step S205. Further, since it is possible to attach the hand-held sensor 5000 to various moving bodies without attaching it to the hand, various timings for acquisition can be considered.

  Then, the difference between the time stored in the memory 1302 in step S205 and the time stored in the memory 1302 in step S207 (that is, the delay time = n / 60 seconds) is obtained, and data indicating the obtained difference is stored in the memory 1302. (Step S208).

  Details of the processing in step S208 will be described with reference to the flowchart of FIG. FIG. 10 is a flowchart showing details of the processing in step S208. In the flowchart of FIG. 5, it is assumed that the delay time n / 60 seconds is smaller than ½ of the period of shaking the hand-held sensor 5000. FIG. 11 shows an outline of a method for calculating the delay time in the flowchart shown in FIG.

  It is assumed that the times acquired in step S205 are arranged in chronological order as ts0, ts1, ..., ts (n-1), ts (n). ts (n) is the latest acquisition time. Similarly, when the times acquired in step S207 are arranged in chronological order, tv0, tv1,..., Tv (n-1), tv (n) are assumed.

  At this time, ts (k) satisfying tv (n-1) <ts (k) <tv (n) is searched from ts0 to ts (n), and | tv (n-1) -ts (k) | , | Tv (n) −ts (k) | are compared (step S210).

  As a result, if | tv (n-1) -ts (k) | is larger, the process proceeds to step S212 via step S211 and tv (n) -ts (k) is set to the delay time = It outputs as n / 60 seconds (step S212).

  On the other hand, if | tv (n) -ts (k) | is greater or equal, the process proceeds to step S213 via step S211 and tv (n-1) -ts (k) is set to the delay time. = N / 60 seconds are output (step S213). At this time, the delay time is a negative number, and the acquisition timing of the image in the real space is advanced.

  Note that, as described above, in the flowchart of FIG. 10, it is assumed that the delay time is smaller than half of the period of shaking the handheld sensor 5000, but x is obtained as shown in FIG. + x)-If ts (k) is the delay time, there is no need to make such an assumption.

  Then, the processing is returned to the flowchart of FIG. 9, and the subsequent processing is repeated.

  Here, if there is a difference between the time when the sensor 5000 has obtained the same event that the hand-held sensor 5000 is substantially stationary and the time obtained from the image, there is a difference between the time when the position / orientation information is stored in the memory 1302 and the memory The difference between the time when the real space image is stored in 1302 and the difference between the time when the position / orientation information is stored in the memory 1302 and the time when the real space image is stored in the memory 1302. Difference (D). Therefore, the position and orientation measured when an image of the real space stored in the memory 1302 at a certain time is captured is stored in the memory 1302 before this “some time” by this “difference D”. Position and posture. Therefore, when the difference D is obtained, a value obtained by multiplying the difference D by 60 may be substituted for n (when the unit of D is seconds).

  Further, in this embodiment, information delay processing is performed using a FIFO, but the same object can be achieved other than using the FIFO, and the present invention is not limited to such means.

  As described above, according to the present embodiment, it is possible to generate the mixed reality space image using the real space image and the position / orientation information that should be acquired at almost the same time. For example, even if the user 1000 moves the hand-held sensor 5000 and moves it, the CAD model can be moved without being delayed, and the positional deviation between the real space and the virtual space can be reduced. Can be reduced.

[Second Embodiment]
In this embodiment, the time from when the camera captures an image until the captured image is stored in the memory 1302 is measured. The system for performing such measurement processing according to the present embodiment is the same as that of the first embodiment.

  FIG. 13 is a flowchart of such a measurement process. The CPU 1301 issues a control signal for controlling turning on / off of the LED 5005 via the parallel I / O 1311. Here, it is assumed that the time required from when the control signal is issued until the LED 5005 is turned on / off is zero.

  First, a variable indicating the internal state (internal state) is set to “idle”, and a control signal for turning off the LED 5005 is issued via the parallel I / O 1311 (step S300).

  Next, since the real space images captured by the camera are sequentially stored in the memory 1302 via the video capture cards 1320 and 1321 (step S301), the CPU 1301 detects this and captures the time at that time as the capture time. Is stored in the memory 1302 (step S302).

  Next, with reference to the image captured in step S301, it is checked whether the LED 5005 is lit (step S303). As the checking process, for example, the pixel value of each pixel constituting the image is referred to, and when the pixel value is viewed in YCrCb, one or more sets of pixels whose Y value is a predetermined value or more are specified and specified. If the size of the largest set among the sets is equal to or larger than a predetermined size, it is determined that the LED 5005 is lit.

  Next, a variable indicating the internal state is referred to (step S304), and if it is “idle”, the process proceeds to step S307. When the internal state is “idle”, that is, when the LED 5005 is turned off, first, the CPU 1301 issues a control signal for turning on the LED 5005 to the LED 5005 via the parallel I / O 1311 (step S307). The CPU 1301 stores the time when the control signal is issued in the memory 1302 (step S308). Then, the variable indicating the internal state is changed to “waiting for detection” (step S309).

  On the other hand, if the variable indicating the internal state is not “idle” in step S304, the process proceeds to step S305 to determine whether or not the variable indicating the internal state is “waiting for detection” (step S305). If the variable indicating the internal state is “waiting for detection”, that is, if the LED 5005 is lit, the process proceeds to step S310, and it is determined whether or not it is determined in step S303 that the LED 5005 is lit (step S303). If it is determined that the LED 5005 is lit, the process proceeds to step S311, and the CPU 1301 issues a control signal for turning off the LED 5005 to the LED 5005 via the parallel I / O 1311 (step S311). To do. Then, a difference between the time at that time and the time stored in the memory 1302 in step S302 is obtained (step S312).

  That is, this difference value represents the time required from when the LED 5005 is actually turned on until an image of the real space including the lit LED 5005 is obtained.

  Then, the CPU 1301 changes the variable indicating the internal state to “wait for completion” (step S313).

  On the other hand, if the variable indicating the internal state is not “waiting for detection” in step S305, the process proceeds to step S306 to determine whether or not the variable indicating the internal state is “waiting for completion”. "", The process proceeds to step S314, and it is determined whether or not it is determined in step S303 that the LED 5005 is lit (step S314). If it is determined that the LED 5005 is lit, the process proceeds to step S314. Proceeding to S315, the CPU 1301 changes the variable indicating the internal state to “idle” (step S315).

  As described above, according to the present embodiment, it is possible to measure the time required for the CPU 1301 to perform processing using the image captured by the camera and stored in the memory 1302 of the computer 1300.

  Moreover, although LED was used as a light-emitting body in this embodiment, you may use light-emitting bodies other than this. Moreover, it is not limited to a light emitter, and a liquid crystal screen or the like may be used. In other words, the first state (corresponding to “lighting” in the present embodiment) and the second state (corresponding to “lighting off” in the present embodiment) can be detected on the photographed image. I just need it.

[Third Embodiment]
In the present embodiment, a CAD viewer system that operates while synchronizing the acquisition timing of position and orientation information and the acquisition timing of an image in the real space, and a tool for obtaining a difference between the acquisition timings (that is, the delay time). An embodiment that switches between and uses will be described.

  In the following description, only parts different from the first embodiment will be described. That is, the second embodiment is the same as the first embodiment except for the points described below.

  FIG. 14 is a diagram showing a hand-held sensor 5900 used in the present embodiment. The hand-held sensor 5900 shown in the figure includes a calibration switch 5006 in the hand-held sensor 5000 used in the first embodiment shown in FIG. The calibration switch 5006 is connected to the parallel I / O 1311 provided in the computer 1300. When the calibration switch 5006 is pressed, a signal indicating that is sent from the calibration switch 5006 to the parallel I / O 1311. Therefore, the CPU 1301 can detect pressing of the calibration switch 5006 by detecting this signal.

  However, the calibration switch 5006 is not indispensable, and a keyboard or a mouse may be connected to the computer 1300 and used instead of the calibration switch 5006.

  Therefore, in this embodiment, the computer 1300 executes the application program of the CAD viewer system when the calibration switch 5006 is not pressed, and performs processing according to the flowchart of FIG. 9 when the calibration switch 5006 is pressed. A process for measuring the delay time is executed.

  FIG. 15 is a flowchart of processing performed by the computer 1300 according to this embodiment.

  When the CPU 1301 detects that a signal indicating that the calibration switch 5006 has been pressed is input via the parallel I / O 1311, the CPU 1301 advances the process to step S 401 via step S 400, and executes the calibration tool. 9 is executed (step S401). In the present embodiment, in the process according to the flowchart of FIG. 9, the process proceeds to step S402 without returning to step S200 after the process in step S208.

  Here, as a result of executing the processing according to the flowchart of FIG. 9, when the delay time = 0 (or substantially 0), the processing returns to step S400, but when the delay time ≠ 0, the processing is stepped. The process proceeds to step S403 via S402, and the delay time obtained in step S401 is set to the delay time used in the CAD viewer system (step S403).

  On the other hand, if the CPU 1301 does not detect that a signal indicating that the calibration switch 5006 is pressed is input via the parallel I / O 1311, the process proceeds to step S404 via step S400, and the calibration tool The work history is cleared (step S404), and the CAD viewer system is activated (step S405). The operation of the CAD viewer system follows the flowchart of FIG. In the present embodiment, in the process according to the flowchart of FIG. 8, the process proceeds to step S400 without returning to step S100 after the process in step S114.

  The clearing of the calibration tool work history in step S404 means the previous hand-held sensor position, the previous marker detection position, and the time history ts (i) and tv (i) in the flowchart of FIG. is there. These are histories that do not make sense if the calibration tool is not operating continuously, and must be cleared when the calibration tool is not operating.

[Fourth Embodiment]
In this embodiment, the CAD viewer system and the calibration tool described in the third embodiment are operated in parallel, and the delay time between the acquisition timing of the real space image and the acquisition timing of the position and orientation information is always obtained. Reflect in the CAD viewer system.

  In the following description, only parts different from the first embodiment will be described. That is, the second embodiment is the same as the first embodiment except for the points described below.

  As described above, in the present embodiment, the CAD viewer system and the calibration tool are operated in parallel. For example, each is operated in multithread.

  FIG. 16 is a flowchart showing the operation of the calibration tool, and FIG. 17 is a flowchart showing the operation of the CAD viewer system. Note that the processing according to each flowchart is performed by the computer 1300 as in the third embodiment.

  In FIG. 16, Steps S500 to S502 are the same as Steps S401 to S403, respectively. However, in order to correspond to the multi-thread operation, exclusion between threads for the variable for setting the delay time set in Step S403. Control is in progress.

  In FIG. 17, steps S510 and S511 are the same as steps S404 and S405, respectively, but exclusive control between threads is performed for the variable from which the delay time is read in step S405 to correspond to the multithread operation. .

  As described above, according to the present embodiment and the third embodiment, a CAD viewer system is constructed in which the acquisition timing of the real space image and the acquisition timing of the position and orientation information are synchronized in consideration of the delay time. can do. In particular, according to the present embodiment, it is possible to construct a CAD viewer system in which the above acquisition timings are synchronized without being aware of the existence of a tool for obtaining the delay time.

[Other Embodiments]
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

FIG. 3 is a diagram illustrating a state in which a CAD model (CAD model of a toner cartridge of a printer) 100 is displayed as a CG object and used for a CAD model preview. It is a figure which shows the structure of HMD. It is a figure which shows the example of a display of the image which the CAD model 300 drawn as a CG image and the real image image | photographed with each camera 1110,1111 displayed on each LCD1130,1311 in HMD were synthesize | combined. It is a figure which shows the apparatus structure of the CAD viewer using MR technique. It is a figure which shows the position and orientation relationship between the handheld sensor 5000 and the CAD model 4000. It is a figure which shows the external appearance of the handheld sensor 5000, and an internal structure. 1 is a block diagram showing a basic configuration of a system including a computer 1300. FIG. 10 is a flowchart of processing performed by a computer 1300 to provide a user 1000 with an image of a mixed reality space. It is a flowchart of the process which determines the number of data n which IFO can hold | maintain (in other words, the parameter which determines the time to shift). It is a flowchart which shows the detail of the process in step S208. It is a figure which shows the outline of the calculation method of the delay time in the flowchart shown in FIG. It is a figure which shows the outline of the calculation method of the delay time in the flowchart shown in FIG. 10 is a flowchart of a process for measuring time from when a camera performs imaging until the captured image is stored in a memory 1302; It is a figure which shows the hand-held sensor 5900 used in the 3rd Embodiment of this invention. It is a flowchart of the process which the computer 1300 which concerns on the 3rd Embodiment of this invention performs. It is a flowchart which shows operation | movement of a calibration tool. It is a flowchart which shows operation | movement of a calibration tool.

Claims (11)

A first input step for continuously inputting the position and orientation information of the pointing tool;
A second input step of continuously inputting an image of the real space including the pointing device;
Based on the position and orientation information input in the first input step, a first detection step of detecting a timing at which the amount of movement of the pointing tool is minimized;
A second detection step of detecting a timing at which the amount of movement of the pointing tool is minimized based on the video input in the second input step;
A calculation step for obtaining a difference between the acquisition timing of the image in the real space and the acquisition timing of the position and orientation information based on the timing detected in the first detection step and the timing detected in the second detection step; ,
And a first output step for outputting the deviation obtained in the calculation step.   2. The timing according to claim 1, wherein in the first detection step, a timing at which the amount of movement of the pointing tool is minimized is detected based on a change in position and orientation information input in the first input step. Information processing method.   In the second detection step, the amount of movement of the pointing tool is minimized based on a change in the position of the marker having a predetermined color attached to the pointing tool on the image input in the second input step. The information processing method according to claim 1, further comprising: detecting a timing at which The calculation step includes:
When the timings detected in the first detection step are arranged in chronological order, it is assumed that ts0, ts1,... When arranged as tv0, tv1,…., Tv (n-1), tv (n),
searching for ts (k) satisfying tv (n-1) <ts (k) <tv (n) from ts0 to ts (n);
Comparing | tv (n-1) -ts (k) | with | tv (n) -ts (k) |
If | tv (n-1) -ts (k) | is larger, tv (n) -ts (k) is determined as the deviation;
If | tv (n) -ts (k) | is greater or equal, tv (n-1) -ts (k) is determined as the deviation. 4. The information processing method according to any one of items 3. Furthermore,
A third input step of inputting a real space image;
A fourth input step for inputting the position and orientation information of the user's viewpoint;
Virtual space video viewed from the viewpoint based on the position and orientation information input in the fourth input step at a time before the shift obtained in the calculation step from the time when the video was input in the third input step A generating step for generating
5. The method according to claim 1, further comprising: a second output step of superimposing the video generated in the generation step on the video input in the third input step and then outputting. The information processing method described. Furthermore,
A third input step of inputting a real space image;
A fourth input step for inputting the position and orientation information of the user's viewpoint;
A generation step of generating a video of a virtual space seen from a viewpoint based on the position and orientation information input in the fourth input step;
The video generated in the generation step on the video input in the third input step at the time before the deviation obtained in the calculation step from the time when the position and orientation information was input in the fourth input step. The information processing method according to any one of claims 1 to 4, further comprising: a second output step of outputting after superimposing. A control process for controlling a device that can be controlled to take either the first state or the second state;
A first detection step of detecting the timing when the device is in the first state according to the control in the control step;
A second detection step of detecting a timing when the device is in the first state based on an image of the real space including the pointing tool;
An information processing method comprising: an output step of outputting a difference between the timing detected in the first detection step and the timing detected in the second detection step. First input means for continuously inputting the position and orientation information of the pointing tool;
Second input means for continuously inputting an image of a real space including the pointing device;
First detection means for detecting a timing at which the amount of movement of the pointing tool is minimized based on the position and orientation information input by the first input means;
Second detection means for detecting a timing at which the amount of movement of the pointing tool is minimized based on the video input by the second input means;
Calculation means for obtaining a difference between the acquisition timing of the image of the real space and the acquisition timing of the position and orientation information based on the timing detected by the first detection means and the timing detected by the second detection means; ,
An information processing apparatus comprising: first output means for outputting the deviation obtained by the calculation means. Control means for controlling a device that can be controlled to take either the first state or the second state;
First detection means for detecting timing when the device is in a first state in accordance with control by the control means;
Second detection means for detecting timing when the device is in the first state based on an image of the real space including the pointing tool;
An information processing apparatus comprising: output means for outputting a difference between the timing detected by the first detection means and the timing detected by the second detection means.   A program that causes a computer to execute the information processing method according to any one of claims 1 to 7.   A computer-readable storage medium storing the program according to claim 10.

Images