JP4144888B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to a technique for superimposing and presenting an image of a virtual space on a real space.

In the mixed reality system, it is necessary to measure the viewpoint position and line-of-sight direction of the operator and the position of an object in the space. As a position / orientation measurement system, one of general methods is to use a position / orientation measurement apparatus represented by FASTRAK (registered trademark) manufactured by Polhemus, USA. Also, a method of measuring only the posture with a gyro etc. and measuring position information and posture drift error from an image is also taken.
JP 2002-271817 A

  However, it is generally difficult to match the time when the image of the real space is acquired in the conventional mixed reality system with the time when the position and orientation measured by the position and orientation measurement device is acquired.

  In particular, in the method in which a real space image is once captured and rendered on a computer and then the virtual space image is superimposed, there is a time lag between capturing the real space image and superimposing the virtual space image. The video acquired at the time later than the acquisition time of the real space video is used, and as a result, the real space video and the virtual space video are inconsistent. As a result, an observer who sees an image obtained by synthesizing these sounds uncomfortable.

  In such a case, it may be possible to eliminate the inconsistency between the real space image and the virtual space image by generating a virtual space image using a slightly old measurement value. It was not possible to easily decide whether or not

  The present invention has been made in view of the above problems, and acquisition of a real space image is to be performed in order to generate a virtual space image based on the position and orientation acquired at substantially the same time as the acquisition time of the real space image. It is an object of the present invention to provide a technique for easily obtaining a difference between a time and a position and orientation acquisition time.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, a first storage step of storing a captured image included in a moving image acquired from an imaging device that captures a moving image in real space and an acquisition time of the captured image in association with each other;
A second storing step of storing the position and orientation data of the imaging device and the acquisition time of the position and orientation data in association with each other;
A deviation amount adjusting step of adjusting data d indicating the deviation amount in accordance with an instruction from the user;
A live-action image acquisition step of acquiring a real-image image corresponding to the acquisition time t using the real-action image stored in the first storage step;
Using the position and orientation data stored in the second storage step, position and orientation data corresponding to the acquisition time (td) is acquired, and the position and orientation data corresponding to the acquisition time (td) is obtained. A virtual image generation step of generating a virtual image using,
Mixed reality space image generation that synthesizes the actual image acquired in the actual image acquisition step and the virtual image generated in the virtual image generation step to generate an image of the mixed reality space corresponding to the acquisition time t Process,
A display step of displaying a list of a plurality of mixed reality space images corresponding to different acquisition times t generated using the real image acquisition step, the virtual image generation step, and the mixed reality space image generation step;
A recording step of recording the data d in response to an instruction from a user.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, a first storage unit that stores a real image included in a moving image acquired from an imaging device that captures a moving image in real space and an acquisition time of the real image in association with each other;
Second storage means for storing the position and orientation data of the imaging apparatus and the acquisition time of the position and orientation data in association with each other;
A deviation amount adjusting step of adjusting data d indicating the deviation amount in accordance with an instruction from the user;
Using the live-action image stored by the first storage means, the live-action image acquisition means for acquiring the real-action image corresponding to the acquisition time t;
Using the position and orientation data stored by the second storage means, the position and orientation data corresponding to the acquisition time (td) is acquired, and the position and orientation data corresponding to the acquisition time (td) is used. Virtual image generation means for generating a virtual image,
A mixed reality space image generating unit that combines the real image acquired by the actual image acquisition unit and the virtual image generated by the virtual image generation unit to generate an image of the mixed reality space corresponding to the acquisition time t; ,
Display means for displaying a list of a plurality of mixed reality space images corresponding to different acquisition times t, generated using the real image acquisition means, the virtual image generation means, and the mixed reality space image generation means;
And recording means for recording the data d in response to an instruction from a user.

  According to the configuration of the present invention, in order to generate a virtual space image based on the position and orientation acquired at substantially the same time as the acquisition time of the real space image, the acquisition time of the real space image and the acquisition time of the position and orientation Can easily be determined.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 is a diagram illustrating an external appearance of a system according to the present embodiment, which executes a process of providing a mixed reality space in which a virtual space is superimposed on a real space to an observer (user).

  In the figure, 120 is a transmitter that generates a magnetic field. In this embodiment, a position / orientation measurement system represented by FASTRAK (registered trademark) manufactured by US Polhemus is used as the position / orientation measurement system. However, the essence of the following description is limited to this. Instead, it will be apparent from the following description that a technique may be used in which only the posture is measured with a gyroscope or the like, and the positional information and the posture drift error are measured from the image.

  Reference numeral 100 denotes a head-mounted display device (hereinafter referred to as HMD: Head Mounted Display) that is attached to the observer's head and provides an image obtained by combining the real space and the virtual space in front of the observer's eyes. , Cameras 102R and 102L, display devices 101R and 101L, and a magnetic receiver 105.

  The cameras 102 </ b> R and 102 </ b> L continuously capture the real space seen from the positions of the right eye and left eye of the observer wearing the HMD 100 on the head, and the captured images of each frame are output to the computer 103 at the subsequent stage. . Hereinafter, the right-eye camera 102R and the left-eye camera 102L may be collectively referred to as “camera 102” unless they are distinguished from each other.

  The display devices 101R and 101L are mounted on the HMD 100 so that they are positioned in front of the right eye and the left eye when the observer wears the HMD 100 on the head. Display the based image. Therefore, an image generated by the computer 103 is provided in front of the right and left eyes of the observer. Hereinafter, the right-eye display device 101R and the left-eye display device 101L may be collectively referred to as “display device 101” unless they are distinguished from each other.

  The magnetic receiver 105 detects a change in the magnetic field generated by the transmitter 120 and outputs a detection result signal to the subsequent position / orientation measurement apparatus 104. The detected signal is a magnetic receiver in a coordinate system (hereinafter referred to as a sensor coordinate system) in which the position of the transmitter 120 is an origin, and three axes orthogonal to each other at the position of the origin are x, y, and z axes. 105 is a signal indicating a change in magnetic field detected according to the position and orientation of 105.

  Based on this signal, the position / orientation measurement apparatus 104 obtains the position / orientation of the magnetic receiver 105 in the sensor coordinate system, and data indicating the obtained position / orientation is output to the computer 103 at the subsequent stage.

  Reference numeral 107 denotes a marker arranged for correcting the measurement value obtained by the magnetic receiver 105, but this marker is not necessary when correction processing is not performed.

  Reference numeral 111 denotes a virtual object that is arranged to be superimposed on the real object 110. As a result, the color or surface pattern of the real object 110 can be changed and presented to the observer. Such a function is used for the purpose of verifying the design by applying various designs as a virtual space image to a mockup that reproduces only the shape.

  A magnetic receiver 106 similar to the magnetic receiver 105 is attached to the real object 110, and the magnetic receiver 106 detects a change in the magnetic field generated by the transmitter 120, and sends the detected signal to the subsequent position / orientation measurement device 104. Output. The detected signal is a signal indicating a change in the magnetic field detected in accordance with the position and orientation of the magnetic receiver 106 in the sensor coordinate system.

  Based on this signal, the position / orientation measurement apparatus 104 obtains the position / orientation of the magnetic receiver 106 in the sensor coordinate system, and data indicating the obtained position / orientation is output to the computer 103 at the subsequent stage.

  A computer 103 performs processing such as generating image signals to be output to the display devices 101R and 101L of the HMD 100 and receiving data from the position / orientation measurement device 104 and managing them. This computer is generally composed of, for example, a PC (personal computer), WS (workstation), or the like. FIG. 10 is a diagram illustrating a hardware configuration of the computer 103.

  A CPU 1101 controls the entire computer 103 using programs and data stored in the RAM 1102 and the ROM 1103 and controls data communication with an external device connected to the I / F 1107. In addition, each process described later is performed as a process to be performed by the computer 103. In addition, a timer (not shown) is provided in the CPU 1101, and the current time can be measured.

  Reference numeral 1102 denotes a RAM, an area for temporarily storing programs and data loaded from the external storage device 1105, an area for temporarily storing various data received via the I / F 1107, and various types of CPU 1101. Various areas such as a work area necessary for executing the process can be provided as appropriate.

  A ROM 1103 stores a boot program, setting data of the computer 103, and the like.

  An operation unit 1104 includes a keyboard, a mouse, a joystick, and the like, and can input various instructions to the CPU 1101.

  Reference numeral 1105 denotes an external storage device that functions as a large-capacity information storage device such as a hard disk drive device, and stores an OS (Operating System) and programs and data for causing the CPU 1101 to execute processes described later. Some or all of these are loaded into the RAM 1102 under the control of the CPU 1101. In addition, what is described as known data (information) in the following description (or data that should be necessary for the processing described below) is also stored in the external storage device 1105, and is controlled by the CPU 1101 as necessary. As a result, the data is loaded into the RAM 1102.

  A display unit 1106 includes a CRT, a liquid crystal screen, and the like, and can display a processing result by the CPU 1101 using images, characters, and the like.

  Reference numeral 1107 denotes an I / F, to which the position / orientation measuring apparatus 104, the HMD 100, and the like are connected. The computer 103 performs data communication with the position / orientation measuring apparatus 104, the HMD 100, and the like via the I / F 1107. Can do.

  A bus 1108 connects the above-described units.

  The computer 103 having the above configuration captures images of the real space obtained from the cameras 102R and 102L, and also images of the virtual object 111 that can be seen from the cameras 102R and 102L based on the position and orientation obtained from the magnetic receivers 105 and 106. Is generated. Then, the generated image is superimposed on the previously captured real space image, and the superimposed image is output to the display devices 101R and 101L. As a result, a mixed reality space image corresponding to the position and orientation of each eye is displayed in front of the right and left eyes of the observer wearing the HMD 100 on the head.

  FIG. 2 is a flowchart of a series of processes for generating an image of such a mixed reality space and outputting it to the HMD 100. Note that a program and data for causing the CPU 1101 to execute the processing according to the flowchart of FIG. 10 are stored in the external storage device 1105, and the computer 103 can be loaded by loading it into the RAM 1102 to be processed by the CPU 1101. Each process described below is executed.

  First, real space images (actual images) captured by the cameras 102R and 102L are input to the computer 103 via the I / F 1107, and are sequentially stored in the RAM 1102 (step S201).

  Also, the “position and orientation in its own sensor coordinate system” measured by the magnetic receiver 105 and the “position and orientation in its own sensor coordinate system” measured by the magnetic receiver 106 are input to the RAM 1102 via the I / F 1107. Is temporarily recorded in the RAM 1102 (step S202).

  Next, the marker 107 is image-recognized from the real space image acquired in the RAM 1102 in step S201, and the “position and orientation of the magnetic receiver 105” obtained in step S202 is corrected from the position of the marker 107 in the image (step S203). ).

  The processing in this step is not necessary when using a sensor that can accurately output information on all positions and orientations. However, generally used magnetic sensors are often used to correct errors due to the surrounding environment. It is processing. Further, when the sensor can measure only the posture, or when the posture deviation occurs in the posture for a long time, the position is measured by the image, or the posture deviation is corrected by the image. Further, this correction method is not limited to this method, and various methods have been proposed.

  Then, the real space image acquired on the RAM 1102 in step S201 is drawn in a predetermined area in the RAM 1102 (step S204). Next, the “data indicating the position and orientation of the magnetic receiver 105 in the sensor coordinate system” acquired in the RAM 1102 in step S202 is used as data (bias) indicating the “position and orientation relationship between the magnetic receiver 105 and the cameras 102R and 102L” measured in advance. Data) to obtain data indicating “position and orientation of the cameras 102R and 102L in the sensor coordinate system”. Then, using this data, an image of the virtual space that can be seen according to the position and orientation of the cameras 102R and 102L is drawn, and the drawing is drawn on the image of the real space drawn in step S204 (step S205).

  As a result, an image of the mixed reality space that can be seen from the respective cameras 102R and 102L is generated in a predetermined area in the RAM 1102.

  Note that when generating an image of the virtual space, the virtual object 111 is placed in the virtual space with the position and orientation measured by the magnetic receiver 106 attached to the real object 110. When the arrangement position of the object 111 is seen, the virtual object 111 can be seen at this position.

  The mixed reality space image (synthesized image) generated in a predetermined area in the RAM 1102 is output to the display device 101 of the HMD 100 or the display unit 1106 of the computer 103 via the I / F 1107 (step S207).

  The above process is a process for displaying the mixed reality space image for one frame on the display device 101 or the display unit 1106. Therefore, by repeating the above process a plurality of times, the mixed reality space image for a plurality of frames is obtained. It can be displayed on the display device 101 or the display unit 1106.

  Then, when the CPU 1101 detects that the above process end instruction has been input by operating the operation unit 1104, for example, the process ends.

  In this embodiment, a magnetic sensor is used as a sensor, but other types such as an ultrasonic sensor may be used.

  Here, in the above processing, the processing in step S201, that is, the acquisition of the real space image takes more time than the processing in step S202 (measurement value acquisition processing by the sensor). Thus, there is a difference (time lag) between the time when the real space image is captured on the RAM 1102 and the time when the position and orientation of the magnetic receiver 105 are captured in step S202.

  That is, even if the real space is imaged and the position and orientation are measured at the same time, the acquisition of the real space image is delayed from the acquisition of the position and orientation. Then, when the acquisition of the real space image is completed, the next (or subsequent) position and orientation are acquired. Therefore, even when the observer moves his / her head, even if the position / orientation is measured and the real space is imaged at the same time, the position / orientation of the viewpoint of the real space indicated by the video captured in step S201, and the step Since the position and orientation of the viewpoint based on the position and orientation of the magnetic receiver 105 acquired in S202 do not coincide with each other, even if a virtual space image is generated based on the position and orientation acquired in step S202, it differs from the real space image. As a result, the real object 110 and the virtual object 111 are drawn out of alignment with each other even if the images are combined.

  FIG. 3 is a diagram for explaining a deviation between the real object 110 and the virtual object 111 that occurs due to the occurrence of this time lag. The images 301 to 308 are not displayed at the same time, but are displayed in this order. Each image is displayed on the display device 101 provided in the HMD 100 or the display unit 1106 provided in the computer 103.

  The virtual object 111 is arranged with the same position and orientation as the real object 110 so as to be superimposed on the position of the real object 110. Therefore, ideally, the virtual object 111 is always displayed on the display screen of the display device 101 so as to overlap the real object 110.

  First, in the image 301, since the observer's head is not moving, the real object 110 and the virtual object 111 are displayed with substantially the same position and orientation. Here, when the observer moves his / her head, the positions of the virtual object 111 and the real object 110 in the display screen of the display device 101 naturally change. Here, when the head is turned to the left, both the virtual object 111 and the real object 110 move to the right in the display screen. However, since the time lag occurs, the real object 110 is a virtual object as shown in an image 302. 111 is viewed from the viewpoint of the position / orientation acquired before the position / orientation of the viewpoint used for rendering 111, and as a result, the display is shifted. This is because the virtual space image is generated following the movement of the head, but the acquisition of the real space image to be acquired after the movement of the head has not yet been completed (that is, because of the time lag described above). ), Because the image of the previous frame remains.

  After that, when the observer continues to look over his / her head to the left side, the real object 110 is displayed with a delay from the movement of the virtual object 111 as shown in images 303 and 304.

  Here, when the observer's head is suddenly turned to the left and then turned to the right, the virtual object 111 moves in the direction indicated by the arrow 350 in the display screen as shown in the image 305. For the above reason, the real object 110 moves with a delay, and therefore moves in the direction indicated by the arrow 351. Therefore, this is the reason why each of the real object 110 and the virtual object 111 appears to be located.

  If the observer continues to turn his / her head to the right after that, the real object 110 is displayed with a delay in the movement of the virtual object 111 as shown in images 306 and 307.

  When the observer stops rotating his / her head, the virtual object 111 and the real object 110 are displayed in the same manner as in the image 301 as shown in the image 308.

  Therefore, in this embodiment, when the time lag is obtained and a virtual space image to be superimposed on the real space is generated, the position and orientation acquired before the real time image acquisition time by the obtained time lag. Based on the above, an image of the virtual space is generated. As a result, the viewpoint positions and orientations of the respective images substantially coincide with each other. Therefore, even if the respective images are combined, the real object 110 and the virtual object 111 are not drawn with being shifted from each other.

  Below, the process which calculates | requires this time lag is demonstrated.

  First, the observer wears the HMD 100 on his / her head and moves his / her head appropriately. Easy to shake from side to side. When the observer is performing such an operation, the computer 103 executes each process described below.

  FIG. 4 shows a process performed by the computer 103 while the observer performs the above operation, that is, an image of the real space captured by the camera 102 provided in the HMD 100 is acquired, and the acquired image is acquired together with the acquisition time. 10 is a flowchart of processing to be stored in an external storage device 1105. Note that a program and data for causing the CPU 1101 to execute processing according to the flowchart of FIG. 10 are loaded from the external storage device 1105 onto the RAM 1102, and the CPU 103 executes the processing using this, whereby the computer 103 Each process described below is executed.

  FIG. 5 shows the processing performed by the computer 103 while the observer performs the above operation, that is, the position and orientation of the HMD 100 measured by the magnetic receiver 105 provided in the HMD 100, and the real object measured by the magnetic receiver 106. 10 is a flowchart of processing for acquiring a position and orientation of 110 and storing the acquired position and orientation in an external storage device 1105 together with an acquisition time. Note that a program and data for causing the CPU 1101 to execute processing according to the flowchart of FIG. 10 are loaded from the external storage device 1105 onto the RAM 1102, and the CPU 103 executes the processing using this, whereby the computer 103 Each process described below is executed.

  4 and 5, it is desirable that the processing according to the flowcharts shown in FIGS. 4 and 5 is executed without delay. Therefore, it is preferable that the processing is executed as an independent task on the real-time OS, and as a separate thread on the multi-thread OS. .

  First, processing according to the flowchart shown in FIG. 4 will be described.

  As described above, the camera 102 provided in the HMD 100 captures a moving image in the real space. Therefore, since the captured image of each frame is input to the computer 103 via the I / F 1107, the CPU 1101 acquires this on the RAM 1102 (step S401). In general, when acquiring a high-frame-rate moving image, there are many examples in which images are captured independently by another thread or task. Even in this embodiment, the capturing process from the basic software API is completed in step S401. In response to the call, it is assumed that data that has already been temporarily stored in the buffer is taken into the computer 103.

  Next, as described above, since the CPU 1101 keeps the current time with the internal timer, the time when the image is acquired in step S401 (the time when the image data input via the I / F 1107 is recorded on the RAM 1102 is completed). ) (Step S402), and the time measured is set as data together with the image data acquired in step S401 and stored in the external storage device 1105 (step S403).

  Depending on the mechanism of the basic software, it may be desirable to measure the time when the end of the capturing process in the basic software is detected before the image acquisition described in step S401.

  Then, a recording area for recording the real space image data to be input to the computer 103 and the acquisition time of the image data is secured in the external storage device 1105, and the data is recorded in the recording area. The recording position is updated so as to be performed (step S404). If the recording area is secured in advance, the recording area securing process is not necessary one by one.

  The above process is performed until a process end instruction is input. Note that the end of processing according to the flowchart of FIG. 4 can be stopped when starting the processing to determine the observer's delay time, which will be described later, to reduce the load on the PC, and there is a margin in processing amount It is also possible to secure the recording area in a ring shape and always keep data within a certain past time.

  Next, processing according to the flowchart shown in FIG. 5 will be described.

  As described above, the magnetic receiver 105 outputs a signal corresponding to its own position and orientation to the position and orientation measurement device 104, and the position and orientation measurement device 104 inputs data based on this signal to the computer 103. Further, as described above, the magnetic receiver 106 outputs a signal corresponding to its own position and orientation to the position and orientation measurement device 104, and the position and orientation measurement device 104 inputs data based on this signal to the computer 103. Therefore, the CPU 1101 acquires data (sensor value) indicating the position and orientation of the magnetic receiver 105 and data (sensor value) indicating the position and orientation of the magnetic receiver 106 on the RAM 1102 (step S501).

  Actually, the sensor is connected to the computer 103 by some communication means such as serial communication, and instead of actively acquiring the sensor value, the communication path is always monitored and the moment when the output from the sensor is received It is desirable that the error can be detected without error or extra delay.

  The CPU 1101 measures the acquisition time of position and orientation data (step S502), and stores the measured time data together with the position and orientation data acquired in step S501 in the external storage device 1105 (step S503). .

  Then, a recording area for recording the position / orientation data to be input to the computer 103 and the acquisition time of the position / orientation data is secured in the external storage device 1105 and the data is recorded in the recording area. Thus, the recording position is updated (step S504). If the recording area is secured in advance, the recording area securing process is not necessary one by one.

  The above process is performed until a process end instruction is input. The end determination of the process in FIG. 5 is the same as the end process determination in FIG.

  Next, the process for obtaining the time lag will be described with reference to FIG. 6 showing a flowchart of the process. Note that programs and data for causing the CPU 1101 to execute processing according to the flowchart of FIG. 10 are loaded from the external storage device 1105 to the RAM 1102, and the CPU 103 performs processing using the programs and data. Will be executed.

  Note that the processing according to the flowchart of FIG. 4 is performed after the processing according to the flowcharts shown in FIGS. That is, the process according to the flowchart shown in FIG. 6 is performed by the “set group of the real space image and the acquisition time of the image” stored in the external storage device 1105 by the process according to the flowchart of FIG. The processing is executed using a “group of positions and orientations and acquisition times” stored in the external storage device 1105 by processing according to the flowchart.

  First, the observer wears the HMD 100 on his / her head and moves his / her head appropriately. Easy to shake from side to side. When the observer is performing such an operation, the computer 103 executes each process described below.

  Since the operator of the computer 103 uses the operation unit 1104 to input the value of “variable d indicating deviation amount” used in the following processing, the CPU 1100 temporarily stores it in the RAM 1102 (step S601).

  Next, an image of the real space at time t is drawn on the RAM 1102 (step S602). In this embodiment, the time t is set by the operator of the computer 103 using the operation unit 1104. Note that this time t is within a range that the “acquisition time” stored in the external storage device 1105 can take by the processing according to the flowchart of FIG. 4 (or FIG. 5).

  Here, when the time t is one of the “acquisition times” stored in the external storage device 1105 by the processing according to the flowchart of FIG. 4 (or FIG. 5), the acquisition time t is set as a set. The processing in step S602 is performed by reading the real space image stored in the external storage device 1105 into the RAM 1102.

  Further, the time t is not any of the “acquisition time” stored in the external storage device 1105 by the processing according to the flowchart of FIG. When the near time is ta, the processing in step S602 is performed by reading the real space image stored in the external storage device 1105 as a set with the acquisition time ta into the RAM 1102.

  The time t is not any of the “acquisition times” stored in the external storage device 1105 by the processing according to the flowchart of FIG. 4 (or FIG. 5). When the adjacent times are ta1 and tb (ta <t <tb), the CPU 1101 sets the acquisition time ta as a set and the real space image stored in the external storage device 1105 and the acquisition time tb. Then, the real space image at time t is interpolated using the real space image stored in the external storage device 1105, and the processing in step S602 is performed. Since the image interpolation generation process is well known, a description thereof will be omitted.

  As described above, the real space image at the designated time t is read out to the RAM 1102.

  Next, using the position and orientation at time (t-d), an image of the virtual space that can be seen from the viewpoint having this position and orientation is generated (step S603). That is, the virtual object 111 is arranged at the position and orientation acquired from the magnetic receiver 106 at time (td), and then the bias data is added to the position and orientation acquired from the magnetic receiver 105 at time (td). An image of the virtual space that can be seen from the viewpoint of the obtained position and orientation is generated. Since generation processing of an image of a virtual space that can be seen from a predetermined position and orientation is well known, a description thereof will be omitted.

  Here, when the time (td) is one of the “acquisition times” stored in the external storage device 1105 by the processing according to the flowchart of FIG. 4 (or FIG. 5), the acquisition time The virtual object 111 is placed at the position and orientation of the real object 110 stored in the external storage device 1105 as a set with (td), and then stored in the external storage device 1105 as a set with time (td). The processing in step S603 is performed by generating a virtual space image that can be seen from the viewpoint of the position and orientation obtained by adding the bias data to the “position and orientation obtained from the magnetic receiver 105”.

  Further, the time (td) is not any of the “acquisition times” stored in the external storage device 1105 by the processing according to the flowchart of FIG. 4 (or FIG. 5). When the time closest to the time (td) is ta, the virtual object 111 is placed at the position and orientation of the real object 110 stored in the external storage device 1105 as a set with the acquisition time ta, and then the time By generating an image of a virtual space that can be seen from the viewpoint of the position and orientation obtained by adding the bias data to the “position and orientation obtained from the magnetic receiver 105” stored in the external storage device 1105 as a set with ta, The process in step S603 is performed.

  Further, the time (td) is not any of the “acquisition times” stored in the external storage device 1105 by the processing according to the flowchart of FIG. 4 (or FIG. 5). When the times adjacent to the time (td) are ta and tb (ta <td <tb), the CPU 1101 is first saved in the external storage device 1105 as a set with the acquisition time ta. Using the “position and orientation obtained from the magnetic receiver 105” and the “position and orientation obtained from the magnetic receiver 105” stored in the external storage device 1105 as a set together with the acquisition time tb at the time (t−d) “Position and orientation obtained from magnetic receiver 105” is generated by interpolation. The position / orientation interpolation generation process will be described later.

  Then, the virtual object 111 is arranged at the position and orientation of the real object 110 stored in the external storage device 1105 as a set with the acquisition time ta (or tb), and then the bias data is added to the position and orientation generated by interpolation. The processing in step S603 is performed by generating a virtual space image that can be viewed from the viewpoint of the position and orientation obtained in step S603. In this case, since the position and orientation of the real object 110 are constant regardless of the time, the interpolation generation processing relating to this position and orientation is not necessary, but if it is not constant, it is necessary to generate the interpolation in the same manner.

  As described above, the image of the virtual space that can be seen from the position and orientation at time (t-d) is drawn on the RAM 1102.

  In the above processing, in order to interpolate the position information, it is only necessary to linearly interpolate each of x, y, z coordinate values, which are typically measured as position information, using the measurement time as a parameter. Regarding posture information, Euler angle expression expressed by azimuth, elevation, and roll is often used, but it is known that Euler angle expression generally cannot be interpolated. In order to perform interpolation, posture information can be interpolated by converting posture information into quaternions and interpolating each element of quaternions.

  Further, by treating the quaternion as a four-dimensional vector and performing spherical linear interpolation on the four-dimensional hypersphere, more accurate interpolation can be performed. Furthermore, by applying cubic interpolation and spline interpolation instead of linear interpolation to interpolate the position and orientation, more accurate interpolation can be performed.

  Then, the virtual space image generated in step S603 is superimposed and drawn on the real space image previously drawn on the RAM 1102 in step S602 (step S604). Thus, an image of the mixed reality space for one frame (an image of the mixed reality space seen from the camera 102R, an image of the mixed reality space seen from the camera 102L) can be generated.

  In step S604, the generated mixed reality space image is output to the display device 101 and the display unit 1106.

  Next, the time t is updated. For example, the time t is updated as much as possible as the next acquisition time. (Step S605). In this case, assuming that the position and orientation acquisition time interval by the magnetic receivers 105 and 106 is Δt, (t + Δt) is updated as a new t.

  Next, it is checked whether a predetermined number of mixed reality space images have been generated (step S606). If not, the process returns to step S602, and the subsequent processes are repeated.

  Therefore, when a predetermined number of mixed reality space images are generated, they are in the position and orientation at the time (td) on the real space image at the time t (t = t1, t2,..., T). An image group in which images of a virtual space that can be seen from the viewpoint based on the image are superimposed is displayed on the display device 101 and the display unit 1106.

  FIG. 7 is a diagram showing a display example of a GUI having a user interface for inputting the deviation amount and having an area for displaying a list of a plurality of generated mixed reality spaces (four in the figure). is there.

  FIG. 7A shows a GUI display example that displays a mixed reality space image group that is generated when d = 10 ms is input as a shift amount using the user interface described below in step S601. Reference numeral 701 denotes a GUI window. In the window 701, images of the mixed reality space are displayed from left to right in the order of generation. The mixed reality space image may be visible from the camera 102R or may be visible from the camera 102L.

  Reference numeral 711 denotes a slider bar for inputting a shift amount. For example, an operation can be performed such that the shift amount is set larger as it is moved to the right side, and the shift amount is set smaller as it is moved to the left side.

  Reference numeral 712 denotes a text box for inputting a deviation amount as a numerical value. By inputting a numerical value here, the input numerical value can be set as the deviation amount.

  Therefore, in step S601, a process is performed in which a value input using the user interface such as the slider bar 711 or the text box 712 is set as a deviation amount. Note that the user interface for the operator of the computer 103 to input the shift amount is not limited to this, and various types can be considered.

  However, at the deviation amount d = 10 ms, the real object 110 and the virtual object 111 do not match because the deviation amount is insufficient as shown in FIG.

  FIG. 7B shows a GUI display example that displays a mixed reality space image group generated when d = 15 ms is input as the shift amount using the user interface in step S601. However, when the deviation amount is d = 15 ms, the real object 110 and the virtual object 111 do not match because the deviation amount is insufficient as shown in FIG. 7B.

  FIG. 7C shows a GUI display example that displays a mixed reality space image group generated when d = 20 ms is input as the shift amount using the user interface in step S601. In this case, as shown in FIG. 7C, the real object 110 and the virtual object 111 substantially coincide with each other, and the deviation d = 20 ms is an appropriate value. This means that the time lag is 20 ms.

  Therefore, the operator of the computer 103 adjusts and inputs the shift amount so that the real object 110 and the virtual object 111 coincide with each other while viewing the mixed reality space image displayed on the GUI. As a result, the amount of deviation such that the virtual object 111 and the real object 110 coincide can be visually determined, adjusted, and input.

  Returning to FIG. 6, when the operator next determines that the virtual object 111 and the real object 110 match, the operation unit 1104 is operated to store the currently set deviation amount in the external storage device 1105. Enter instructions to record. Unless this instruction is detected, the CPU 1100 returns the process to step S601 via step S607 and prompts the user to input a new deviation amount. If this instruction is detected, the process proceeds to step S608 via step S607. In step S608, the currently set deviation amount d is stored in the external storage device 1105 as “delay time x”.

  The time lag can be acquired by the above processing.

  Next, using the acquired time lag, even if the position and orientation of the HMD 100 changes, a process of generating a mixed reality space image in which the real object 110 and the virtual object 111 match on the mixed reality space image. The process will be described with reference to FIG. In the figure, the same processing steps as those in FIG. 2 are denoted by the same step numbers, and the description thereof is omitted.

  Note that programs and data for causing the CPU 1101 to execute processing according to the flowchart of FIG. 10 are loaded from the external storage device 1105 to the RAM 1102, and the CPU 103 performs processing using the programs and data. Will be executed.

  In step S801, the “position and orientation in its own sensor coordinate system” measured by the magnetic receiver 105 and the “position and orientation in its own sensor coordinate system” measured by the magnetic receiver 106 are input to the RAM 1102 via the I / F 1107. However, this is recorded on the RAM 1102 up to at least the time lag x obtained in the above process, and the one x before the current time is acquired as used in step S205.

  As a result, in step S205, an image in the virtual space can be generated using the position and orientation before the current time t by the time lag x, and the reality on the mixed reality space image caused by the time lag is generated. Deviation between the object 110 and the virtual object 111 can be reduced.

  In this embodiment, the setting of the time t for performing the process in step S602 is input by the operator of the computer 103 using the operation unit 1104. However, the present invention is not limited to this, and is determined on the computer 103 side. You may make it do.

  For example, an average position / orientation is obtained using the position / orientation data group obtained by the processing according to the flowchart shown in FIG. 5, and the time when the obtained position / orientation is obtained is set as “time t”. May be.

  In the present embodiment, the magnetic receiver 106 is attached to the real object 110. However, when the position and orientation of the real object 110 are fixed, the position and orientation in the sensor coordinate system is not attached to the real object 110 in advance. It may be measured and stored as data in the external storage device 1105 and used.

  As described above, according to the present embodiment, one or a plurality of mixed reality images are presented to the user with the same shift time, and the user inputs the shift time, and the mixed reality with respect to the input shift time. By enabling the confirmation of the video, there is an effect that the time lag between the image and the sensor value can be easily measured.

[Second Embodiment]
In the first embodiment, as shown in FIG. 7, a plurality of generated mixed reality space images are displayed as a list, but in this embodiment, each generated mixed reality space image is displayed as a moving image. For example, in the GUI shown in FIG. 7, images of the mixed reality space of each frame (each time t) are displayed side by side. However, in this embodiment, each image is sequentially displayed at the same position to display each image. The frame mixed reality space image is displayed as a moving image.

  For this purpose, in the flowchart of FIG. 6, in step S605, the currently generated mixed reality space image is displayed in the area where the previous mixed reality space image was displayed.

  In this way, the measurer can comprehensively verify the deviation time from more mixed reality space images.

  In addition, since a still image may be easier to evaluate, a function of switching between a still image mode and a moving image mode is also effective.

  As described above, according to the present embodiment, according to the input shift time, a plurality of mixed reality space images are presented as still images or moving images, or the shift time is more easily determined by switching between them. I can do things.

[Third Embodiment]
In the present embodiment, the computer 103 generates a mixed reality space image and outputs the mixed reality space image to the display device 101 or the display unit 1106. When the deviation amount measurement mode) is set, processing according to the flowcharts of FIGS. 4, 5, and 6 described in the above embodiment is further executed.

  FIG. 9 is a flowchart of processing performed by the computer 103 according to the present embodiment. First, in the flowchart of FIG. 2, processing of Step S201 to Step S206 is performed to generate a mixed reality space image and output it to the display device 101 and the display unit 1106 (Step S1101).

  Next, the operator of the computer 103 determines whether or not the deviation amount measurement mode is set using the operation unit 1104 (step S1102). As a result, if the deviation amount measurement mode is not set, the process proceeds to step S1105, and then it is determined whether or not to end the present process (step S1105).

  The end of the process may be performed, for example, when an operator of the computer 103 inputs an instruction to that effect using the operation unit 1104, or when the CPU 1100 detects that a predetermined condition is satisfied. In addition, this process may be terminated by the control of the CPU 1100.

  In any case, if the present process is not terminated, the process returns to step S1101 to perform the subsequent processes.

  On the other hand, if the shift amount measurement mode is set in the determination process in step S1102, the process proceeds to step S1103, and the processes according to the flowcharts of FIGS. 4, 5, and 6 are performed (step S1103). Then, the deviation amount recorded in the external storage device 1105 in step S1103 is set to be used in step S1101 (step S1106), and the process returns to step S1101.

  As described above, according to the present embodiment, it is possible to easily and quickly perform correction when the time lag changes due to some external factor.

[Other Embodiments]
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or CPU) of the system or apparatus. Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a figure which shows the external appearance of the system which concerns on the 1st Embodiment of this invention which performs the process which provides the observer (user) with the mixed reality space which superimposed the virtual space on the real space. 4 is a flowchart of a series of processes for generating an image of a mixed reality space and outputting it to the HMD 100. It is a figure explaining the shift | offset | difference of the real object 110 and the virtual object 111 which generate | occur | produces because time lag arises. 10 is a flowchart of processing for acquiring an image of a real space imaged by a camera provided in the HMD and storing the acquired image in an external storage device 1105 together with an acquisition time. Processing for acquiring the position and orientation of the HMD 100 measured by the magnetic receiver 105 included in the HMD 100 and the position and orientation of the real object 110 measured by the magnetic receiver 106, and storing the acquired position and orientation together with the acquisition time in the external storage device 1105 It is a flowchart of. It is a flowchart of the process which calculates | requires a time lag. It is a figure which shows the example of a display of the GUI which has a user interface for inputting deviation | shift amount, and has the area | region for carrying out a list display of the produced | generated multiple (4 pieces in the same figure) mixed reality space. 10 is a flowchart of processing for generating an image of a mixed reality space in which a real object 110 and a virtual object 111 are matched on an image in a mixed reality space even if the position and orientation of the HMD 100 change using an acquired time lag. . It is a flowchart of the process which the computer 103 which concerns on the 3rd Embodiment of this invention performs. 2 is a diagram illustrating a hardware configuration of a computer 103. FIG.

Claims (5)

A first storage step of storing a real image included in a moving image acquired from an imaging device that captures a moving image of a real space and an acquisition time of the real image in association with each other;
A second storing step of storing the position and orientation data of the imaging device and the acquisition time of the position and orientation data in association with each other;
A deviation amount adjusting step of adjusting data d indicating the deviation amount in accordance with an instruction from the user;
A live-action image acquisition step of acquiring a real-image image corresponding to the acquisition time t using the real-action image stored in the first storage step;
Using the position and orientation data stored in the second storage step, position and orientation data corresponding to the acquisition time (td) is acquired, and the position and orientation data corresponding to the acquisition time (td) is obtained. A virtual image generation step of generating a virtual image using,
Mixed reality space image generation that synthesizes the actual image acquired in the actual image acquisition step and the virtual image generated in the virtual image generation step to generate an image of the mixed reality space corresponding to the acquisition time t Process,
A display step of displaying a list of a plurality of mixed reality space images corresponding to different acquisition times t generated using the real image acquisition step, the virtual image generation step, and the mixed reality space image generation step;
An image processing method comprising: a recording step of recording the data d in response to an instruction from a user.   2. The image according to claim 1, wherein in the real image acquisition step, a real image corresponding to the acquisition time t is generated by interpolation using a plurality of real image images stored in the first storage step. Processing method.   In the virtual image generation step, position and orientation data corresponding to the acquisition time (td) is generated by interpolation using the plurality of position and orientation data stored in the second storage step. The image processing method according to claim 1. First storage means for storing a real image included in a moving image acquired from an imaging device that captures a moving image of a real space and an acquisition time of the real image in association with each other;
Second storage means for storing the position and orientation data of the imaging apparatus and the acquisition time of the position and orientation data in association with each other;
A deviation amount adjusting step of adjusting data d indicating the deviation amount in accordance with an instruction from the user;
Using the live-action image stored by the first storage means, the live-action image acquisition means for acquiring the real-action image corresponding to the acquisition time t;
Using the position and orientation data stored by the second storage means, the position and orientation data corresponding to the acquisition time (td) is acquired, and the position and orientation data corresponding to the acquisition time (td) is used. Virtual image generation means for generating a virtual image,
A mixed reality space image generating unit that combines the real image acquired by the actual image acquisition unit and the virtual image generated by the virtual image generation unit to generate an image of the mixed reality space corresponding to the acquisition time t; ,
Display means for displaying a list of a plurality of mixed reality space images corresponding to different acquisition times t, generated using the real image acquisition means, the virtual image generation means, and the mixed reality space image generation means;
An image processing apparatus comprising: a recording unit that records the data d in response to an instruction from a user. Computer program for executing the image processing method according to any one of claims 1 to 3 on a computer.

Images