JP2006252468A - Image processing method and image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique capable of disposing a virtual object providing information to an observer in a suitable position and attitude observed from a point of view of the observer. <P>SOLUTION: The image processing method comprises the steps of: converting the desired position and attitude in a coordinates system formed by an axis directed from the original point to a position of an indicating tool and two axes intersected with this axis at the original point when a position of the point of view is used as an original point to the position attitude in a world coordinate system (S205); disposing an information providing virtual object in the converted position attitude (S206); generating an image obtained by viewing a virtual space including the disposed virtual object from the point of view (S207); and outputting the generated image to the outside (S208). <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for arranging a virtual object in a virtual space.

  Conventionally, there is a virtual reality (VR) presentation device. The virtual reality presentation device is, for example, a video display device such as a head mounted display (HMD) and a position and orientation detection means (for example, a position and orientation) for detecting the position and orientation of the observer's viewpoint Sensor) and CG video generation means.

  As the position / orientation detection means, a magnetic position / orientation sensor or the like is used, and the position / orientation value of the observer's head is detected by attaching this to an HMD worn by the observer. A magnetic position / orientation sensor detects the relative position / orientation between a magnetic generator (transmitter) and a magnetic sensor (receiver), such as the FASTRAK product from the US company Polhemus. It is done. This is a device that detects the three-dimensional position (X, Y, Z) and posture (Pitch, Yaw, Roll) of a sensor in real time in a specific region.

  The CG video generation means places the CG modeled three-dimensionally in a virtual space of the same scale as the real space, and renders the virtual space from the observer's line-of-sight position / posture detected by the position / posture detection means.

  By displaying the CG video generated in this way on a video display device (such as an HMD), the observer can feel as if they are immersed in a virtual CG space.

Further, a position / orientation sensor as described above can be newly provided and used to input the position and orientation of the CG object in the virtual space. For example, by moving / rotating the position / orientation sensor to move / rotate the CG object in the virtual space, or by attaching the position / orientation sensor to the hand and using it for operation input of the virtual object, It has also been conventionally performed to realize an interaction between a human and a human action.
Japanese Patent Laid-Open No. 11-136706

In conventional systems,
For example, when an information display object such as a menu or an information bulletin board is arranged and displayed in a virtual three-dimensional space, the following methods are used, each of which has a problem.

  1) As shown in FIG. 3, the information display object 302 is arranged at a fixed position in a coordinate system based on the position and orientation of the HMD 108. No matter how the head (HMD 108) is moved or rotated, the appearance of the information display object displayed on the screen of the HMD 108 does not change. Since the information object is arranged in the virtual space so that it is always visible on a part of the screen, the observer can always confirm the information.

  In this case, there is a problem that a part of the visual field is always blocked by the information display object so that the information object can always be seen on a part of the screen.

2) An information display object is placed at a specific position and orientation in the virtual three-dimensional space. As shown in FIG. 4, the information display object 302 is fixed at a specific position in the world coordinate system. In this case, unlike the case of 1), since the information display object can be confirmed for the first time when facing the place where the information display object is arranged, the observer is not always obstructed by the information display object.

  However, there is a problem that the observer needs to move to a specific position in order to view the information, and there is a problem that the information presentation location (arrangement position of the information display object) must be searched in the virtual three-dimensional space. . In addition, when an attempt is made to move the CG object to be observed to the position where the information display object is arranged, there is a problem in that both displays overlap and it is difficult to observe.

3) The information display object is arranged at a relative position and orientation from the object to be observed. As shown in FIG. 5, the information display object 302 is arranged at a specific position in the coordinate system based on the position and orientation of the CG object. This is convenient because it is easy to observe the observation target object and the information object at the same time, and the information display object follows if the observation target object is moved to another position.

  However, on the other hand, since the information display object 302 is always near the observation target CG object 303, there is a problem that it is easy to obstruct observation, and it is impossible to meet a request to view information only when necessary. Further, if the observation target CG object 303 is rotated, the information display object 302 rotates around the observation target CG object 303, so that the information display object enters the field of view unexpectedly or the information display object moves. There was a problem that information could be difficult to observe.

4) An information display object is displayed in a specific relative azimuth / distance based on the observer's head position. Rather than being fixed at a specific position in the coordinate system in the order of the position and orientation of the HMD, it is arranged at a specific orientation / distance seen from the position of the observer's head, regardless of the posture of the observer's head. That's it. As shown in FIG. 6, when the HMD 108 is changed from the position and orientation shown in the upper part of the figure to the position and orientation shown in the lower part of the figure, the information display object 302 changes as shown. In the case of 1), since the information display object 302 is fixed to the HMD coordinate system, the position of the information display object changes when the observer rotates the head. In this case, the information display object is displayed even if the attitude of the HMD changes. It does not affect the position of the object, but only changes in the position of the HMD. The advantage in this case is that the information display object is associated with the position of the head of the observer, so that the position of the information display object can be easily found compared to the method 2). It is possible to easily find and check the information display object by facing the screen. Further, since it is unrelated to the observation target object, the information display object does not move when the observation target object is rotated as in 3).

  However, even in this case, since the direction in which the information display object is displayed is fixed, there is a problem in that when the observation target object moves in the direction in which the information display object is placed, the two overlap.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for arranging a virtual object that provides information to an observer in a suitable position and orientation when viewed from the viewpoint of the observer. And

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, a first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held and operated by the observer;
When the position of the viewpoint is the origin, a desired position and orientation in a coordinate system composed of an axis from the origin to the position of the pointing tool and two axes that intersect the axis at the origin are represented as world coordinates. A conversion step for converting the position and orientation in the system;
An arrangement step of arranging a virtual object that provides information to the observer in a position and orientation in the world coordinate system obtained in the conversion step;
A generation step of generating an image that is visible when the virtual space including the virtual object arranged in the arrangement step is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
An output step of outputting the image generated in the generation step to the outside.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, first acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool that is held and operated by the observer;
When the position of the viewpoint is the origin, a desired position and orientation in a coordinate system composed of an axis from the origin to the position of the pointing tool and two axes that intersect the axis at the origin are represented as world coordinates. Conversion means for converting the position and orientation in the system;
Arranging means for arranging a virtual object that provides information to the observer at a position and orientation in the world coordinate system obtained by the converting means;
Generating means for generating an image that is visible when the virtual space including the virtual object placed by the placing means is viewed from a viewpoint having the position and orientation obtained by the first obtaining means;
Output means for outputting the image generated by the generating means to the outside.

  According to the configuration of the present invention, a virtual object that provides information to an observer can be arranged in a suitable position and orientation as viewed from the observer's viewpoint.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
<System configuration>
FIG. 1 is a diagram illustrating a hardware configuration of a system according to the present embodiment. As shown in the figure, the system according to this embodiment includes a CPU 101, an HMD 108, position and orientation sensors 104 and 105, a sensor controller (CTL) 103, and memories 106 and 107.

  The memory 106 stores a program for causing the CPU 101 to execute various processes described below, and the memory 107 stores data used when the CPU 101 executes various processes. Each information stored in the memories 106 and 107 will be described later.

  As is well known, the HMD 108 has a display screen attached so that an image is displayed in front of the eye when an observer wears it on the head. In this embodiment, an optical see-through type is used. Therefore, this display screen allows the real space to be seen through, and can display an image of the virtual space generated by the processing described below. Therefore, the observer can observe the real space with the image of the virtual space superimposed on the display screen. Since the optical see-through type HMD is well known, further explanation is omitted.

  A position / orientation sensor 104 is attached to the HMD 108. The position and orientation sensor 104 is assumed to be a magnetic sensor in this embodiment. Therefore, a magnetic field generation source is installed at a predetermined position in the real space. A magnetic field is generated from the generation source, and the position / orientation sensor 104 detects a change in magnetism corresponding to its own position / orientation. A signal indicating the detected result is output to the sensor controller 103. Based on the received signal, the sensor controller 103 “position and orientation in the sensor coordinate system (coordinate system in which the origin position is the origin and the three axes orthogonal to each other at the origin are the x, y, and z axes, respectively). Data indicating the position and orientation of the sensor 104 ”is generated and output to the memory 107.

  The position / orientation sensor 105 is the same as the position / orientation sensor 104 and detects a change in magnetism corresponding to its own position and orientation. A signal indicating the detected result is output to the sensor controller 103. The sensor controller 103 generates “data indicating the position and orientation of the position and orientation sensor 105 in the sensor coordinate system” based on the received signal, and outputs the data to the memory 107.

  Here, the position / orientation sensor 105 is used as an indicator that can be held by the observer and can be freely changed in position and orientation. In addition, a virtual object (operation target virtual object) is arranged on the position and orientation sensor 105. Therefore, when the observer changes the position and orientation of the position / orientation sensor 105, the position and orientation of the operation target virtual object are also changed in conjunction therewith.

<About virtual space image generation processing>
Next, a series of processing until a virtual space image is generated by the system having the above configuration and output to the HMD 108 will be described.

  In this embodiment, in addition to the operation target virtual object, a virtual object (information providing virtual object) in which various types of information provided to the observer are described is arranged in the virtual space. Hereinafter, processing for controlling the arrangement position and orientation of the information providing virtual object will be described in detail. The information providing virtual object is, for example, a menu on which various menus are described on a two-dimensional plane, a system status is displayed as appropriate, or information on its shape itself to the observer. Although it is a virtual object to be presented, the information provision form is not particularly limited. In addition, other virtual objects may be arranged in the virtual space.

  FIG. 2 is a flowchart of processing for generating a virtual space image for one frame and outputting it to the HMD 108. Note that the CPU 101 can execute processing according to the flowchart of FIG. 5 by appropriately executing each program stored in the memory 106 using each data stored in the memory 107 as appropriate. it can.

  First, since the position / orientation sensor 104 attached to the HMD 108 detects a change in magnetism according to its own position and orientation, it outputs a signal indicating the detected result to the sensor controller 103. Based on this signal, the sensor controller 103 generates “data indicating the position and orientation of the position and orientation sensor 104 in the sensor coordinate system” and inputs the data to the memory 107.

  Upon receiving this data, the CPU 101 executes the position / orientation detection program 111 and performs processing for obtaining the position / orientation of the viewpoint of the observer (step S201).

  Here, the “viewpoint” is the position where the apex of the frustum representing the visual volume of the virtual camera that draws the CG is placed (either the right eye or the left eye, or both), and the HMD This is a position that is placed in front of the display surface by a focal length and is expected to have the viewer's eyes located at that position when worn. The position and orientation relationship between the viewpoint and the position and orientation sensor 104 is measured in advance, and the measurement result is stored in the memory 107 as data (bias data). When “data indicating the position and orientation of the position and orientation sensor 104 in the sensor coordinate system” is received from the sensor controller 103, the “position and orientation of the viewpoint in the sensor coordinate system” can be obtained by adding the bias data to this data. it can.

  Furthermore, the world coordinate system (a coordinate system with one point in the real space as the origin and three axes orthogonal to each other at the origin, the x axis, the y axis, and the z axis, is shared between the virtual space and the real space. If the position and orientation of the magnetic field generation source is measured in advance and stored in the memory 107 as data (second bias data), data indicating “the position and orientation of the viewpoint in the sensor coordinate system” is stored. By adding the second bias data to “the position and orientation of the viewpoint in the world coordinate system”, it can be obtained.

  Note that the definition of the “viewpoint” and the method for obtaining the position and orientation of the viewpoint in the world coordinate system are not limited thereto.

  Then, the obtained data indicating the “position and orientation of the viewpoint in the world coordinate system” is stored in the memory 107 as data 121.

  On the other hand, the position / orientation sensor 105 detects a change in magnetism according to its own position / orientation, and therefore outputs a signal indicating the detected result to the sensor controller 103. Based on this signal, the sensor controller 103 generates “data indicating the position and orientation of the position and orientation sensor 105 in the sensor coordinate system” and inputs the data to the memory 107.

  Upon receiving this data, the CPU 101 executes the position / orientation detection program 111 and adds the second bias data to this data to obtain “data indicating the position / orientation of the position / orientation sensor 105 in the world coordinate system”. The data 122 is stored in the memory 107 (step S202). Next, the CG generation program 112 is executed to read the drawing data of the operation target virtual object from the CG data 123 to generate the operation target virtual object, which is arranged at the position and orientation indicated by the data 122 (step S203).

  Here, the CG data 123 is composed of drawing data for each virtual object. For example, when the virtual object is composed of polygons, the drawing data is a virtual object (operation target virtual object, information provision virtual). (Excluding the object)) arrangement position and orientation data, coordinate data of each vertex constituting the polygon constituting the virtual object, polygon normal vector data, polygon color data, and when performing texture mapping on this virtual object, It consists of texture data.

  Thereby, the operation target virtual object can be arranged at the position and orientation in the world coordinate system of the position and orientation sensor 105.

  Next, the CPU 101 executes the observation coordinate calculation program 113 and obtains a conversion matrix for performing coordinate value conversion between the observation coordinate system and the world coordinate system described below (step S204). The observation coordinate system is an axis (first axis) from the origin to the position of the virtual object to be operated when the viewpoint position is the origin, and two axes (first axes) orthogonal to the first axis at the origin. 2 axis, third axis), for example, the first axis can be the z axis, and the second and third axes can be the x axis and the y axis, respectively.

  Here, the observation coordinate system will be described in more detail.

  FIG. 7A shows a viewpoint coordinate system, and FIG. 7B shows an observation coordinate system. As shown in FIG. 7A, the viewpoint coordinate system is a coordinate system in which the position of the viewpoint is the origin, and three axes orthogonal to each other at the origin are the x axis, the y axis, and the z axis. In the figure, the axis in the line-of-sight direction is the z axis, the axis from the origin to the overhead direction is the y axis, and the axis orthogonal to the z axis and the y axis at the viewpoint position is the x axis.

  On the other hand, as shown in FIG. 7B, the observation coordinate system has a first axis that is directed from the origin to the position of the virtual object to be operated, and the origin is orthogonal to the first axis when the position of the viewpoint is the origin. This is a coordinate system composed of a second axis and a third axis. In the figure, the first axis is the z-axis. Various determination methods can be considered for the x-axis and the y-axis. For example, the x-axis is orthogonal to the z-axis and is parallel to the xz plane (horizontal plane) in the world coordinate system, and the y-axis is the x-axis. , An axis that is orthogonal to the z-axis at the viewpoint position and that faces in a direction that indicates upward in the world coordinate system.

  Note that there are other possible settings for the x-axis of the observation coordinate system. For example, when the y-axis of the observation coordinate system is viewed on the zy plane in the viewpoint coordinate system, it is set to match the y-axis of the viewpoint coordinate system. In this case, it is possible to make the upward direction of the observation coordinate system (the Y-axis direction) always coincide with the upward direction for the observer. It is also possible to make the x axis in the observation coordinate system parallel to the horizontal plane (XZ plane) in the viewpoint coordinate system.

  If the observation coordinate system as described above is defined, a transformation matrix between the world coordinate system and the observation coordinate system can be obtained by a known technique. That is, the obtained conversion matrix can convert the coordinate value in the world coordinate system into the coordinate value in the observation coordinate system, and conversely, the coordinate value in the observation coordinate system can be converted into the coordinate value in the world coordinate system. The obtained transformation matrix data is stored in the memory 107 as data 124.

Returning to FIG. 2, next, the CPU 101 executes the information display position / orientation setting program 114 to obtain the arrangement position / orientation of the information providing virtual object in the world coordinate system (step S205). In the present embodiment, a desired arrangement position / posture of the information providing virtual object in the observation coordinate system is determined before the processing according to the flowchart, and data indicating the determined position / posture is stored in the memory 107 as data 125. ing. Accordingly, the arrangement position / posture Q of the information providing virtual object in the world coordinate system is expressed by the following formula: Q = M · P, where the position / posture indicated by the data 125 is a vector P and the transformation matrix indicated by the data 124 is M.
Can be obtained by calculating. The obtained data indicating the “location position and orientation of the information providing virtual object in the world coordinate system” is stored in the memory 107 as data 126.

  In this embodiment, the arrangement position and orientation of the information providing virtual object in the observation coordinate system are determined in advance. However, the present invention is not limited to this, and each time a virtual space image of each frame is generated, a suitable position is provided. The position and orientation may be calculated. For example, when the information provision virtual object is a virtual object in which a character string is described, the arrangement posture of the information provision virtual object may be appropriately changed so that the description surface faces the viewer.

  In addition, the information providing virtual object may be arranged in an attitude that always stands upright in the world coordinate system. In other words, the information providing virtual object is arranged so that the upward direction of the information providing virtual object substantially coincides with the upward direction in the world coordinate system.

  In any case, after the arrangement position of the information provision virtual object in the world coordinate system is determined, the information provision virtual object may be further rotated so as to have a desired posture.

  Returning to FIG. 2, next, the CPU 101 executes the CG generation program 112, reads the drawing data of the information providing virtual object from the CG data 123 to generate the information providing virtual object, and this is the position indicated by the data 126. It arrange | positions to a attitude | position (step S206).

  Next, the CPU 101 executes the CG generation program 112 to display a virtual space in which a virtual object group including the operation target virtual object and the information providing virtual object is arranged as “a viewpoint position in the world coordinate system”. An image (a virtual space image) seen when viewed from a viewpoint having “attitude” is generated (step S207), and the generated image is sent to the HMD 108 (step S208).

  With the above processing, a virtual space image for one frame can be generated and sent to the HMD 108. By performing this processing for each frame, the display screen of the HMD 108 has a virtual space of each continuous frame. An image is displayed.

  If the HMD 108 can display in stereo, the position and orientation of each of the right eye and left eye are acquired in step S201, the observation coordinate system is obtained with either the right eye or the left eye as the viewpoint in step S204, and in step S207. The image of the virtual space seen from each of the right eye and the left eye is generated.

  As described above, according to the present embodiment, the arrangement position of the information providing virtual object does not change even if the viewpoint or the operation target virtual object is rotated.

  Further, when observing a virtual object other than the operation target virtual object, the information providing virtual object can be arranged outside the observer's field of view.

  In addition, since the positional relationship among the viewpoint, the operation target virtual object, and the information providing virtual object is constant, it is easy to find an information providing virtual object that is not in the observer's field of view.

[Second Embodiment]
In the first embodiment, the position and orientation are measured using all magnetic sensors as the sensors. However, other sensors may be used, and other sensors such as ultrasonic sensors and optical sensors may be used. May be.

  The means for obtaining the position and orientation of the viewpoint is not limited to using a sensor. For example, a new camera is added to the HMD 108 and a marker is arranged in the real space. Therefore, a method of acquiring the position and orientation of the HMD 108 using the coordinate position on the captured image may be used. Therefore, as long as the position and orientation of the viewpoint can be obtained, the use of the sensor is not limited.

  In the first embodiment, the optical see-through type is used for the HMD 108, but a video see-through type may be used.

  In the first embodiment, only one position / orientation sensor 105 is provided. However, a plurality of position / orientation sensors 105 may be provided and an operation target virtual object may be arranged in each position / orientation sensor 105.

  Further, in the first embodiment, the operation target virtual object is arranged in the position / orientation sensor 105, but this arrangement is not essential.

[Other Embodiments]
An object of the present invention is to supply a recording medium (or storage medium) that records software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and the computer of the system or apparatus (or CPU or MPU). Needless to say, this can also be achieved by reading and executing the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a figure which shows the hardware constitutions of the system which concerns on the 1st Embodiment of this invention. 10 is a flowchart of processing for generating an image of a virtual space for one frame and outputting the image to the HMD. It is a figure which shows the example of the conventional arrangement | positioning of an information display object. It is a figure which shows the example of the conventional arrangement | positioning of an information display object. It is a figure which shows the example of the conventional arrangement | positioning of an information display object. It is a figure which shows the example of the conventional arrangement | positioning of an information display object. (A) is a view coordinate system, (b) is a diagram showing an observation coordinate system.

Claims (6)

A first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held and operated by the observer;
When the position of the viewpoint is the origin, a desired position and orientation in a coordinate system composed of an axis from the origin to the position of the pointing tool and two axes that intersect the axis at the origin are represented as world coordinates. A conversion step for converting the position and orientation in the system;
An arrangement step of arranging a virtual object that provides information to the observer in a position and orientation in the world coordinate system obtained in the conversion step;
A generation step of generating an image that is visible when the virtual space including the virtual object arranged in the arrangement step is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
An image processing method comprising: an output step of outputting the image generated in the generation step to the outside.   The image processing method according to claim 1, further comprising a step of arranging a virtual object to be operated by the observer at the position and orientation acquired in the second acquisition step. And a third acquisition step of acquiring an image of the real space visible from the viewpoint,
3. The image processing method according to claim 1, wherein in the output step, the image generated in the generation step is superimposed on the image acquired in the third acquisition step and then output to the outside. First acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool that is held and operated by the observer;
When the position of the viewpoint is the origin, a desired position and orientation in a coordinate system composed of an axis from the origin to the position of the pointing tool and two axes that intersect the axis at the origin are represented as world coordinates. Conversion means for converting the position and orientation in the system;
Arranging means for arranging a virtual object that provides information to the observer at a position and orientation in the world coordinate system obtained by the converting means;
Generating means for generating an image that is visible when the virtual space including the virtual object placed by the placing means is viewed from a viewpoint having the position and orientation obtained by the first obtaining means;
An image processing apparatus comprising: output means for outputting the image generated by the generating means to the outside.   A program for causing a computer to execute the image processing method according to any one of claims 1 to 3.   A computer-readable storage medium storing the program according to claim 5.

Images