JP2006268351A - Image processing method, and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for integrating a part or all of interfaces into a single interface for manipulating the position and posture of virtual objects when there are a plurality of interfaces for manipulating the position and posture of the virtual objects and the virtual objects are arranged in the position and posture of the interfaces. <P>SOLUTION: A virtual object is arranged in the position and posture of each of a first interface and a second interface with a structure allowing integration with the first interface (S504), an image obtained by viewing a virtual space with the virtual objects arranged therein from a viewer angle is generated (S507), and the generated image is outputted to the outside (S509). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for generating an image of a virtual space.

  A virtual reality (VR) system is a system that makes a virtual space feel as if it is real by presenting a computer with a three-dimensional CG created by a computer. In recent years, technology that presents users with information that does not exist in the real world by synthesizing 3D CG with real-world images has been developed, such as AR (Augumented Reality) systems and MR ( It is called a mixed reality system.

  In the MR system, it is possible to superimpose 3D CG on a real object. For example, in the system disclosed in Patent Document 1, a virtual object is superimposed on a real object so that the user can freely manipulate the virtual object.

  Hereinafter, a real object serving as an auxiliary tool for operating a virtual object is referred to as a “virtual object movement interface”.

In Patent Document 1, the movement of a virtual object from a virtual object movement interface to another virtual object movement interface is realized.
JP 2003-242527 A

  However, in conventional systems, only a virtual object is transferred from one virtual object movement interface to another virtual object movement interface, and both are combined and do not change to one virtual object movement interface. When both have separate position and orientation recognition methods, it is considered that it is very effective to integrate the two.

  The present invention has been made in view of the above problems, and there are a plurality of interfaces for operating the position and orientation of a virtual object, and when a virtual object is arranged at the position and orientation of the interface, some or all of the interfaces are provided. An object of the present invention is to provide a technique for integrating an interface into an interface for operating the position and orientation of a virtual object as one interface.

In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement. That is, a first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of each of the one or more virtual object operation interfaces and the first virtual object operation interface having a structure that can be integrated with the one or more virtual object operation interfaces;
An arrangement step of arranging a virtual object at each position and orientation acquired in the second acquisition step;
A generation step of generating an image that is visible when the virtual space in which the virtual object is arranged in the arrangement step is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
An output step of outputting the image generated in the generation step to the outside.

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement. That is, first acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of each of the one or more virtual object operation interfaces and the first virtual object operation interface having a structure that can be integrated with the one or more virtual object operation interfaces;
Arrangement means for arranging a virtual object at each position and orientation acquired by the second acquisition means;
Generating means for generating an image that is visible when the virtual space in which the virtual object is arranged by the arranging means is viewed from a viewpoint having the position and orientation acquired by the first acquiring means;
Output means for outputting the image generated by the generating means to the outside.

  According to the configuration of the present invention, when there are a plurality of interfaces for manipulating the position and orientation of the virtual object, and a virtual object is arranged at the position and orientation of the interface, a part of or all of the interfaces are integrated into one interface. As an interface for operating the position and orientation of the virtual object.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing a functional configuration of a system according to the present embodiment. As shown in the figure, the system according to this embodiment includes a computer 101, a video see-through HMD (hereinafter simply referred to as HMD) 132, a transmitter 142, a magnetic virtual object movement interface (hereinafter referred to as a first interface). 140), a marker type virtual object movement interface (hereinafter referred to as a second interface) 141.

  First, the HMD 132 will be described. As shown in the figure, the HMD 132 includes an image input unit 135, an image display unit 136, a camera 133, an image output unit 134, and an observer viewpoint position / orientation measurement unit 105.

  The image input unit 135 receives the image signal sent from the computer 101 and outputs it to the image display unit 136. The image display unit 136 is attached to the HMD 132 so as to be positioned in front of the observer wearing the HMD 132 on the head, and displays an image according to the image signal received from the image input unit 135. Thereby, the observer can observe the image according to the image signal sent from the computer 101 in front of the eyes.

  The camera 133 captures a moving image in the real space, and the captured image signal of each frame is sequentially transmitted to the computer 101 by the image output unit 134.

  The observer viewpoint position and orientation measurement unit 105 is a magnetic position and orientation sensor. In the real space, a transmitter 142 as a magnetic field generation source is disposed at a predetermined position, and generates a magnetic field. Therefore, the observer viewpoint position / orientation measurement unit 105 detects a change in magnetism according to its own position and orientation, and sends the detection result to the computer 101 as a signal. This detection result is obtained by the observer viewpoint position / orientation measurement unit 105 in the sensor coordinate system (a coordinate system in which the position of the transmitter 142 is the origin and three axes orthogonal to each other are the x axis, the y axis, and the z axis). It shows the position and orientation.

  The first interface 140 is one that can be operated by the observer while holding it in his / her hand, and, similar to the observer viewpoint position / orientation measurement unit 105, according to his / her position / orientation. It functions as a position and orientation sensor that detects a change in magnetism and sends a signal indicating the detection result to the computer 101. This detection result indicates the position and orientation of the first interface 140 in the sensor coordinate system.

  The second interface 141 is one that can be operated by the observer while holding it in his / her hand, and is used for measuring the position and orientation of the second interface 141 on the surface thereof. Markers are marked. Further, a structure for integrating with the first interface 140 is added to the second interface 141. Details of the first interface 140 and the second interface 141 will be described later.

  Next, the computer 101 will be described. As shown in FIG. 1, the computer 101 includes an image output unit 104, an image composition unit 103, a CG generation unit 106, an image input unit 102, and a virtual object movement interface position / orientation measurement unit 109.

  The image input unit 102 receives the image signal of each frame sequentially sent from the image output unit 134 of the HMD 132 as data, and outputs this to the image composition unit 103.

  The virtual object movement interface position / orientation measuring unit 109 obtains the signal received from the first interface 140 as “data indicating the position / orientation (viewpoint position / orientation) of the first interface 140 in the sensor coordinate system”.

  In addition, the virtual object movement interface position / orientation measurement unit 109 detects this marker when the marker described in the second interface 141 is reflected in the real space image input from the image input unit 102. By doing so, the position and orientation of the second interface 141 in the sensor coordinate system is obtained.

  Then, the virtual object movement interface position / orientation measurement unit 109 outputs the obtained position / orientation of the first interface 140 in the sensor coordinate system and the position / orientation of the second interface 141 in the sensor coordinate system to the CG generation unit 106.

  Further, since the virtual object movement interface position / orientation measuring unit 109 receives “data indicating the position and orientation of the observer viewpoint position / orientation measuring unit 105 in the sensor coordinate system” from the observer viewpoint position / orientation measuring unit 105, this data In addition, a bias value indicating the position and orientation relationship between the camera 133 and the observer viewpoint position and orientation measurement unit 105 that has been measured in advance is added to obtain the position and orientation of the camera 133 (viewpoint position and orientation) in the sensor coordinate system. .

  First, the CG generation unit 106 arranges the first virtual object at the “position and orientation of the first interface 140” received from the virtual object movement interface position and orientation measurement unit 109, and from the virtual object movement interface position and orientation measurement unit 109. The second virtual object is placed in the received “position and orientation of the second interface 141”.

  The virtual space in which each virtual object is arranged is seen when viewed from the viewpoint of the position and orientation indicated by the “data indicating the position and orientation of the viewpoint in the sensor coordinate system” obtained from the virtual object movement interface position and orientation measurement unit 109. An image (virtual space image) is generated. Note that processing for generating an image that is visible when a virtual space in which a virtual object is arranged at a predetermined position and orientation is viewed from a viewpoint having the predetermined position and orientation is a well-known technique, and thus description thereof is omitted. .

  The image composition unit 103 superimposes the virtual space image generated by the CG generation unit 106 on the real space obtained from the image input unit 102 to generate an image of the mixed reality space. Then, the image output unit 134 outputs the mixed reality space image generated by the image composition unit 103 to the image input unit 102 of the HMD 132. Since the image input unit 102 outputs the received mixed reality space image to the image display unit 136, the image display unit 136 displays an image of the mixed reality space according to the position and orientation of the viewpoint of the observer.

  Thereby, the observer can see the image of the mixed reality space according to the position and orientation of his / her viewpoint in front of the eyes. In addition, the mixed reality space image is obtained by arranging the first virtual object at the position and orientation of the first interface 140 and arranging the second virtual object at the position and orientation of the second interface 141. .

  FIG. 9 is a block diagram showing a hardware configuration of the computer 101 having the above functional configuration.

  Reference numeral 901 denotes a CPU that controls the entire computer 101 using programs and data stored in the RAM 902 and the ROM 903 and executes each process described below performed by the computer 101.

  Reference numeral 902 denotes a RAM, an area for temporarily storing programs and data loaded from the external storage device 906, an area for temporarily storing data received from the outside via the I / Fs 907 and 908, and Various areas such as a work area used when the CPU 901 executes each process can be provided.

  Reference numeral 903 denotes a ROM that stores setting data, a boot program, and the like of the computer 101.

  An operation unit 904 includes a keyboard and a mouse, and various instructions can be input to the CPU 901 by an operator of the computer 101.

  A display unit 905 includes a CRT, a liquid crystal screen, and the like, and can display a processing result by the CPU 901 with an image, text, or the like.

  Reference numeral 906 denotes an external storage device that functions as a large-capacity information storage device typified by a hard disk drive device and the like for causing the CPU 901 to execute an OS (operating system) and each process described later performed by the computer 101. Are stored in the RAM 902 under the control of the CPU 901 and are processed by the CPU 901.

  Reference numeral 907 denotes an I / F that functions as an interface for connecting the HMD 132 to the computer 101, and the computer 101 can perform data communication with the HMD 132 via the I / F 907.

  Reference numeral 908 denotes an I / F that functions as an interface for connecting the first interface 140 to the computer 101, and the computer 101 performs data communication with the first interface 140 via the I / F 907. It can be carried out.

  Reference numeral 909 denotes a bus connecting the above-described units.

  FIG. 2 is a diagram illustrating a configuration of the first interface 140. As shown in the figure, the first interface 140 includes a position / orientation sensor 203 for detecting a change in magnetism according to its own position and orientation, and a main body covered with a rectangular parallelepiped slightly larger than that. Yes. FIG. 3 is a diagram illustrating a state in which the first virtual object 301 is disposed at the position and orientation of the position and orientation sensor 203. In the drawing, the center position of the first virtual object 301 is arranged at the position and orientation measured by the position and orientation sensor 203 (position and orientation of the first interface 140).

  FIG. 4 is a diagram illustrating an appearance of the second interface 141. As shown in the figure, the second interface 141 has a marker 403 that uniquely identifies the second interface 141 and is used to measure the position and orientation of the second interface 141. The computer 101 can detect the position and orientation in the sensor coordinate system of the second interface 141 by detecting this marker, and arranges the second virtual object at the obtained position and orientation. Note that the shape and size of the marker are not limited to those shown in the figure as long as they function as markers.

  In addition, the second interface 141 includes attachments 402a and 402b for integrating with the first interface 140, and the first interface 140 is arranged between the two attachments 402a and 402b. As shown in FIG. 6, the first interface 140 and the second interface 141 can be integrated. FIG. 6 is a diagram illustrating a state in which the first interface 140 and the second interface 141 are integrated. Therefore, when the second interface 141 is held in the hand and the position and posture thereof are changed, the position and posture of the first interface 140 integrated with the second interface 141 are also changed accordingly.

  In addition, since the first virtual object is arranged at the position and orientation of the first interface 140 and the second virtual object is arranged at the position and orientation of the second interface 141, the first interface 140 and the second interface As a result, the first virtual object and the second virtual object can be integrated, and as a result, the second interface 141 is held in the hand and the position and orientation of the second virtual object can be integrated. By changing it, not only the second virtual object but also the position and orientation of the first virtual object can be changed in conjunction with it.

  FIG. 8 is a diagram illustrating a state in which the first virtual object and the second virtual object are integrated, and 801 corresponds to the second virtual object. As shown in the figure, the first virtual object 301 and the second virtual object 801 have substantially the same center position. Therefore, as shown in FIG. 7, the observer places the first interface 140 between the attachments 402 a and 402 b, and the center position of the first virtual object 301 matches the center position of the second virtual object 801. In this way, the first interface 140 is moved along the direction indicated by the arrow in FIG. FIG. 7 is a diagram illustrating movement of the first interface 140 so that the center positions of the respective virtual objects substantially coincide with each other.

  Here, it is not easy to completely match the center position of the first virtual object 301 with the center position of the second virtual object 801. Therefore, when the center position of the first virtual object 301 approaches the center position of the second virtual object 801 within a predetermined distance, the center position of the first virtual object 301 is the center of the second virtual object 801. The first virtual object 301 is moved to a position and orientation that substantially matches the position. Thereby, the observer can substantially match the center position of the first virtual object 301 with the center position of the second virtual object 801 by a rough operation.

  Note that the “predetermined distance” is not particularly limited as to which part between the virtual objects is to be inserted, and how the position and orientation of one virtual object is changed when each approaches within a predetermined distance. There is no particular limitation on whether or not to do so, and it may be determined as appropriate according to the use.

  FIG. 5 is a flowchart of a series of processes for generating an image of the mixed reality space for one frame and outputting it to the image display unit 136 of the HMD 132. Note that programs and data for causing the CPU 901 to execute the processing according to the flowchart of FIG. 9 are stored in the external storage device 906, and these are loaded into the RAM 902 as appropriate under the control of the CPU 901, and the CPU 901 uses this processing. By executing the above, the computer 101 performs processing according to the flowchart of FIG.

  First, since a signal indicating the position and orientation of the observer viewpoint position / orientation measurement unit 105 in the sensor coordinate system is input to the computer 101 from the observer viewpoint position / orientation measurement unit 105, the CPU 901 acquires this as data in the RAM 902. Then, by adding the bias value to the acquired data, “data indicating the position and orientation of the observer's viewpoint in the sensor coordinate system” is acquired (step S501).

  Further, since each frame constituting the moving image in the real space is sequentially input from the camera 133, the CPU 901 sequentially acquires these in the RAM 902 (step S502). A part of the acquired frame may be stored in the external storage device 906.

  In addition, since signals indicating the position and orientation in the respective sensor coordinate systems are input from the first interface 140 and the second interface 141, the CPU 901 acquires these as data in the RAM 902 (step S503).

  In addition, since each process in step S501, S502, S503 is performed independently, it does not limit that the process in each step of step S501, S502, S503 is performed in this order.

  Next, the CPU 901 arranges the first virtual object at the position and orientation of the first interface 140, and arranges the second virtual object at the position and orientation of the second interface 141 (step S504).

  Next, the CPU 901 determines the distance between the position of the first interface 140 and the position of the second interface 141, that is, between the center position of the first virtual object and the center position of the second virtual object. Is calculated, and it is determined whether or not the calculated result is within a predetermined distance (step S505). As a result, if the distance is within the predetermined distance, the process proceeds to step S506, and the first virtual object is set in a predetermined position and orientation (in the case of FIG. 8, the center position of the first virtual object is the second virtual object). Rearrangement is performed at a position and orientation of the first virtual object that substantially matches the center position of the object (step S506).

  Then, after the rearrangement, or when the distance between the center position of the first virtual object and the center position of the second virtual object is equal to or greater than the predetermined distance, the process proceeds to step S507, and the position acquired in step S501. An image of a virtual space (a virtual space in which the first virtual object and the second virtual object are arranged) that can be seen from the viewpoint of the posture is generated (step S507). Then, the mixed virtual space image is generated on the RAM 902 by superimposing (combining) the generated virtual space image on the real space image acquired in step S502 (step S508). Then, the generated mixed reality space image is output to the image display unit 136 of the HMD 132 via the I / F 907 (step S509).

  Then, by repeating the process according to the flowchart of FIG. 5 a plurality of times, a mixed reality space image corresponding to the position and orientation of its own viewpoint is displayed as a moving image in front of the observer's eyes.

  In this embodiment, the position / orientation of the first interface 140 is obtained by a magnetic sensor, and the position / orientation of the second interface 141 is obtained by a marker. However, the acquisition method of the position / orientation of each interface is particularly limited. Not what you want.

  In the present embodiment, the second interface 141 has a structure (two attachments) that can be integrated with the first interface 140. However, one interface includes one or more other interfaces. If a structure that can be integrated with one or more is added, the structure is not limited to such a structure.

  In the present embodiment, the video see-through type is used for the HMD 132, but it goes without saying that an optical see-through type may be used.

  Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or CPU) of the system or apparatus. Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structure of the system which concerns on embodiment of this invention. 2 is a diagram illustrating a configuration of a first interface 140. FIG. FIG. 6 is a diagram illustrating a state when a first virtual object 301 is arranged at a position and orientation of a position and orientation sensor 203. FIG. 3 is a diagram illustrating an appearance of a second interface 141. It is a flowchart of a series of processes for generating an image of the mixed reality space for one frame and outputting it to the image display unit 136 of the HMD 132. It is a figure which shows a mode that the 1st interface 140 and the 2nd interface 141 were integrated. It is the figure shown about the movement of the 1st interface 140 in order to make the center position of each virtual object substantially correspond. It is a figure which shows a mode that the 1st virtual object and the 2nd virtual object were integrated. 2 is a block diagram illustrating a hardware configuration of a computer 101. FIG.

Claims (6)

A first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of each of the one or more virtual object operation interfaces and the first virtual object operation interface having a structure that can be integrated with the one or more virtual object operation interfaces;
An arrangement step of arranging a virtual object at each position and orientation acquired in the second acquisition step;
A generation step of generating an image that is visible when the virtual space in which the virtual object is arranged in the arrangement step is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
An image processing method comprising: an output step of outputting the image generated in the generation step to the outside. And a third acquisition step of acquiring an image of the real space visible from the viewpoint,
The image processing method according to claim 1, wherein in the output step, the image generated in the generation step is superimposed on the image acquired in the third acquisition step and output to the outside.   In the arrangement step, the second virtual object arranged at the position and orientation of the second virtual object operation interface among the one or more virtual object operation interfaces and the position and orientation of the first virtual object interface are arranged. 3. The image processing according to claim 1, wherein the second virtual object is moved to a predetermined position and orientation when the first virtual object is located within a predetermined distance. 4. Method. First acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of each of the one or more virtual object operation interfaces and the first virtual object operation interface having a structure that can be integrated with the one or more virtual object operation interfaces;
Arrangement means for arranging a virtual object at each position and orientation acquired by the second acquisition means;
Generating means for generating an image that is visible when the virtual space in which the virtual object is arranged by the arranging means is viewed from a viewpoint having the position and orientation acquired by the first acquiring means;
An image processing apparatus comprising: output means for outputting the image generated by the generating means to the outside.   A program for causing a computer to execute the image processing method according to any one of claims 1 to 3.   A computer-readable storage medium storing the program according to claim 5.

Images