JP3848092B2 - Image processing apparatus and method, and program - Google Patents

Description

[0001]
[Industrial application fields]
The present invention relates to an image processing apparatus, method, and program , and in particular, superimposes a real-space video and a video generated by three-dimensionally modeled computer graphics (hereinafter referred to as “CG”), In addition, the present invention relates to an image processing apparatus and method, and a program capable of displaying both images while aligning them.
[0002]
[Prior art]
Conventionally, images in real space and images generated by 3D modeled CG images are superimposed and displayed while aligning both images, as if they were drawn in CG in the real world. There is a mixed reality presentation device that can make it appear as if an object (virtual object) exists. This apparatus includes both a real video shooting means (for example, a video camera) for shooting a real video, and a CG video generation means for creating a CG video as seen from the position where the real video is shot. It comprises video display means (for example, a head mounted display (HMD) or a monitor) that can be displayed in a synthesized manner. Here, the video display means displays the CG video and the real video in the correct positional relationship even if the gaze position of the real video shooting means changes, and the real video shooting means displays the gaze position and the gaze direction. A line-of-sight position / orientation detection means (for example, a position / orientation sensor) is provided for detection.
[0003]
The CG image generation means places the CG image that has been three-dimensionally modeled in a virtual space having the same scale as the real space, and renders the CG image as observed from the line-of-sight position and the line-of-sight direction detected by the line-of-sight position / orientation detection means. . When the CG video generated in this way and the real video are superimposed, as a result, the target of the CG video is correctly placed in the real space regardless of the gaze position or gaze direction observed by the real video shooting means. It is possible to display an image as if it were displayed.
[0004]
Further, a further position / orientation detection device is attached to a place where a CG image is desired to appear, and the CG image is caused to appear at a position obtained therefrom. For example, by attaching a position / orientation detection device (position / orientation sensor) to the hand and generating a CG image at the position of the sensor, it is as if the object of the CG image is always on the hand, Even if it moves like this, it is possible to display an image as if a CG image is on the hand.
[0005]
A video camera as a real image photographing means for photographing an object to be photographed in a real space captures an image in the direction of the line of sight and captures the captured image data in a memory.
[0006]
For example, an HMD is used as a video display device that synthesizes and displays an actual video and a CG video. By using the HMD instead of a normal monitor and mounting the video camera in the direction of the line of sight of the HMD, the image in the direction that the observer is facing can be displayed on the HMD, and the observer can change the direction. Since the CG image can be displayed when facing the viewer, it is possible to enhance the immersive feeling of the observer.
[0007]
As the position / orientation detection means, a magnetic position / orientation sensor or the like is used. By attaching this to the video camera or the HMD to which the video camera is attached, the position / orientation value of the line of sight of the video camera is obtained. To detect. A magnetic position / orientation sensor detects the relative position and orientation between a magnetism generator (transmitter) and a magnetic sensor (receiver), such as the product FASTRAK from PolhemuS of the United States. It is done. This is a device that detects the three-dimensional position (X, Y, Z) and posture (rolling, pitching, yawing) of a sensor in real time in a specific region.
[0008]
With the above configuration, the observer can observe the world in which the real image and the CG image are superimposed through the HMD. When the observer looks around, the real image capturing device (video camera) provided in the HMD captures a real image, and the line-of-sight position / orientation detection means (position / posture sensor) provided in the HMD detects the position of the video camera. The direction is detected, and the CG image generation means generates a CG image viewed from the line-of-sight position / orientation in response to the detected direction, and displays the CG image superimposed on the actual image.
[0009]
It is also possible to provide a plurality of observers. Video cameras and display devices (such as HMDs) that shoot real space, and position and orientation sensors are prepared as many as the number of viewers, and real space and CG video are generated by shooting real space from the viewpoint of each viewer. These may be combined and displayed to each observer.
[0010]
[Problems to be solved by the invention]
However, the above-described conventional mixed reality presentation apparatus renders a CG image based on values obtained from a gaze position / orientation sensor for detecting a gaze position and a gaze direction (orientation), and there is an error in these values. If there is, the positions of all the generated CG images are shifted, and CG images viewed from positions and directions shifted by the error are generated.
[0011]
Further, even when the line-of-sight position / orientation sensor has a small error, if there is an error in the CG image position / orientation sensor installed to detect the location where the CG image is to appear, the CG intended to appear at that position. The position and direction of the image are shifted.
[0012]
The values obtained from such position and orientation sensors generally include more or less errors. For example, in a position and orientation sensor using a magnetic system, the region where the position and orientation can be detected is a spatial region centered on the magnetic transmitter. Even within that region, the detection error increases as the receiver (sensor) moves away from the magnetic transmitter. For this reason, the farther the observation position, that is, the position of the camera for taking a real image, is farther from the transmitter, the farther the position / orientation sensor is from the transmitter, resulting in more errors in the values of the gaze position / orientation. . When the position / orientation sensor is placed at a position where the CG image appears, even if the observer position is close to the transmitter position and the observation is performed from a range with little error, the position where the CG image appears is the transmitter position. As the distance from the position increases, the error of the position / orientation sensor increases, resulting in a shift in the position where the CG image appears.
[0013]
In the conventional mixed reality presentation device, even when the error of the position / orientation sensor becomes large due to the movement of the observer or the movement of the CG image position, the CG image is drawn. Even if the CG video display that is planned to be on the hand, for example, that is out of alignment, is displayed as it is even if it is an unnatural video that is displayed on a place that is not on the hand. There is a problem of end. In the case of a virtual reality system that does not require alignment with the real space (a system that displays only CG images without real images), there is a big problem because it is not so unnatural even if there is a slight error. However, the mixed reality system requires high-precision alignment with the real object, so even a small error that occurs in the effective measurement area of the position and orientation sensor makes the image unnatural. And may not be practical.
[0014]
Also, for example, if there is an error in the line-of-sight position, an error will occur in the alignment of all CG images, but if there is an error in the CG image position and orientation, only an error will occur in the drawing of the CG image. The phenomenon that occurs is different between when the image position / orientation sensor error increases and when the line-of-sight position / orientation sensor error increases, and there is no system that can take appropriate measures in each case.
[0015]
An object of the present invention is to provide an image processing apparatus and method, and a program capable of preventing unnatural video display.
[0016]
[Means for Solving the Problems]
In order to achieve the above object, the image processing apparatus according to claim 1, a real image photographing unit that acquires a real image at a predetermined line-of-sight position, a line-of-sight position detection unit that detects the predetermined line-of-sight position, An image processing apparatus comprising: a CG video generation unit that generates a CG video corresponding to the detected line-of-sight position; and a video display unit that displays the generated CG video on a display device so as to be superimposed on the acquired real video. The effective line-of-sight position area setting means for setting an effective area of the line-of-sight position surrounded by a plurality of positions, and whether or not the detected line-of-sight position is within the effective area of the line-of-sight position surrounded by the plurality of positions Determining means for determining whether or not the image display means displays the CG image when the detected line-of-sight position is not within the effective region of the line-of-sight position surrounded by the plurality of positions. Not superimposed on the image, and displaying only the physical image.
[0017]
The image processing apparatus according to claim 2, in the image processing apparatus according to claim 1, detects effective CG image position area setting means for setting an effective area of the CG image position, and detects the position of the CG image to be superimposed. CG video position detection means, and other determination means for determining whether or not the position of the detected CG video is within the effective area of the set CG video position, the video display means When the position of the detected CG video is not within the effective area of the set CG video position, the detected CG video is not superimposed on the real video and only the real video is displayed. To do.
[0018]
In order to achieve the above object, the image processing method according to claim 3 includes a real image capturing step of acquiring a real image at a predetermined line-of-sight position, a line-of-sight position detecting step of detecting the predetermined line-of-sight position, An image processing method comprising: a CG video generation step of generating a CG video according to the detected line-of-sight position; and a video display step of superimposing the generated CG video on the acquired real video and displaying it on a display device. And an effective line-of-sight position region setting step of setting an effective area of the line-of-sight position surrounded by a plurality of positions, and whether or not the detected line-of-sight position is within the effective area of the line-of-sight position surrounded by the plurality of positions And the image display step displays the CG image when the detected line-of-sight position is not within the effective region of the line-of-sight position surrounded by the plurality of positions. Not superimposed on the image, and displaying only the physical image.
[0019]
The image processing method according to claim 4 is the image processing method according to claim 3, wherein an effective CG image position region setting step for setting an effective region of the CG image position and a position of the CG image to be superimposed are detected. A CG video position detection step, and another determination step for determining whether or not the position of the detected CG video is within an effective area of the set CG video position, wherein the video display step includes: When the position of the detected CG video is not within the effective area of the set CG video position, the detected CG video is not superimposed on the real video and only the real video is displayed. To do.
[0020]
In order to achieve the above object, a program according to claim 5 is a program for causing a computer to execute an image processing method, a real image photographing module for acquiring a real image at a predetermined line-of-sight position, and the predetermined line-of-sight. A line-of-sight position detection module for detecting a position, a CG video generation module for generating a CG video corresponding to the detected line-of-sight position, and the generated CG video superimposed on the acquired real video on the display device A video display module to display, an effective line-of-sight position area setting module for setting an effective area of the line-of-sight position surrounded by a plurality of positions, and an effective area of the line-of-sight position in which the detected line-of-sight position is surrounded by the plurality of positions A discriminating module that discriminates whether or not the video line is within the plurality of detected visual line positions. When not in the effective area of the viewpoint position surrounded by the position does not overlap the CG image to the actual image, and displaying only the physical image.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an image processing (mixed reality presentation) apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings.
[0022]
FIG. 1 is a block diagram showing a schematic configuration of an image processing (mixed reality presentation) apparatus according to an embodiment of the present invention.
[0023]
1, the mixed reality presentation apparatus according to the embodiment of the present invention includes a CPU 101, a memory 103, a memory 104, a position and orientation sensor main body 105a, a head mounted display (HMD) 106, and a video camera 107. Are connected to each other via a computer bus 102. Position / orientation sensors 105b, 105c, and 105d are connected to the position / orientation sensor main body 105a. Among these, the position / orientation sensor 105b is attached to the HMD 106 in order to detect the line-of-sight position and orientation of the video camera 107. Both of the position / orientation sensors 105c and 105d are for CG image position / orientation detection.
[0024]
The memory 103 and the memory 104 have the same hardware configuration, but the memory 103 has a gaze position / orientation detection module 110, a real image module 111, a gaze position validity determination module 112, and a CG as programs for executing the processing of FIG. A video position / attitude detection module 113, a CG video position validity determination module 114, a CG video generation module 115, a video display module 116, an effective line-of-sight position area setting module 117, and an effective CG video position area setting module 118 are stored. Reference numeral 104 denotes a sensor effective area central data area 120, a video memory area 121, a visual line position / attitude data area 122, a CG video position / attitude data area 123, an effective visual line position data area 124, and an effective data area used in the program. CG movie Position data area 125 has a line-of-sight position distance data area 126, CG video location distance data area 127.
[0025]
FIG. 2 is a flowchart of the video display process executed by the mixed reality presentation apparatus of FIG.
[0026]
In FIG. 2, first, the gaze position / orientation detection module 110 uses a conventionally used technique, for example, a magnetic sensor or the like, to detect the three-dimensional position (X, Y, Z) and attitude (rolling, (Pitching, yawing) is detected, and the position and orientation data of the video camera 107 are written into the line-of-sight position / orientation data area 122 of the memory 104 (step S201), and then captured by the video camera 107 by the real image capturing module 111. The video is captured, and the video in the real space (real video) is written in the video memory area 121 of the memory 104 (step S202).
[0027]
In the subsequent step S203, the gaze position validity determination module 112 of FIG. 3 described later determines whether or not the detected gaze position of the video camera 107 is in the valid area, and when the gaze position is in the valid area, In step S204, a variable N consisting of a positive integer for counting the number of position / orientation sensors for CG is initialized to 1 (step S204). In the present embodiment, the number of position and orientation sensors is two, that is, the position and orientation sensors 105c and 105d.
[0028]
Next, the CG image position / orientation detection module 113 detects the position / orientation at which the Nth CG image is to be displayed using the position / orientation sensors 105 c and 105 d, and stores it in the CG image position / orientation data area 123 in the memory 104. (Step S205). The CG image position / orientation detection module 113 has the same configuration as the above-described line-of-sight position / orientation detection module 110.
[0029]
In step S206, the later-described CG video position validity determination module 114 in FIG. 4 determines whether or not the detected CG video position is within the valid area. If the CG video position is within the valid area, Based on the data in the line-of-sight position / orientation data area 122 and the data in the CG image position / orientation data area 123, the image generation module 115 superimposes the Nth CG image modeled three-dimensionally on the image memory area 121 ( The process proceeds to step S207) and step S208. On the other hand, if the result of determination in step S206 is that the CG video position is not within the valid area, step S207 is skipped and processing proceeds to step S208.
[0030]
The CG video generation module 115 renders the CG video as if viewed from the line of sight indicated by the data in the line-of-sight position / orientation data area 122, which is placed in the position / posture indicated by the data in the CG video position / attitude data area 123. In the video memory area 121, the video of the real space has already been written by the real video shooting module 111 in step S203, and the CG video is drawn on top of this.
[0031]
In the subsequent step S208, the variable N is incremented by 1, and in the next step S209, it is determined whether or not the variable N is 2 (the number of CG video images) or less. On the other hand, if the process is repeated and the number exceeds 2, the process proceeds to step S210.
[0032]
As a result of the determination in step S203, if the line-of-sight position is not within the effective area, the process immediately proceeds to step S210 without executing the processing after step S204 (storage of the Nth CG video in the video memory area).
[0033]
In the subsequent step S210, the video display module 116 writes the video written in the video memory area 121 into the frame memory of the HMD 106 and performs actual display.
[0034]
Next, in step S211, it is determined whether or not the entire process is to be ended. If not, the process after step S201 is repeated again. By repeating the processing, a real image is captured one after another from the video camera 107, and a cycle in which the CG image is overwritten and displayed is continuously performed.
[0035]
According to the processing of FIG. 2, when the line-of-sight positions of the position / orientation sensors 105c and 105d are not within the effective region (NO in step S203), only the real video is displayed on the HMD without rendering all the CG video ( Step S210) When the CG video position is not within the effective area (NO in Step S206), drawing of only the CG video is stopped and other CG video is drawn (Steps S206 to S207), which is unnatural. Display of video can be prevented.
[0036]
FIG. 3 is a flowchart of the gaze position validity determination module 112 in step S203 of FIG.
[0037]
It is assumed that the effective line-of-sight position data area 124 of the memory 104 has previously recorded the value of the effective line-of-sight position of the video camera 107 by the effective line-of-sight position area setting module 117. Further, it is assumed that the coordinate (X, Y, Z) that is the center of the effective area of the position and orientation sensor is stored in the sensor effective area center data area 120 of the memory 104.
[0038]
The data in the effective line-of-sight position data area 124 is, for example, the value of the effective distance of the line-of-sight position of the video camera 107 from the center of the sensor effective area. In this case, the value is one numerical value.
[0039]
In FIG. 3, first, the distance between the line-of-sight position of the video camera 107 and the center of the sensor effective area is calculated from the data of the line-of-sight position / orientation data area 122 of the memory 104 and the data of the sensor effective area center data area 120. Is stored in the line-of-sight position distance data area 126 (step S301).
[0040]
Next, it is determined whether or not the data in the line-of-sight position distance data area 126 is less than the data in the effective line-of-sight position data area 124 (step S302), and when the line-of-sight position distance of the video camera 107 is less than the effective line-of-sight position distance. Then, it is determined that the line-of-sight position is within the effective area (step S303), and when the line-of-sight position distance is equal to or greater than the effective line-of-sight position distance, it is determined that the line-of-sight position is not within the effective area (step S304). .
[0041]
The data stored in the effective line-of-sight position data area 124 is not limited to the distance from the center of the sensor effective area as described in the present embodiment, but may be a set of a plurality of positions indicating a specific area. In that case, the gaze position validity determination module 112 determines whether or not the gaze position / posture is within an area surrounded by the plurality of positions.
[0042]
FIG. 4 is a flowchart of the CG video position validity determination module 114 in step S206 of FIG.
[0043]
It is assumed that the effective CG image position data area 125 of the memory 104 stores the value of the effective area of the CG image position by the effective CG image position area setting module 118 in advance. Further, it is assumed that coordinates (X, Y, Z) serving as the center of the effective area of the position and orientation sensor are stored in the sensor effective area center 120 of the memory 104.
[0044]
The effective CG image position data area 125 is, for example, the value of the effective distance of the CG image position from the center of the sensor effective area, and in this case, is a single numerical value.
[0045]
In FIG. 4, in step S 401, the distance between the CG image position and the sensor effective area center is calculated from the data in the CG image position / orientation data area 123 and the data in the sensor effective area center data area 120 in the memory 104. Is stored in the CG video position distance data area 127 (step S401).
[0046]
Next, it is determined whether or not the data in the CG video position / attitude data area 123 is less than the data in the valid CG video position data area 125 (step S402), and when the CG video position distance is less than the valid CG video position distance. The CG video position is determined to be within the effective area (step S403). If the CG video position distance is equal to or greater than the effective CG video position distance, it is determined that the CG video position is not within the effective area (step S404). Exit.
[0047]
In the above embodiment, if the position / orientation sensors 105b, 105c, and 105d for detecting the gaze position / orientation and the CG image position / orientation go out of the effective area of the sensor, the sensor cannot detect a correct value and the detected value Therefore, the effective line-of-sight position data area 124 and the effective CG video position data area 125 are usually within the effective area of the sensor. In addition, since the position and orientation sensors 105b to 105c may include many errors in the detection values near the boundary of the effective area, the effective line-of-sight position area and the effective CG video position area are based on the detection values of the position and orientation sensors 105b to 105c. It is preferable to set the area according to the installation status of the position and orientation sensor so that the CG image to be drawn does not become unnatural.
[0048]
In addition, an effective area may be individually set for each of the position and orientation sensors 105b to 105c. In this case, a plurality of effective CG video position data areas 125 in the memory 104 may be prepared for each position and orientation sensor. .
[0049]
It goes without saying that the present invention can also be applied to a case in which a storage medium storing software program modules for realizing the above-described embodiments is achieved by supplying a program to a system or apparatus. In this case, the program module itself read from the storage medium realizes the novel function of the present invention, and the storage medium storing the program constitutes the present invention.
[0050]
In the above embodiment, the program module is stored in the memory 103. As a storage medium for supplying the program module, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, an MO, a CD-R, a DVD, Although various things, such as a magnetic tape and a non-volatile memory card, can be considered, it is not necessary to limit to a specific thing, What is necessary is just to be able to memorize | store the said program.
[0051]
【The invention's effect】
According to the present invention, the line-of-sight position of the real image which is detected, when not in the effective area of the line-of-sight position of the set real images, not overlap the CG image generated in accordance with the viewpoint position in the physical image Therefore, since only the real image is displayed, it is possible to prevent an unnatural image from being displayed.
Furthermore, according to the present invention, the effective region of the line-of-sight position surrounded by a plurality of positions is set, and it is determined whether or not the detected line-of-sight position is within the effective region surrounded by the plurality of positions. Can be arbitrarily set spatially.
[0052]
According to the image processing device of claim 2 and the image processing method of claim 4, when the position of the detected CG video is not within the effective area of the set CG video position, the detected CG video Since the video is not superimposed on the real video and only the real video is displayed , drawing of the CG video is stopped and another CG video is drawn, thereby preventing an unnatural video from being displayed.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a schematic configuration of an image processing (mixed reality presentation) apparatus according to an embodiment of the present invention.
FIG. 2 is a flowchart of video display processing executed by the mixed reality presentation device of FIG.
FIG. 3 is a flowchart of a gaze position validity determination module 112 in step S203 of FIG.
4 is a flowchart of a CG video position validity determination module 114 in step S206 of FIG.
[Explanation of symbols]
101 CPU
102 Computer bus 103, 104 Memory 105a Position / orientation sensor main body 105b, 105c, 105d Position / orientation sensor 106 HMD
107 video camera

Claims (5)

A real image photographing means for acquiring a real image at a predetermined line-of-sight position;
Eye-gaze position detecting means for detecting the predetermined eye-gaze position;
CG video generation means for generating a CG video according to the detected line-of-sight position;
An image processing apparatus comprising: a video display unit configured to superimpose the generated CG video on the acquired real video and display the video on a display device;
Effective line-of-sight position area setting means for setting an effective area of the line-of-sight position surrounded by a plurality of positions;
Determining means for determining whether or not the detected line-of-sight position is within an effective area of the line-of-sight position surrounded by the plurality of positions;
The video display means displays only the real video without superimposing the CG video on the real video when the detected gaze position is not within the effective region of the gaze position surrounded by the plurality of positions. An image processing apparatus. Effective CG image position setting means for setting an effective area of the CG image position;
CG image position detecting means for detecting the position of the CG image to be superimposed;
Other discriminating means for discriminating whether or not the position of the detected CG video is within the effective area of the set CG video position,
When the position of the detected CG image is not within the effective area of the set CG image position, the image display means does not superimpose the detected CG image on the actual image and only the actual image the image processing apparatus according to claim 1, wherein the displaying the. A real image shooting process of acquiring a real image at a predetermined line-of-sight position;
A line-of-sight position detecting step of detecting the predetermined line-of-sight position;
A CG image generation step for generating a CG image corresponding to the detected line-of-sight position;
An image processing method comprising: a video display step of displaying the generated CG video on a display device in a superimposed manner on the acquired real video;
An effective line-of-sight position area setting step for setting an effective area of the line-of-sight position surrounded by a plurality of positions;
A determination step of determining whether or not the detected line-of-sight position is within an effective region of the line-of-sight position surrounded by the plurality of positions,
The video display step displays only the real video without superimposing the CG video on the real video when the detected gaze position is not within the effective region of the gaze position surrounded by the plurality of positions. An image processing method. An effective CG image position area setting step for setting an effective area of the CG image position;
A CG image position detecting step for detecting a position of the CG image to be superimposed;
Another determination step of determining whether or not the position of the detected CG image is within an effective area of the set CG image position,
In the video display step, when the position of the detected CG video is not within the effective area of the set CG video position, the detected CG video is not superimposed on the real video and only the real video is displayed. the image processing method according to claim 3, wherein the displaying the. In a program for causing a computer to execute an image processing method,
A real image capturing module for acquiring a real image at a predetermined line-of-sight position;
A line-of-sight position detection module for detecting the predetermined line-of-sight position;
A CG video generation module that generates a CG video according to the detected line-of-sight position;
A video display module that superimposes the generated CG video on the acquired real video and displays it on a display device;
An effective line-of-sight position area setting module for setting an effective area of the line-of-sight position surrounded by a plurality of positions;
A determination module that determines whether or not the detected line-of-sight position is within an effective region of the line-of-sight position surrounded by the plurality of positions;
When the detected line-of-sight position is not within the effective area of the line-of-sight position surrounded by the plurality of positions, the video display module displays only the real video without superimposing the CG video on the real video The program characterized by doing.

Images