JP6971590B2 - Image processing equipment, image processing methods and programs - Google Patents

Description

The present invention relates to a technique for correcting an image captured by an imaging unit.

In recent years, mixed reality, so-called MR (Mixed Reality) technology, has been known as a technology for seamlessly fusing the real world and the virtual world in real time. As one of the MR techniques, a technique using a video see-through HMD (Head Mounted Display) is known. With this technology, a subject that substantially matches the subject observed from the pupil position of the HMD user (user) is imaged with a video camera or the like, and the user can observe an image in which CG (Computer Graphics) is superimposed and displayed on the captured image. To do so.

In a system using this technology, a CG to be drawn is rendered for an image captured by a video camera, and a display image in which the generated CG is superimposed on the image is displayed on the internal panel of the HMD as an image display device. Is common. At that time, since the calculation time for performing the above series of processing always occurs, a delay (latency) occurs between the time of the captured image and the time of the displayed image. Therefore, when the user moves his / her head, the field of view of the image observed through the HMD does not change at the moment when the user moves his / her head, but the direction of the field of view starts to move according to the direction of the head after a delay of the image has elapsed. It will be. Such a delay in the image makes the user feel uncomfortable.

To solve this problem, a technique has been proposed in which the deviation between the real world to be observed by the user and the displayed image is corrected by using the delay time and the operation predicted value of the HMD. In Patent Document 1, a display image to be viewed by the user is cut out from the frame memory from the captured image of a wide angle of view stored in the frame memory based on the head information of the user detected by the rotation angle sensor, and displayed on the display. The HMD to be displayed is disclosed.

Japanese Unexamined Patent Publication No. 8-191419

However, in the technique of Patent Document 1, the image display device does not have an image pickup optical system and a display optical system. Therefore, when the image display device has an image pickup optical system and a display optical system, the influence of the image correction of each optical system is not taken into consideration. Therefore, according to the present invention, when correcting the display image so as to reduce the deviation from the real world to be observed by the user, even when an image display device having an image pickup optical system and a display optical system is used, each of them is used. The purpose is to ensure that the image correction of the optical system is properly performed.

In order to solve the above problems, according to the present invention, an image pickup unit having an image pickup optical system for capturing an image in a real space, a detection unit for detecting the movement of an image display device, and a display optical system are provided for display. A first correction means for correcting an image captured by an image pickup unit based on the optical characteristics of the image pickup optical system on an image processing device connected to an image display device including a display unit for displaying an image. And, the drawing position of the virtual image is determined based on the information in the real space, the generation means for generating the virtual image based on the drawing position, and the drawing position of the virtual image are determined, and based on the drawing position. Based on the measuring means for measuring the processing time required to generate the virtual image, the measured processing time, and the movement of the image display device, the generated virtual image is deformed or modified. A correction means for modifying the virtual image by performing at least one process of movement, a synthesis means for combining the modified virtual image and the corrected captured image to generate a display image, and the display optics. It is characterized by comprising a second correction means for correcting the generated display image based on the optical characteristics of the system.

According to the above configuration, in the present invention, when the display image is corrected so as to reduce the deviation from the real world to be observed by the user, a case where an image display device having an image pickup optical system and a display optical system is used. In addition, the image correction of each optical system will be appropriately performed.

The block diagram which shows the structure of the image processing system which concerns on 1st Embodiment. The figure explaining the head movement of a user in 1st Embodiment. The block diagram which shows the structure of the image conversion part which concerns on 1st Embodiment. The flowchart of the process of the deformation movement calculation unit which concerns on 1st Embodiment. The figure explaining the effect by the image conversion part which concerns on 1st Embodiment. The block diagram which shows the structure of the image processing system which concerns on 2nd Embodiment. The block diagram which shows the structure of the image processing system which concerns on the modification of 2nd Embodiment. The block diagram which shows the structure of the image processing system which concerns on 3rd Embodiment. The figure explaining the effect by the image conversion part which concerns on 3rd Embodiment. The block diagram which shows the structure of the image processing system which concerns on 4th Embodiment.

[First Embodiment]
Hereinafter, the details of the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a functional configuration of an image processing system according to the first embodiment. In the figure, the image processing system of the present embodiment has a configuration in which a head-mounted display (hereinafter, HMD) 1 as a head-mounted image display device and an image processing device 14 are connected. The HMD 1 and the image processing device 14 each have a hardware configuration such as a CPU, ROM, RAM, and HDD, and the CPU executes a program stored in the ROM, HD, or the like to realize each functional configuration and processing of a flowchart. Will be done. The RAM has a storage area that functions as a work area in which the CPU develops and executes a program. The ROM has a storage area for storing a program or the like executed by the CPU. The HDD has a storage area for storing various data including various programs required for the CPU to execute a process, data related to a threshold value, and the like.

The HMD 1 includes an image pickup unit 11, a display unit 12, and an motion detection unit 13. The image pickup unit 11 includes an image pickup optical system 111 and an image pickup sensor 112, and generates an image pickup image of the real world. The inside of the image pickup optical system 111 mounted on the image pickup unit 11 is an image pickup lens or the like. The image pickup sensor 112 is an image pickup element such as a CMOS sensor that photoelectrically converts a subject image formed by the image pickup optical system 111. In this way, the image pickup unit 11 forms an image of the light in the real world on the image pickup element via the image pickup lens, converts the light into an electric signal, and acquires the light as an image pickup image.

The display unit 12 includes the display optical system 121 and the display panel 122, and the user observes the display image displayed on the display panel 122 through the display optical system 121. The display optical system 121 mounted on the display unit 12 is composed of a lens, a prism-type optical element having a plurality of reflective surfaces, and the like, and by optically magnifying the image of the display panel 122, an optimum display as an HMD is displayed. Present the video to the user.

The motion detection unit 13 detects the motion of the HMD 1 by detecting the position and posture (position, direction, and posture) of the head of the user wearing the HMD 1. The motion detection unit 13 is configured by using an acceleration sensor, an angular velocity sensor, or the like.

FIG. 2 shows a conceptual diagram for conceptually explaining the user's head movement. In the present embodiment, the line-of-sight direction of the user wearing the HMD1 is defined as the Roll axis (R), the horizontal direction with respect to the line-of-sight direction is defined as the Pitch axis (P), and the direction perpendicular to the line-of-sight direction is defined as the Yaw axis (Y). .. Then, the case of rotation with respect to each axis is defined as Roll, Pitch, and Yaw, respectively. That is, Roll is when the user tilts his head, Pitch is when the user nods, and Yaw is when the user shakes his head. The motion detection unit 13 outputs data such as acceleration in the Roll axis, Pitch axis, and Yaw axis direction and angular velocity of rotation.

The image processing device 14 includes an image pickup optical system correction unit 141 that performs image correction of the image pickup optical system, and a display optical system correction unit 142 that performs image correction of the display optical system. Further, the image conversion unit 143 is provided to convert an image so as not to be affected by the image delay caused by the movement of the HMD 1 detected by the motion detection unit 13 with respect to the image image-corrected image. The image conversion unit 143 functions as a generation unit that generates an image to be displayed on the display unit 12 by converting the image as described above. In the present embodiment, the image captured by the image pickup unit 11 is displayed as it is as a display image, but for example, an enlarged image of the captured image may be observed by the user as a display image.

The HMD 1 may include a part or all of each function of the image processing device 14.

Generally, an image obtained via an optical system has different image magnifications near the center and the periphery of the optical axis of the optical system, and the image magnification differs depending on the wavelength of light, resulting in distortion, chromatic aberration of magnification, and reduction in peripheral illumination. It is affected by the image quality. The image pickup optical system correction unit 141 and the display optical system correction unit 142 have a function of performing image correction for correcting the influence on the image quality generated via the optical system. The contents of the image correction include shading correction and distortion correction. The image pickup optical system correction unit 141 corrects the optical characteristics generated by the image pickup optical system 111, and the display optical system correction unit 142 corrects the optical characteristics generated by the display optical system 121 in advance.

Although it has been described in FIG. 1 on the premise that the image pickup unit 11 and the display unit 12 are each one, the present embodiment is not limited to this. Both the image pickup unit 11 and the display unit 12 may be configured to be plural, or either of them may be configured to be plural. For example, in a head-mounted image display device such as an HMD, the image pickup unit 11 and the display unit 12 are generally composed of two for each eye.

Next, the details of the image conversion unit 143 will be described. FIG. 3 shows the functional configuration of the image conversion unit 143 according to the present embodiment. The delay amount storage unit 31 stores the delay amount generated between the time of the captured image and the time of the displayed image. This delay amount may be a fixed value, or the delay amount that fluctuates systematically may be acquired and stored from the outside.

The deformation movement calculation unit 32 is a functional unit that calculates the movement amount and the deformation amount of the captured image based on the head motion information input from the motion detection unit 13 and the delay amount stored in the delay amount storage unit 31. Is. The operation of the deformation movement calculation unit 32 will be described in detail below with reference to the flowchart of FIG.

First, the deformation movement calculation unit 32 acquires the head motion information Q from the motion detection unit 13 (ST401). As described above, the head motion information Q is sensor data measured by using sensors such as an acceleration sensor and an angular velocity sensor, and is data obtained by quantifying the movement of the head (swing or movement of the head). Next, the deformation movement calculation unit 32 acquires the delay amount Dt from the delay amount storage unit 31 (ST402).

After that, the deformation movement calculation unit 32 performs a process of determining vertical or horizontal swing of the head from the head motion information Q (ST403). As a method of determining the vertical or horizontal swing of the head, for example, there is a method of determining from the angular velocity of Yaw and the angular velocity of Pitch. In the present embodiment, the deformation movement calculation unit 32 _{determines that the head swings when the angular velocity Y ω of} Yaw is equal to or more than a certain value, and that the head swings when the angular velocity P _{ω of} Pitch exceeds a certain value. .. When it is determined that the head is shaken vertically or horizontally, the deformation movement calculation unit 32 calculates the horizontal shift amount (Hd) and the vertical shift amount (Vd) of the image (ST404). Further, when it is determined that the head is not shaken vertically or horizontally, the deformation movement calculation unit 32 sets the horizontal shift amount (Hd) and the vertical shift amount (Vd) of the image to 0, respectively (ST405).

Next, the deformation movement calculation unit 32 performs a process of determining the inclination of the head from the head motion information Q (ST406). As a method of determining the inclination of the head, for example, there is a method of determining from the angular velocity of Roll. In the present embodiment, the deformation movement calculation unit 32 _{determines that the head is tilted when the angular velocity R ω of} Roll is equal to or more than a certain value. When it is determined that the head is tilted, the deformation movement calculation unit 32 calculates the rotation angle (Θd) of the image (ST407). If it is determined that the head is not tilted, the deformation movement calculation unit 32 sets the rotation angle (Θd) of the image to 0 (ST408).

Next, the deformation movement calculation unit 32 performs a process of determining the front-back movement of the head from the head movement information Q (ST409). As a method of determining the back-and-forth movement of the head, for example, there is a method of determining from the acceleration information in the Roll axis direction. In the present embodiment, the deformation movement calculation unit 32 determines that the head has moved back and forth when _{the acceleration Ra in the Roll axis direction is equal to or higher than a certain value.} When it is determined that the head moves back and forth, the deformation movement calculation unit 32 calculates the enlargement ratio (Ed) of the image (ST410). Further, when it is determined that the head does not move back and forth, the deformation movement calculation unit 32 sets the enlargement ratio (Ed) of the image to 0 (ST411).

Finally, the deformation movement calculation unit 32 transmits the calculated horizontal shift amount Hd, vertical shift amount Vd, rotation angle Θd, and enlargement ratio Ed to the deformation movement execution unit 33 (ST412). In this way, the deformation movement calculation unit 32 moves and deforms the captured image based on the head motion information input from the motion detection unit 13 and the delay amount stored in the delay amount storage unit 31. Calculate the amount.

The order of calculation of the movement amount and the deformation amount is not particularly limited. The configuration may be such that the magnification due to the forward / backward movement of the head is calculated first, and then the shift amount due to the vertical / horizontal swing of the head is calculated. Further, the means for calculating the movement amount and the deformation amount may also be image-converted so that the user does not feel uncomfortable within a range predictable from the head motion information Q and the delay amount Dt.

The deformation movement execution unit 33 is a functional unit that performs image conversion of the captured image input from the image pickup optical system correction unit 141 using the image conversion parameters input from the deformation movement calculation unit 32. Image conversion is at least one of movement and deformation, such as horizontal / vertical shift, rotation / enlargement / reduction of an image, and the like. Image conversion methods include a method of calculating using an image conversion algorithm (bilinear, bicubic, etc.) using a frame buffer / line buffer, and a method of pseudo-shifting the video by shifting the video synchronization signal. be. By such image conversion, it is possible to alleviate the discomfort caused by the difference between the experience image and the real world caused by the latency.

FIG. 5 is a diagram for explaining the effect of the image conversion unit 143 according to the present embodiment. FIG. 5 shows that when the user wearing the HMD1 has (a) no head movement, (b) Yaw movement of the head, (c) Pitch movement of the head, and (d) Roll of the head. When it operates, it shows the case where (e) the head is moved back and forth. Further, in FIGS. 5A to 5E, in order from the top, the image seen by the user through the display unit 12, the state of the user using the HMD seen from above, and the user using the HMD seen from the side. The state, the state which the user who is using HMD is seen from the front is shown.

_{As shown in FIG. 5A, it can be seen that when there is no head movement, the image within the range of the image pickup angle of view A θ} generated by the image pickup optical system 111 is displayed within the range of the resolution HxV. Further, as shown in FIG. 5B, when the head is operated Yaw, the deformation movement calculation unit 32 has the angular velocity Y _ω and the delay amount of Yaw included in the head operation information Q as described in the operation of ST404 described above. Obtain the Yaw angle Y _{θ from Dt.} Then, the horizontal shift amount Hd of the image can be obtained from the relationship between the Yaw angle Y _{θ and the reference distance L.} The deformation movement execution unit 33 shifts the image in the horizontal direction based on the horizontal shift amount Hd so that the user can be presented with an image that matches the line-of-sight direction of the user. Further, as shown in FIG. 5C, even when the head is Pitch-operated, the deformation movement calculation unit 32 calculates the vertical shift amount Vd of the image, and the deformation movement execution unit 33 calculates the image, as in the Yaw operation. Is moved vertically. This makes it possible to present the user with an image that matches the line-of-sight direction of the user.

As shown in FIG. 5D, when the head is moved in Roll, the deformation movement calculation unit 32 is based on the angular velocity R _{ω of} Roll and the delay amount Dt included in the head movement information Q as described in the operation of ST407 described above. Find the Roll angle R _θ . Then, the rotation angle Θd of the image is obtained from the Roll angle R _θ. The deformation movement execution unit 33 rotates the image based on the rotation angle Θd so as to present the image matching the user's line of sight to the user.

As shown in FIG. 5 (e), when the head is moved back and forth, the deformation movement calculation unit 32 has an acceleration R _a and a delay amount in the Roll axis direction included in the head motion information Q as described in the operation of ST410 described above. calculating the moving amount _{R M} of Roll axis from dt. Magnification Ed of the Roll axis direction of the moving amount R _M and the image from the relationship of imaging angle A _theta is determined. The transformation / movement execution unit 33 enlarges / reduces the image based on the enlargement ratio Ed so that the user can be presented with an image that matches the user's back-and-forth movement. As described above, the image conversion unit 143 can reduce the discomfort due to the image delay by executing the correction process on the displayed image based on the operation of the motion detection unit 13.

Next, the effect of executing the processing of the image conversion unit 143 after the image pickup optical system correction unit 141 and before the display optical system correction unit 142, which is a feature of the image processing system of the present embodiment, will be described. For example, it is assumed that the image conversion unit 143 is executed before the image pickup optical system correction unit 141. The image output from the image conversion unit 143 is image-converted according to the information of the motion detection unit 13. That is, the image pickup optical characteristics generated by the image pickup optical system 111 are also moved and deformed by the image conversion. Normally, in the correction process executed by the image pickup optical system correction unit 141, information on the image pickup optical characteristics is stored in advance, and the correction process is performed based on the information stored in advance. That is, when the image pickup optical system correction unit 141 is executed for the image output from the image conversion unit 143, the intended image pickup optical correction is not performed, and the image quality may be deteriorated in some cases.

The same applies when the image conversion unit 143 is executed after the display optical system correction unit 142. The display optical system correction unit 142 corrects in advance the display optical characteristics that occur when observing through the display optical system 121. When the image conversion unit 143 is executed for the image after the display optical system correction, the display optical characteristics that have been image-corrected in advance are also moved and deformed, so that the display optical correction is performed as intended. It may not be done and the image quality may deteriorate.

On the other hand, in the present embodiment, in an image processing system including an image display device (HMD) having an image pickup optical system and a display optical system, the image processing system is after the image pickup optical system correction unit 141 and before the display optical system correction unit 142. The processing of the image conversion unit 143 is executed. Therefore, the image correction of each optical system can be appropriately performed, and it is possible to suppress the deviation of the displayed image due to the delay of the captured image without causing a sense of discomfort in each optical correction.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. The image processing system according to the present embodiment corresponds to a case where the delay is not constant with respect to the captured image captured in the real world. The configurations already described in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 6 is a block diagram showing a configuration of an image processing system according to the present embodiment. The image processing system according to the present embodiment is different from the first embodiment in that it includes a delay amount measuring device 61 for measuring a delay amount of an image and a CG superimposing device 62. Further, the image conversion unit 143 executes the image correction process by using the delay amount information measured by the delay amount measuring device 61 and the display image from the CG superimposing device 62. The delay amount measuring device 61 and the CG superimposing device 62 may be configured as an integrated device with the image processing device 14, or may be configured as a separate device as shown in FIG.

The delay amount measuring device 61 has a delay amount measuring unit 611, and is a functional unit that measures the time difference between the captured image input from the image pickup optical system correction unit 141 and the display image input from the CG superimposing unit 622 described later. be. As a method of measuring the time difference, there is a method of preparing an internal counter and measuring the difference in the input timing of the vertical synchronization signals of the captured image and the displayed image. Another method is to embed the time stamp information as metadata in the captured image and measure the time difference until the time stamp information of the captured image and the time stamp information of the displayed image match. As long as the delay time of the video can be measured, the method is not limited to a specific method. In this way, the delay amount measuring device 61 can notify the image conversion unit 143 of the accurate delay amount Dt even when the delay time fluctuates depending on the video transmission and the processing load.

Next, the configuration of the CG superimposing device 62 will be described. The CG superimposition device 62 is composed of a CG superimposition position analysis unit 621 and a CG superimposition unit 622. The CG superimposition position analysis unit 621 performs an operation to calculate the drawing position of the CG by analyzing the image in the input captured image, and outputs the CG superimposition position information to the CG superimposition unit 622. In order to specify the drawing position of CG, for example, there is a method of arranging a physical index (hereinafter, MR marker) such as a two-dimensional code in the real space and detecting the MR marker by image analysis of the captured image. Further, instead of the method by image analysis from the captured image, a method of specifying the position of the HMD1 and calculating the drawing position of the CG according to the position of the HMD1 by mounting an optical sensor or the like in the space where the HMD1 is used. There is also. By these methods, the CG superimposition position analysis unit 621 calculates the CG drawing position information which is the information on the position where the CG should be drawn with respect to the captured image captured in the real space. The calculation of the drawing position of the CG is performed using a CPU or the like, but when a large number of MR markers are arranged in the real space and image analysis is performed, the calculation process for detecting the MR marker is performed multiple times. The time may not be constant. In such a case, the amount of delay of the video with respect to the input may change dynamically.

The CG superimposition unit 622 generates a display image in which a virtual image (CG data) is superposed on the captured image by using the CG superimposition position information calculated by the CG superimposition position analysis unit 621. The CG data corresponds to, for example, 3D CAD data created by using CAD software or the like, 3DCG data created by using graphic design software or the like, and the like. The CG superimposition unit 622 renders the CG data prepared in advance and superimposes it on the captured image based on the CG superimposition position information. This makes it possible to create an image (composite image) in which CG data that does not exist in the real space is drawn in the image. At this time, the rendering of the CG data is calculated using a GPU (Graphic Processing Unit) or the like, but the rendering process may become heavy and the processing time may not be constant depending on the content of the CG data. Even in such a case, the amount of delay of the video with respect to the input may change dynamically.

That is, in the CG superimposing device 62, the time difference from the input of the video to the output is often not constant. Even in such a case, the delay amount measuring device 61 can measure the video delay amount generated by the CG superimposing device 62 in real time.

The image conversion unit 143 estimates and determines the latency according to the delay amount Dt, and executes deformation and movement of the image. In the present embodiment, the delay amount Dt of the image changes dynamically, but by inputting the delay amount Dt from the delay amount measuring unit 611, the display image due to the delay of the captured image even when the delay changes in real time. It is possible to suppress the deviation of. This makes it possible to reduce the sense of discomfort due to video delay.

Here, a modified example of the present embodiment will also be described with reference to FIG. 7. FIG. 7 is a block diagram showing a configuration of an image processing system according to a modified example of the present embodiment. The image processing system of FIG. 7 includes a wireless device 71 instead of the CG superimposing device 62 shown in FIG. The wireless device 71 includes a wireless transmission unit (transmission unit) 711 that wirelessly transmits video to an external device (not shown) and a wireless reception unit (reception unit) 712 that wirelessly receives video processed by an external device (not shown). It is composed of.

In both the wireless transmission unit 711 and the wireless reception unit 712, it is assumed that the communication throughput may decrease and video transmission may become difficult depending on the radio wave condition of the radio. When the communication throughput decreases, video delays occur or video frames are dropped, so the amount of video delay may change dynamically.

As described above, even when the delay amount fluctuates in real time in a system via wireless transmission such as the wireless device 71, it is possible to suppress the deviation of the displayed image due to the delay of the captured image.

As described above, according to the present embodiment, even when the image processing system performs processing in which the delay is not constant for the captured image, the delay amount is measured and the image is corrected according to the delay amount. , It is possible to suppress the deviation of the displayed image due to the delay of the captured image.

In the present embodiment, it has been described that the delay amount dynamically fluctuates in the processing of the CG superimposing device 62 and the wireless device 71, but the processing of the CG superimposing device 62 and the wireless device 71 does not necessarily have a delay. It is not a constant process. Therefore, the delay amount measuring device 61 may not be added to the configuration of the first embodiment, but only the CG superimposing device 62 and the wireless device 71 may be added, and these are also included in the scope of the present invention. It is something that can be done.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. The image processing system according to the present embodiment performs CG generation, which is a process in which the delay is not constant with respect to the captured image captured in the real world, and only the generated CG is converted by the image conversion unit 143. The configurations already described in the first and second embodiments are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 8 is a block diagram showing a configuration of an image processing system according to the present embodiment. The difference between the image processing system according to the present embodiment and the second embodiment is that the CG generation device 82 does not superimpose the CG on the captured image, but only generates and transfers the CG, and the composition with the captured image is an image. This is a point performed by the synthesis unit 144. The merit of this configuration is that the captured image is not affected by the fluctuation of the delay amount, so that the image can be displayed to the user with the minimum delay. However, on the other hand, with respect to the CG image, at the time of compositing by the image compositing unit 144 due to the influence of the CG generation device 82, the CG image is displayed in a state including a delay amount with respect to the captured image. Therefore, in the present embodiment, only the CG data is processed by the image conversion unit 143 to suppress a sense of discomfort in the displayed image after the CG composition.

FIG. 9 is a diagram illustrating the effect of the image conversion unit 143 according to the present embodiment. FIG. 9 shows (a) a case where the user wearing the HMD1 does not move the head, and (b) a case where the user moves the head Yaw. Further, in each of FIGS. 9A and 9B, in order from the top, an image viewed by the user through the display unit 12, a state in which the user using the HMD is viewed from above, and a user in use of the HMD are viewed from the side. It shows how the user who is using the HMD is seen from the front.

As shown in FIG. 9A, when there is no head movement, a display image on which CG data 90, which does not exist in the real world, is superimposed is observed. On the other hand, as shown in FIG. 9B, when the head is Yaw-operated, the captured image in the real world is displayed without a delay amount, and the superimposed CG data is displayed with a delay amount included. It ends up. Therefore, if the CG data is simply superimposed on the captured image, the CG data is superimposed on the position 91 in the figure, which causes a sense of discomfort between the real world and the CG data.

Therefore, in the present embodiment, the deformation movement calculation unit 32 obtains the _{Yaw angle Y θ from the angular velocity Y ω of} Yaw and the delay amount Dt included in the head motion information Q. Further, the horizontal shift amount Hd of the image is obtained from the relationship between the Yaw angle Y _{θ and the reference distance L.} Based on this horizontal shift amount Hd, the deformation movement execution unit 33 can provide a comfortable image by shifting the CG data in the horizontal direction and then synthesizing it with the captured image.

As described above, in the present embodiment, even in the configuration in which only the generated CG is converted by the image conversion unit 143, the image is corrected according to the delay amount to suppress the deviation of the displayed image due to the delay of the captured image. can do.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described. The image processing system according to the present embodiment performs CG generation, which is a process in which the delay is not constant for the captured image, converts the generated CG by the image conversion unit 143, and further outputs the generated CG from the image pickup optical system correction unit 141. The captured image is converted by the image conversion unit 145. The configurations already described in the first to third embodiments are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 10 is a block diagram showing a configuration of an image processing system according to the present embodiment. The image processing system according to the present embodiment differs from the third embodiment in that both the captured image and the CG data are image-converted by the image conversion unit 143 or 145. The image conversion unit 145 has the same function as the image conversion unit 143, and has an image conversion function as described above. With this configuration, the image conversion unit 145 can correct the amount of delay that occurs on the hardware of the video information input from the image pickup unit 11 with the display image observed by the display unit 12. Therefore, in the present embodiment, it is possible to observe an image in which CG data is superimposed on the real world with the minimum delay amount.

[Other embodiments]
Further, in the present invention, software (program) that realizes the functions of the embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device provides the program. This is a process to read and execute. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. Further, the present invention may be applied to a system composed of a plurality of devices or a device composed of one device. The present invention is not limited to the above embodiments, and various modifications (including organic combinations of each embodiment) are possible based on the gist of the present invention, and these are excluded from the scope of the present invention. is not it. That is, all the configurations in which each of the above-described embodiments and modifications thereof are combined are also included in the present invention.

14 Image processing device 141 Image pickup optical system correction unit 142 Display optical system correction unit 143 Image conversion unit

Claims (5)

An image display device having an image pickup optical system for capturing an image in a real space, a detection unit for detecting the movement of the image display device, and a display unit having a display optical system for displaying a display image. A connected image processing device
A first correction means for correcting an image captured by the image pickup unit based on the optical characteristics of the image pickup optical system, and a correction means.
A generation means for determining a drawing position of a virtual image based on the information in the real space and generating the virtual image based on the drawing position.
A measuring means for determining the drawing position of the virtual image and measuring the processing time required to generate the virtual image based on the drawing position.
A correction means for modifying the virtual image by performing at least one processing of deformation or movement on the generated virtual image based on the measured processing time and the movement of the image display device.
A compositing means for generating a display image by compositing the modified virtual image and the corrected captured image, and
A second correction means for correcting the generated display image based on the optical characteristics of the display optical system, and
An image processing device characterized by comprising. The image processing device according to claim 1, wherein the detection unit detects changes in position and posture in each of roll, pitch, and yaw as the movement of the image display device. The image processing device according to claim 1 or 2, wherein the image display device is attached to a user's head and used. An image display device including an image pickup unit having an image pickup optical system to capture an image in a real space, a detection unit to detect the movement of the image display device, and a display unit having a display optical system to display a display image. An image processing method in a connected image processing device.
A step of correcting an image captured by the image pickup unit based on the optical characteristics of the image pickup optical system, and a step of correcting the image captured by the image pickup unit.
A step of determining a drawing position of a virtual image based on the information in the real space and generating the virtual image based on the drawing position.
A step of determining the drawing position of the virtual image and measuring the processing time required to generate the virtual image based on the drawing position.
A step of modifying the virtual image by performing at least one processing of deformation or movement on the generated virtual image based on the measured processing time and the movement of the image display device.
A step of combining the modified virtual image and the corrected captured image to generate a display image, and
A step of correcting the generated display image based on the optical characteristics of the display optical system, and
An image processing method characterized by comprising. A program for operating a computer as the image processing device according to any one of claims 1 to 3.

Images