JP7402660B2 - Head-mounted display device, control method for head-mounted display device - Google Patents

Description

The present invention relates to a control technique for a head-mounted display device.

In recent years, research on mixed reality (MR), which aims to seamlessly connect real space and virtual space, has been actively conducted. As one of these techniques, a technique using a video see-through head mounted display (HMD) is known. This technology superimposes and displays computer graphics (CG) images generated according to the position and orientation of the imaging device on images of real space captured by an imaging device such as a video camera, and displays them on an HMD display, etc. It is observed through a device.

In order to superimpose and display CG on images in real space without any deviation, it is necessary to estimate the position and orientation of the HMD with high precision. In addition, in order to correctly reflect the context of the real space image and CG and display them in a superimposed manner, it is necessary to accurately estimate the distance to the object shown in the real space image. A stereo camera is used for this estimation.

As pre-processing for estimation, distortion correction processing that removes camera distortion and rectify processing (non-patent document 1) that virtually parallelizes images captured by two cameras are performed. This rectify processing is a process of projecting images captured by two cameras onto a common image plane. This common image plane is a plane including an axis parallel to the baseline axis connecting the principal point positions of the two cameras. The captured images of the two cameras projected onto this common image plane match in the horizontal direction and the vertical direction on the captured images.

In order to perform distortion correction processing and rectify processing, it is necessary to estimate with high precision the internal parameters (principal point position, focal length, distortion correction value) and external parameters (relative position and orientation between stereo cameras) of the stereo cameras. There is. Highly accurate estimation can be achieved by creating a distortion correction table (correcting the optical axis of the camera at the same time) using highly accurately estimated internal and external parameters and correcting images captured by the stereo camera.

JP2012-147281A Japanese Patent Application Publication No. 2010-26994

A. Fusiello et al. “A Compact Algorithm for rectification of stereo pairs”, Machine Vision and Applications, 2000.

However, even if the internal and external parameters are estimated with high accuracy, as the temperature of the HMD increases over time, the molded parts and metal parts inside the camera expand due to the heat, and as a result, the internal Parameters change. Furthermore, the base material to which the stereo camera is attached is similarly affected by thermal deformation, causing external parameters to change as well. In other words, when the temperature of the HMD body increases, the error between the estimated parameter value and the actual parameter value increases, and as a result, the discrepancy between the real space image and the CG increases, or the There is a problem that the difference in the context between the spatial image and the CG becomes large.

Patent Document 1 discloses a method of correcting changes in distortion due to changes in camera temperature by switching and using correction parameters (specifically polynomial coefficients) created for each temperature. However, correction using correction parameters requires calculating a correction value for each coordinate position within the image sensor each time, resulting in a large processing load. Furthermore, there is a problem in that the processing load becomes enormous if calculations including rectify processing are attempted.

Patent Document 2 describes a method of performing correction by switching and using a correction table created for each temperature (specifically, a table of correction values determined based on the coordinate position within the image sensor) instead of using correction parameters. Disclosed. In this method, the mapping process is simply performed according to the values in the table, so the processing load can be minimized. However, compared to the correction parameters, a larger memory capacity is required. Therefore, it is difficult to have a detailed correction table for each temperature. Furthermore, when switching the correction table, a clear change can be seen in the video, which feels strange. In particular, when a plurality of cameras such as a stereo camera are used, there is a problem in that the sense of discomfort increases when changes occur in each camera separately. The present invention provides a technique for reducing discomfort caused by updating correction parameters used for image correction in image correction in a head-mounted display device.

One aspect of the present invention is a correction unit that corrects, for each of the plurality of imaging units, an image captured by the imaging unit using a correction parameter set for the imaging unit;
and a control means for updating the correction parameters set for the imaging section of interest to correction parameters corresponding to the internal temperature of the imaging section of interest, if the setting conditions are met ;
If the setting condition is satisfied when the internal temperature of the imaging section of interest is out of the internal temperature range corresponding to the correction parameter set for the imaging section of interest, the control means updating the correction parameters set for the imaging unit of interest to correction parameters corresponding to the internal temperature of the imaging unit of interest;
The control means is configured to control whether the setting condition is satisfied even when the internal temperature of the imaging unit of interest is outside an internal temperature range corresponding to a correction parameter set for the imaging unit of interest. If not, the above update will not be performed.
It is characterized by

According to the configuration of the present invention, in image correction in a head-mounted display device, it is possible to reduce discomfort caused by updating correction parameters used for image correction.

FIG. 1 is a block diagram showing a configuration example of an MR system. 2 is a flowchart showing operations of HMD 110 and information processing device 100. The figure which shows an example of the result of the rectify process with respect to a captured image. FIG. 7 is a diagram showing an example of distortion correction values corresponding to each internal temperature range. FIG. 2 is a block diagram illustrating a configuration in which an imaging unit 111, a temperature acquisition unit 112, a determination unit 113, and a distortion correction unit 116 are divided into a configuration corresponding to the right eye and a configuration corresponding to the left eye.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

[First embodiment]
First, a configuration example of an MR system according to the present embodiment will be described using the block diagram of FIG. 1. As shown in FIG. 1, the MR system according to this embodiment includes an HMD 110 that is an example of a head-mounted display device, and an information processing device 100 that generates images displayed on the HMD 110. The HMD 110 and the information processing device 100 are configured to be capable of data communication with each other via a wired and/or wireless network.

First, the HMD 110 will be explained. The imaging unit 111 is a stereo camera or a multi-lens camera with three or more eyes, and acquires a plurality of captured images (images of the real space) with parallax mutually by capturing images of the real space. In this embodiment, the imaging unit 111 will be described as a stereo camera. That is, in this embodiment, the imaging unit 111 includes a left eye camera and a right eye camera.

The distortion correction unit 116 corrects the captured images from each camera included in the imaging unit 111 using correction parameters set for the cameras, and transmits the corrected captured images to the information processing device 100. . The distortion correction unit 116 performs distortion correction processing, which includes rectify processing that not only corrects the distortion of the captured image but also corrects the optical axis of the camera that captured the captured image.

FIG. 3 shows an example of the results of rectify processing on a captured image. The captured image 301 is an image captured by the left camera (camera for the left eye) in the imaging unit 111, and the captured image 302 is an image captured by the right camera (camera for the right eye) in the imaging unit 111. Both the captured image 301 and the captured image 302 include distortion, deviations in principal point position, focal length, rotation, etc. The captured image 303 is a captured image obtained by applying rectify processing to the captured image 301. The captured image 304 is a captured image obtained by applying rectify processing to the captured image 302. In each of the captured image 303 and the captured image 304, the above various deviations included in the captured image 301 and the captured image 302 are corrected.

The temperature acquisition unit 112 acquires the internal temperature of each camera included in the imaging unit 111 by measurement. Note that the method for acquiring the internal temperature of the camera is not limited to a specific acquisition method.

The storage unit 114 stores a table in which distortion correction values (correction parameters) for correcting the distortion of captured images are registered, which are determined based on the coordinate position within the image sensor of the camera, for each internal temperature range. is stored in. That is, the storage unit 114 includes a correction parameter corresponding to the internal temperature range R1 (a range in which the internal temperature is greater than or equal to T1 and less than T2), a correction parameter corresponding to the internal temperature range R2 (a range in which the internal temperature is greater than or equal to T2 and less than T3), ..., correction parameters corresponding to each internal temperature range are stored.

For each camera included in the imaging unit 111, the determining unit 113 acquires from the storage unit 114 a correction parameter corresponding to an internal temperature range including the internal temperature acquired by the temperature acquiring unit 112 for the camera.

FIG. 4 is a diagram showing an example of distortion correction values corresponding to each internal temperature range. In FIG. 4, the horizontal axis represents the internal temperature of the camera, and the vertical axis represents the distortion correction value. In the example of FIG. 4, v1 is registered as the distortion correction value for a range where the internal temperature is less than d1. Further, v2 is registered as the distortion correction value for a range where the internal temperature is d1 or more and less than d2. Further, v3 is registered as a distortion correction value for a range where the internal temperature is d2 or higher. Therefore, when the internal temperature acquired from the temperature acquisition unit 112 is less than d1, the determination unit 113 acquires v1 as the distortion correction value. Furthermore, when the internal temperature acquired from the temperature acquisition unit 112 is greater than or equal to d1 and less than d2, the determination unit 113 acquires v2 as the distortion correction value. Furthermore, when the internal temperature acquired from the temperature acquisition unit 112 is equal to or higher than d2, the determination unit 113 acquires v3 as the distortion correction value.

In this way, the distortion correction value does not change continuously in response to changes in internal temperature, but changes discontinuously. For example, when the internal temperature changes from ds (< d1) to de (d1 < e < d2), the image that was distorted using distortion correction value = v1 will be distorted using distortion correction value = v2. It turns out. As a result, a clear change is seen in the image before and after the distortion correction value changes, creating a sense of discomfort. Moreover, if the distortion correction values are reduced from three, v1 to v3, to two, v1 and v2, in order to save the amount of data in the table, the sense of discomfort will further increase.

Therefore, in this embodiment, if the internal temperature of the camera of interest (imaging unit of interest) is outside the internal temperature range corresponding to the correction parameter set for the camera of interest, if the setting conditions are met, , updates the correction parameters set for the camera of interest to correction parameters corresponding to the internal temperature of the camera of interest. Note that in the initial state, default correction parameters are set for each camera.

The application control unit 115 determines, for each camera included in the imaging unit 111, that the internal temperature acquired by the temperature acquisition unit 112 for the camera is out of the internal temperature range corresponding to the correction parameter set for the camera. Determine if there are any. Then, if at least one of the cameras included in the imaging unit 111 is determined to be “out of place,” the application control unit 115 then determines whether or not the setting conditions are met. . If the application control unit 115 determines that the setting conditions are satisfied, the application control unit 115 transfers the correction parameters set for the camera to the determination unit 115 for the camera determined to be “off”. 113 updates the correction parameters to the correction parameters acquired from the storage unit 114.

For example, assume that the internal temperature of the camera of interest is lower than the temperature d1, and v1 is set as the distortion correction value of the camera of interest. After that, as a result of continuing to operate the MR system, if the internal temperature of the camera of interest becomes higher than temperature d1 and lower than temperature d2, the determining unit 113 acquires v2 as the distortion correction value of the camera of interest. do. The application control unit 115 updates the distortion correction value v1 set for the camera of interest to the distortion correction value v2 according to the current internal temperature of the camera of interest only when the setting conditions are met. On the other hand, if the setting conditions are not met, the application control unit 115 updates the distortion correction value v1 set for the camera of interest to a distortion correction value v2 according to the current internal temperature of the camera of interest. do not.

In this embodiment, the setting condition is ``For all cameras included in the imaging unit 111, the internal temperature acquired by the temperature acquisition unit 112 for the camera is an internal temperature corresponding to the correction parameter set for the camera. Assume that the condition is ``out of range''.

FIG. 5 shows a configuration in which the imaging unit 111, temperature acquisition unit 112, determination unit 113, and distortion correction unit 116 in FIG. 1 are divided into a configuration corresponding to the right eye and a configuration corresponding to the left eye. The imaging unit 111a, temperature acquisition unit 112a, determination unit 113a, and distortion correction unit 116a are configured for the left eye, and the imaging unit 111b, temperature acquisition unit 112b, determination unit 113b, and distortion correction unit 116b are configured for the right eye. .

In the configuration of FIG. 5, the internal temperature acquired by the temperature acquisition unit 112a is outside the internal temperature range corresponding to the correction parameter set for the imaging unit 111a (left eye camera), and the temperature acquisition unit 112a Assume that the setting information that the internal temperature acquired by the unit 112b is outside the internal temperature range corresponding to the correction parameter set for the imaging unit 111b (right eye camera) is satisfied. In this case, the application control unit 115 updates the correction parameters set for the imaging unit 111a to the correction parameters acquired by the determination unit 113a, and updates the correction parameters set for the imaging unit 111b to the determination unit 113b. Update to the correction parameters obtained by. In this way, by controlling the update of the correction parameters set for the imaging unit 111a and the correction parameters set for the imaging unit 111b with conditions, the imaging unit 111a and the imaging unit 111b can be updated. It is possible to reduce the sense of discomfort that occurs when the captured image changes randomly.

Next, the information processing device 100 will be explained. The information processing device 100 is a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device.

The processing unit 101 acquires the positions and orientations of the left eye camera and the right eye camera included in the imaging unit 111. There are various methods for acquiring the position and orientation of each camera, and the acquisition method is not limited to a specific one. For example, the position and orientation of each camera may be determined based on the position and orientation of the HMD 110 measured by a sensor (gyro sensor, magnetic sensor, ultrasonic sensor, etc.) attached to the HMD 110. Alternatively, the position and orientation of the camera may be estimated using features in an image captured by the camera.

The processing unit 101 then displays an image of the virtual object seen from a viewpoint having the position and orientation acquired for the camera for the left eye (virtual object image for left eye), and an image of the virtual object seen from the viewpoint having the position and orientation acquired for the camera for the right eye. An image of the virtual object (virtual object image for right eye) is generated. The processing unit 101 then generates a composite image (left-eye composite image) of the image captured by the left-eye camera (after correction by the distortion correction unit 116) and the left-eye virtual object image. The processing unit 101 also generates a composite image (right-eye composite image) of the captured image corresponding to the right-eye camera (after correction by the distortion correction unit 116) and the right-eye virtual object image. All of the generated composite images are images of a mixed reality space in which virtual space and real space are fused.

Note that the processing unit 101 performs a process of estimating the position and orientation of the HMD 110 (camera) with high precision in order to superimpose and display the image of the virtual object on the captured image without deviation. The processing unit 101 also performs processing for estimating the distance to the object in the real space with high accuracy in order to accurately reflect the front-back relationship between the object in the real space and the virtual object and display them in a superimposed manner. Furthermore, the processing unit 101 estimates the three-dimensional shape of the user's arm and the distance from the HMD 110, and performs hand masking processing and the like. The processing unit 101 then outputs the composite images (the left eye composite image and the right eye composite image) generated for each camera to the display unit 117 of the HMD 110.

In this embodiment, the display unit 117 has a display screen for the left eye and a display screen for the right eye, and displays a composite image for the left eye on the display screen for the left eye, and a display screen for the right eye. A composite image for the right eye is displayed on the display screen.

FIG. 2 is a flowchart showing the operations of the HMD 110 and the information processing device 100. In step S201, the imaging unit 111 acquires a plurality of captured images (images of the real space) with parallax mutually by capturing images of the real space. In step S202, the temperature acquisition unit 112 acquires the internal temperature of each camera included in the imaging unit 111 by measurement. The processing in step S201 and the processing in step S202 may be performed in parallel.

In step S203, if there is a camera among the cameras included in the imaging unit 111 in which the difference between the internal temperature acquired in step S202 and the internal temperature acquired in the previous step S202 is equal to or greater than the threshold value, the determining unit 113 determines that It is determined that the internal temperature of the camera has changed.

If there is one or more cameras determined that the internal temperature has changed, the process proceeds to step S204, and if there is no camera determined that the internal temperature has changed, the process proceeds to step S207. .

In step S204, the determining unit 113 acquires, from the storage unit 114, a correction parameter corresponding to the internal temperature range including the internal temperature of the camera for which the internal temperature has been determined to have changed.

In step S205, the application control unit 115 determines whether the setting conditions are satisfied. As a result of this determination, if the setting conditions are met, the process proceeds to step S206, and if the conditions are not met, the process proceeds to step S207.

In step S206, the application control unit 115 applies the correction parameters set for the camera for which it is determined that the internal temperature has changed to the correction parameters that the determination unit 113 acquired from the storage unit 114 in step S204 for the camera. Update to parameters.

In step S207, the distortion correction unit 116 corrects the captured images from each camera included in the imaging unit 111 using the correction parameters set for the cameras. The distortion correction unit 116 then transmits the captured images corrected for each camera included in the imaging unit 111 to the information processing device 100.

In step S208, the processing unit 101 generates a left eye virtual object image and a right eye virtual object image. The processing unit 101 then generates a composite image of the corrected image taken by the left eye camera and the left eye virtual object image (left eye combined image), a combined image of the corrected image taken by the right eye camera, and the right eye virtual object image. A composite image with the virtual object image for the right eye (composite image for the right eye) is generated. The processing unit 101 then outputs the composite images (the left eye composite image and the right eye composite image) generated for each camera to the display unit 117 of the HMD 110. As a result, the left-eye composite image is displayed on the left-eye display screen of the display unit 117, and the right-eye composite image is displayed on the right-eye display screen.

Note that in this embodiment, the HMD 110 and the information processing device 100 are separate devices, but the HMD 110 and the information processing device 100 may be integrated to form the HMD 110 having the functions of the information processing device 100. In that case, the HMD 110 generates a composite image of the image captured by itself and the image of the virtual object generated by itself, and causes the display unit 117 to display the composite image. Further, some functions of the information processing device 100 may be installed in the HMD 110.

Further, the correction parameters used by the distortion correction section 116 may be correction parameters using polynomial coefficients, and various correction parameters can be considered. Further, the purpose of correction performed using correction parameters is not limited to distortion correction or optical axis correction, and other correction targets may be corrected in addition to or in place of these.

Furthermore, in this embodiment, the correction target is a captured image obtained by performing deblocking processing on the captured image from the image sensor, but the correction target is not limited to this, and for example, the captured image from the image sensor (deblocking The captured image (before processing) may also be the correction target.

Note that among the functional units shown in FIG. 1, the distortion correction unit 116, the application control unit 115, the determining unit 113, and the processing unit 101 may be implemented by hardware or by a computer program. In the latter case, this computer program is executed by the processor of HMD 110, and as a result, the functions of the corresponding functional units are realized.

[Modification 1]
In each of the following modified examples including Modified Example 1, differences from the first embodiment will be explained, and unless otherwise mentioned below, it is assumed that the modified examples are the same as the first embodiment. In Modification 1, the setting condition is "the user is not looking into the display section 117 of the HMD 110". In this case, in order to detect whether the user is looking into the display section 117, the HMD 110 is provided with a proximity determination section such as a proximity sensor. For example, when the user wears the HMD 110 on his or her head, the proximity determination unit applies the fact that the user's eyes are close to the display unit 117 (the user is looking into the display unit 117 of the HMD 110). The control unit 115 is notified. For example, the proximity determination unit determines that when the user is not wearing the HMD 110 on his or her head, the user's eyes are not close to the display unit 117 (the user is not looking into the display unit 117 of the HMD 110). The application control unit 115 is notified of this fact.

The application control unit 115 determines that the setting condition is satisfied when the notification from the proximity determination unit is “the user is not looking at the display unit 117”. On the other hand, if the notification from the proximity determination unit is "the user is looking at the display unit 117", the application control unit 115 determines that the setting condition is not satisfied. According to such a configuration, it is possible to prevent the user from visually recognizing changes in the image due to switching of the correction parameters.

[Modification 2]
In the second modification, the setting condition is "the user is looking at the display section 117 of the HMD 110 in a blinking state". In this case, in order to detect where the user is looking on the display unit 117, the HMD 110 is provided with a line-of-sight determination unit such as a line-of-sight sensor that determines eye movement. The line of sight determination unit notifies the application control unit 115 of the position that the user is gazing at on the display screen of the display unit 117.

The application control unit 115 determines that the user is not blinking while the gaze position of the user is notified from the line of sight determination unit, and as a result, determines that the setting condition is not satisfied. On the other hand, if the gaze position of the user is not notified from the line-of-sight determination unit, the application control unit 115 determines that the user is blinking, and as a result, determines that the setting condition is satisfied. According to such a configuration, it is possible to prevent the user from visually recognizing changes in the image due to switching of the correction parameters.

[Other variations]
Note that among the above-mentioned functional units, the determination unit 113, the application control unit 115, and the distortion correction unit 116 may be processed by using a learned model that has been machine learned instead. In that case, for example, multiple combinations of input data and output data to the processing unit are prepared as learning data, knowledge is acquired from them through machine learning, and output data for the input data is determined based on the acquired knowledge. Generate a trained model that outputs as a result. The learned model can be configured, for example, by a neural network model. Then, the trained model is processed by the processing unit as a program for performing the same processing as the determination unit 113, application control unit 115, and distortion correction unit 116 by operating in cooperation with the CPU or GPU. Process. Note that the learned model may be updated after certain processing as necessary.

The specific numerical values used in the above explanation are for the purpose of providing a specific explanation, and are not intended to limit the above embodiments and modified examples to these numerical values. do not have. Furthermore, some or all of the embodiments and modifications described above may be combined as appropriate. Furthermore, some or all of the embodiments and modifications described above may be selectively used.

(Other embodiments)
The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

100: Information processing device 101: Processing unit 110: HMD 111: Imaging unit 112: Temperature acquisition unit 113: Determination unit 114: Storage unit 115: Application control unit 116: Distortion correction unit 117: Display unit

Claims (8)

For each of the plurality of image capture units, a correction unit that corrects an image captured by the image capture unit using a correction parameter set for the image capture unit;
and a control means for updating the correction parameters set for the imaging section of interest to correction parameters corresponding to the internal temperature of the imaging section of interest, if the setting conditions are met ;
If the setting condition is satisfied when the internal temperature of the imaging section of interest is out of the internal temperature range corresponding to the correction parameter set for the imaging section of interest, the control means updating the correction parameters set for the imaging unit of interest to correction parameters corresponding to the internal temperature of the imaging unit of interest;
The control means is configured to control whether the setting condition is satisfied even when the internal temperature of the imaging unit of interest is outside an internal temperature range corresponding to a correction parameter set for the imaging unit of interest. If not, the above update will not be performed.
A head-mounted display device characterized by: The setting condition is characterized in that the internal temperature of each of the plurality of imaging units is outside an internal temperature range corresponding to a correction parameter set for the imaging unit. Item 1. The head-mounted display device according to item 1. 2. The head-mounted display device according to claim 1, wherein the setting condition is that the user's eyes are not close to a display unit included in the head-mounted display device. The head-mounted display device according to claim 1, wherein the setting condition is that the user wearing the head-mounted display device is blinking. Furthermore,
The head-mounted type according to any one of claims 1 to 4 , further comprising storage means for storing, for each internal temperature range, a table in which correction parameters corresponding to the internal temperature range are registered. Display device. Furthermore,
Display means for displaying an image in a mixed reality space that is a composite image of an image of a virtual object generated based on the position and orientation of the head-mounted display device and a captured image corrected by the correction means. The head-mounted display device according to any one of claims 1 to 5 . A method for controlling a head-mounted display device, the method comprising:
a correction step in which the correction means of the head-mounted display device corrects, for each of the plurality of imaging units, an image captured by the imaging unit using a correction parameter set for the imaging unit;
If the setting condition is satisfied, the control means of the head-mounted display device updates a correction parameter set for the imaging section of interest to a correction parameter corresponding to an internal temperature of the imaging section of interest. Equipped with a control process and
In the control step, if the setting condition is satisfied when the internal temperature of the imaging unit of interest is out of the internal temperature range corresponding to the correction parameter set for the imaging unit of interest, updating the correction parameters set for the imaging unit of interest to correction parameters corresponding to the internal temperature of the imaging unit of interest;
In the control step, even if the internal temperature of the imaging unit of interest is outside an internal temperature range corresponding to a correction parameter set for the imaging unit of interest, the setting condition is satisfied. If not, the above update will not be performed.
A method for controlling a head-mounted display device, characterized in that: A computer program for causing a computer of a head-mounted display device to execute each step of the control method according to claim 7.

Images