JP6735557B2 - Image processing apparatus and image processing method - Google Patents

Description

The present invention relates to a technique for adjusting color balance between a plurality of cameras.

In recent years, mixed reality, so-called MR (Mixed Reality) technology is known as a technology for seamlessly fusing the real world and the virtual world in real time. As one of the MR technologies, a video see-through HMD (Head Mounted Display) is used to image a subject that substantially matches the subject observed from the pupil position of the HMD wearer with a video camera or the like, and the captured image is CG ( A technique is known in which an HMD wearer can observe an image in which Computer Graphics) are displayed in a superimposed manner.

Patent Document 1 discloses, as an MR system using a video see-through HMD, a first camera that captures a field of view of an HMD user, and a second camera that captures an image for detecting the position and orientation of the HMD. , Using HMD, is disclosed.

Patent Document 2 discloses, as an MR system using a video see-through HMD, a technique of matching brightness and tint between left and right picked-up images based on picked-up luminance information between a plurality of right-eye and left-eye cameras. ing.

JP, 2004-205711, A Patent No. 4522307

However, the techniques disclosed in the above patent documents have the following problems.

Patent Document 2 discloses performing unified WB (white balance) correction control using a common luminance value obtained by averaging luminance from a plurality of different luminance information captured by a plurality of cameras. ing. However, in the technique disclosed in Patent Document 2, if the characteristics, settings, etc. of a plurality of cameras exceed an allowable range, a unified WB correction control with a tint between the plurality of cameras is provided. Could not be done.

The same applies to Patent Document 1, and in a configuration including different types of cameras (imaging devices) as disclosed in Patent Document 1, a different image sensor or a brightness adjustment process according to the application for different applications is performed. However, there is a problem in that it is not possible to have a common brightness value among a plurality of cameras due to such reasons as described above, and it is not possible to perform unified WB correction control.

The present invention has been made in view of such a problem, and provides a technique for performing uniform color balance adjustment in which a plurality of cameras have different colors.

According to one aspect of the present invention, colors in each of a first captured image captured by a first imaging device and a second captured image captured by a second imaging device different from the first imaging device. An acquisition means for acquiring the temperature,
Determination means for determining an average color temperature of a part or all of the color temperatures acquired by the acquisition means as a color temperature common to the first image pickup device and the second image pickup device;
Control means for controlling color balance adjustment in the first captured image and the second captured image based on the color temperature determined by the determination means.

According to the configuration of the present invention, it is possible to perform uniform color balance adjustment that has a tint among a plurality of cameras.

9 is a flowchart of processing performed by the head-mounted display device 200. The figure which shows the structural example of MR system. The figure which shows the main structural examples in the head-mounted display device 200. The figure explaining the structure of the head mounted display device 200. FIG. 3 is a block diagram showing a more detailed configuration example of the image processing unit 113. FIG. The figure which shows the natural feature and its feature point in an image. The flowchart of the process of step S101-S104, S161-S164. The figure which shows the structural example of a color temperature detection table. The figure which shows the tint in color temperature. The figure which shows the structural example of the table referred in step S190. The figure which shows the structural example of the head mounted display apparatus 200 in the modification 1 of 1st Embodiment. The figure which shows the structural example of the head mounted display apparatus 200 in 2nd Embodiment. 9 is a flowchart of processing performed by the head-mounted display device 200. FIG. 6 is a diagram illustrating a phenomenon in which the detection accuracy of color temperature varies and a method of determining a priority camera. The color temperature detection tables 1501 and 1502 are overlaid on FIG.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the embodiment described below shows an example of the case where the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
Hereinafter, an example of the following image processing device will be described. The image processing apparatus acquires color temperatures of a first captured image captured by the first image capturing apparatus and a second captured image captured by a second image capturing apparatus different from the first image capturing apparatus. To do. Then, the image processing device determines a common color temperature between the first image pickup device and the second image pickup device based on the acquired color temperature, and based on the determined color temperature, the first color temperature is determined. The color balance adjustment in the captured image and the second captured image is controlled. In this example, the case where the image processing device is a head-mounted display device such as an HMD will be described. However, the application destination of the image processing device having such a configuration is not limited to the head-mounted display device. For example, the present invention may be applied to a stereoscopic display device that displays a left-eye image and a right-eye image in a stripe shape on one screen.

First, a configuration example of an MR system including the head-mounted display device according to this embodiment will be described with reference to FIG. As shown in FIG. 2, the MR system is a head-mounted display device 200 that provides an image of a mixed reality space, which is a space in which a real space and a virtual space (virtual object) are fused in front of the user, an image of the mixed reality space. And a cable 240 connecting the head-mounted display device 200 and the computer device 250. Although the cable 240 is shown as a wired communication path, a wireless communication path may be used instead.

Next, the configuration of the head-mounted display device 200 will be described in more detail. The head-mounted display device 200 includes a sub camera (a right-eye sub camera 21R and a left-eye sub camera 21L) that captures an image used to determine the position and orientation of the head-mounted display device 200, and a mixed reality space. A so-called video see-through head-mounted display device having a main camera (a right-eye main camera 20R and a left-eye main camera 20L) that captures an image of a real space to be combined with an image of a virtual space when an image is generated. is there.

The sub-camera is a camera for capturing an image of the marker 210 arranged in the physical space, and the computer device 250 executes a known process by using the image of the marker 210 captured by the sub-camera. The position and orientation of the component-mounted display device 200 can be calculated. More precisely, the position and orientation of the left-eye main camera 20L are calculated using the image captured by the left-eye sub camera 21L, and the position and orientation of the right-eye main camera 20R are calculated using the image captured by the right-eye sub camera 21R. (Derive). Then, the computer device 250 generates an image of the virtual space viewed from the position and orientation of the main camera based on the position and orientation of the main camera, and combines the generated image of the virtual space and the captured image of the real space by the main camera. By doing so, an image of the mixed reality space is generated. More precisely, an image of a virtual space viewed from the position and orientation of the left-eye main camera 20L is generated based on the position and orientation of the left-eye main camera 20L, and the generated virtual space image and the left-eye main camera 20L are used. An image of the mixed reality space to be provided to the user's left eye is generated by combining the image with the captured image of the real space. Further, an image of the virtual space viewed from the position and orientation of the right-eye main camera 20R is generated based on the position and orientation of the right-eye main camera 20R, and the generated virtual space image and the real-space of the right-eye main camera 20R are displayed. An image of the mixed reality space provided to the right eye of the user is generated by combining the captured image. Then, the computer device 250 sends the generated mixed reality space image (the image of the mixed reality space for the right eye and the image of the mixed reality space for the left eye) to the head-mounted display device 200.

Thus, the main camera and the sub camera have different uses. As shown in FIG. 2, since the main camera is intended to capture an image of the physical space provided to the user (image of the user's visual field), the image sensor used has a wide horizontal angle of view. An image sensor of a type (tentatively defined as X type) having characteristics such as low noise and wide color gamut is used. In this embodiment, a rolling shutter type is used as the X type. On the other hand, since the sub camera aims to capture an image used for obtaining the position and orientation, the sub camera has a wide angle of view in the vertical direction where the marker is present, and a dynamic body distortion of the marker called image drift does not occur. An image sensor of the shutter type (tentatively defined as Y type) is used.

The number of sub-cameras may be two or more in order to widen the imaging range and increase the imaging possibility of the marker, or to reduce the detection accuracy but reduce the cost, or vice versa. Only one high-performance camera with a wide angle of view may be used.

FIG. 3 shows a main configuration example of the head-mounted display device 200. Since there is a pair of configurations for the right eye and the left eye, only one is shown in FIG. The head-mounted display device 200 includes a display element 121 such as a small liquid crystal display for the right eye and a left eye, and a free-form surface prism for enlarging and displaying the images for the right eye and the left eye displayed on them. The display optical system 320, the main camera 20 for taking an image of a subject that substantially matches the subject observed from the position of the pupil 300 of the wearer of the head-mounted display device 200, the position of the pupil 300, and the position of the main camera 20. And an image pickup optical system 310 for making them substantially coincide with each other. The sub camera (not shown) is arranged outside the main camera 20 with respect to the face center of the wearer of the head-mounted display device 200.

Next, a hardware configuration example of the head-mounted display device 200 will be described with reference to the block diagram of FIG.

The right-eye main image pickup unit 100R is an image pickup unit including a right-eye main camera 20R, an optical system of the right-eye main camera 20R, and various processing circuits for the right-eye main camera 20R.

The left-eye main image pickup unit 100L is an image pickup unit including a left-eye main camera 20L, an optical system of the left-eye main camera 20L, and various processing circuits for the left-eye main camera 20L.

The right sub imaging unit 110R is an imaging unit that includes the right eye sub camera 21R, the optical system of the right eye sub camera 21R, and various processing circuits for the right eye sub camera 21R.

The left sub imaging unit 110L is an imaging unit including the left eye sub camera 21L, the optical system of the left eye sub camera 21L, and various processing circuits for the left eye sub camera 21L.

The right-eye display unit 120R is attached to the head-mounted display device 200 so as to be positioned in front of the right eye of the user who wears the head-mounted display device 200, and is generated on the computer device 250 side. The image of the mixed reality space for the right eye is displayed.

The left-eye display unit 120L is attached to the head-mounted display device 200 so as to be positioned in front of the left eye of the user who wears the head-mounted display device 200, and is generated on the computer device 250 side. The image of the mixed reality space for the left eye is displayed.

The MPU 130 executes processing using a computer program or data stored in the memory 131 in the head-mounted display device 200 to control the operation of each of the above functional units connected to the bus 190. At the same time, operation control of the entire head-mounted display device 200 is performed.

The memory 131 is a memory for storing a computer program or data for causing the MPU 130 to execute each process described later as what the MPU 130 executes, information described as known information in the following description, and the MPU 130 executes various processes. It includes various memories such as a memory having a work area used for execution.

FIG. 4B shows a configuration example of imaging units applicable to each of the right-eye main imaging unit 100R, the left-eye main imaging unit 100L, the right sub-imaging unit 110R, and the left sub-imaging unit 110L.

The image sensor 111 is a device such as a CCD image sensor that converts external light into an analog electric signal and outputs the analog electric signal. The image sensor 111 receives a signal from the TG (timing generator) 115 and the TG 115 and generates a vertical signal V-Dr. It is driven according to a signal from the (V driver) 116.

The A/D converter 112 converts the analog electric signal output from the image sensor 111 into a digital electric signal.

The image processing unit 113 performs various image processing on the digital electric signal converted by the A/D conversion unit 112 to generate and output a captured image that has been subjected to image processing. This image processing also includes color balance adjustment processing, which will be described later.

The picked-up image output unit 114 outputs the picked-up image subjected to the image processing by the image processing unit 113 to the computer device 250 in an appropriate image format. The output destination of the captured image output unit 114 is not limited to the computer device 250.

FIG. 4C shows a configuration example of display units applicable to each of the right-eye display unit 120R and the left-eye display unit 120L.

The display image input unit 123 receives the image of the mixed reality space output from the computer device 250, and transfers the received image of the mixed reality space to the display drive unit 122.

The display drive unit 122 drives the display element 121 to display the image of the mixed reality space transferred from the display image input unit 123.

The display element 121 is a display element such as p-SiTFT or LCOS, and is driven by the display driving unit 122 to display the image in the mixed reality space received by the display image input unit 123 from the computer device 250.

As described above, the computer device 250 calculates the position and orientation of the main camera using the image captured by the sub camera. More specifically, the marker is detected from the captured image by image analysis and the detected marker is detected. Acquire information such as the size and shape of, and the fill pattern. Then, the computer device 250 uses the information obtained by the marker detection to determine the relative positional relationship between the marker and the head-mounted display device 200, and the user who wears the head-mounted display device 200. 3D position/orientation information regarding the direction in which is observed. In this way, by using a plurality of markers and defining the positional relationship of each marker in advance as the marker placement information, it is possible to calculate the direction in which the marker is observed from the relative positional relationship. Becomes Therefore, it is possible to use a marker having one-dimensional information that does not have directional information, such as a color marker or a light emitting element such as an LED, instead of a marker that can identify the direction by an internal filling pattern. It will be possible. Further, instead of the marker 210 as shown in FIG. 2, natural features in the image such as the outlines of the door 600, the table 605, and the window 610 as shown in FIG. 6 and a specific color in the image are extracted. It is also possible to calculate three-dimensional position and orientation information using them. Reference numerals 650 and 655 denote "x" marks, which indicate some of the characteristic points of the door 600. By using a plurality of markers of the same type, using several types of markers at the same time, or combining the marker information and the information of the feature points in the image, it is possible to generate more accurate three-dimensional position/orientation information.

With the above configuration, the observer can experience the mixed reality space in which the real space and the virtual space are seamlessly fused in real time by mounting the head-mounted display device 200 on his/her head.

Next, a more detailed configuration example of the image processing unit 113 will be described with reference to the block diagram of FIG.

The color interpolating unit 501 includes adjacent pixels (adjacent pixels) in an image represented by the digital electric signal output from the A/D conversion unit 112 (each pixel is one of R, G, and B, for example, a Bayer pattern image). Pixels having different colors from each other are obtained (restored) from the R, G, and B pixel values of each pixel forming the captured image by an interpolation method or the like.

The WB detection unit 505 sets the entire captured image generated by the color interpolation unit 501 or a partial region thereof (for example, a partial region of a specified size centered on the center position of the captured image) as a target region, and an average pixel in the target region. Values (average pixel value of R, average pixel value of G, average pixel value of B) are obtained. For the average pixel value of R in the target area, the total value of the pixel values of the R components of all the pixels included in the target area is calculated, and the total value is divided by the number of pixels included in the target area. The result. This is the same when obtaining the average pixel value of G and the average pixel value of B. Then, the WB detection unit 505 sends the average pixel value of each of R, G, and B obtained for the target area to the MPU 130.

The WB correction unit 502 receives the WB correction value for R, the WB correction value for G, and the WB correction value for B, which the MPU 130 has determined based on the average pixel value of R and the average pixel value of B, and receives the color. By performing processing such as gain correction according to these WB correction values on the captured image generated by the interpolation unit 501, the achromatic color balance is adjusted, and thereby the WB correction for the captured image is realized. ..

The color correction unit 503 corrects the tint of the captured image whose color balance is adjusted by the WB correction unit 502, and the gamma correction unit 504 is the brightness level of the captured image whose color tint is corrected by the color correction unit 503. Correct the key.

Next, a process performed by the head-mounted display device 200 (mainly the image processing unit 113 and the MPU 130) will be described using FIG. 1 showing a flowchart of the process.

<Step S101>
The right-eye main image pickup unit 100R and the MPU 130 obtain the color temperature of the entire image captured by the right-eye main image pickup unit 100R or a partial region thereof by performing the processing according to the flowchart of FIG. 7A.

<Step S102>
The left-eye main image pickup unit 100L and the MPU 130 obtain the color temperature of the entire image captured by the left-eye main image pickup unit 100L or a partial region thereof by performing the process according to the flowchart of FIG. 7A.

<Step S103>
The right sub-imaging unit 110R and the MPU 130 obtain the color temperature of the entire image captured by the right sub-imaging unit 110R or a partial region thereof by performing the processing according to the flowchart of FIG. 7A.

<Step S104>
The left sub-imaging unit 110L and the MPU 130 obtain the color temperature of the entire image captured by the left sub-imaging unit 110L or a partial region thereof by performing the process according to the flowchart of FIG. 7A.

Hereinafter, a case will be described in which the process according to the flowchart of FIG. 7A is executed in step S101. In this case, the functional unit in the right-eye main image pickup unit 100R and the MPU 130 execute the process according to the flowchart of FIG. In addition, when performing the processing according to the flowchart of FIG. 7A in step S102, the processing according to the flowchart of FIG. 7A is performed by the functional unit in the left-eye main imaging unit 100L and the MPU 130. Become. Further, when the process according to the flowchart of FIG. 7A is executed in step S103, the functional unit in the right sub imaging unit 110R and the MPU 130 execute the process according to the flowchart of FIG. 7A. Become. Further, when the processing according to the flowchart of FIG. 7A is executed in step S104, the processing according to the flowchart of FIG. 7A is executed by the functional unit in the left sub imaging unit 110L and the MPU 130. Become.

<Step S180>
The WB detection unit 505 sets the entire captured image generated by the color interpolation unit 501 or a partial region thereof as a target region, and average pixel values (average pixel value of R, average pixel value of G, average of B) in the target region. Pixel value). Then, the WB detection unit 505 sends the average pixel value of each of R, G, and B obtained for the target area to the MPU 130.

<Step S181>
The MPU 130 obtains (average pixel value of B/average pixel value of R) as a WB ratio using the average pixel value of B and the average pixel value of R received from the WB detection unit 505.

<Step S182>
The MPU 130 acquires the color temperature corresponding to the WB ratio obtained in step S181 using the color temperature detection table created in advance for each imaging unit. FIG. 8 shows a configuration example of the color temperature detection table. In FIG. 8, the horizontal axis represents the WB ratio and the vertical axis represents the color temperature. Reference numeral 800 represents a color temperature detection table showing the correspondence relationship between the WB ratio and the color temperature corresponding to the image sensor (X type) used in the main camera, and 801 is used in the sub camera. The color temperature detection table showing the correspondence relationship between the WB ratio and the color temperature corresponding to the existing image sensor (Y type) is shown. Regarding the color temperature conversion table, the characteristic difference between each type of image sensor is larger than the individual difference between the same type of image sensor. Is assumed to be registered. That is, in steps S101 and S102, the common color temperature detection table 800 is used to acquire the color temperatures, respectively, and in steps S103 and S104, the common color temperature detection table 801 is used to acquire the color temperatures.

As described above, by executing the processing according to the flowchart of FIG. 7A in steps S101 to S104, the image captured by the right-eye main imaging unit 100R or the color temperature T1 in the partial area thereof, and the image captured by the left-eye main imaging unit 100L. It is possible to obtain the color temperature T2 in the image or the partial area thereof, the image temperature captured by the right sub imaging unit 110R or the color temperature T3 in the partial area thereof, and the image captured by the left sub imaging unit 110L or the color temperature T4 in the partial area thereof. ..

<Step S110>
The MPU 130 obtains the maximum difference (color temperature difference) ΔT among T1, T2, T3, and T4. This first obtains Tmax=MAX (T1, T2, T3, T4) and Tmin=MIN (T1, T2, T3, T4). Here, MAX (x1, x2,..., Xn) is a function that returns the maximum value of x1 to xn, and MIN (x1, x2,..., xn) is a function that returns the minimum value of x1 to xn. .. Then, ΔT=Tmax−Tmin is calculated.

For example, when T1=3500K, T2=3550K, T3=3650K, T4=3900K (the unit K of color temperature is Kelvin), color temperature Tmax=MAX(T1, T2, T3, T4)=3900K. , And the color temperature Tmin=MIN(T1, T2, T3, T4)=3500K, the color temperature difference ΔT=400K.

The cause of variations in the color temperature acquired from the images picked up by the image pickup units in this way will be described. First, as shown in FIG. 2, since the light input of the ambient light is different due to the different mounting positions of the image pickup units (20R, 20L, 21R, 21L), a difference occurs in the actual color temperature between the image pickup units. There is a case. Especially, the sub camera collects the ambient light different from the main camera by arranging it on the left and right outside of the main camera, widening the angle of view in the vertical direction, and shifting the center of the image in the vertical direction. It may happen. This is a mixture of various ambient lights such as light from ceiling lights, indirect illumination light in which light from ceiling lights is reflected on a wall, or sunlight entering from windows, and enters the imaging unit depending on the location and direction. This is because the light may be different. Another cause of variation in color temperature is the individual difference of the sensors in the image pickup unit.

<Step S120>
The MPU 130 determines whether or not the color temperature difference ΔT obtained in step S110 is within the allowable range ΔTerror (within the allowable range). For example, it is determined whether or not the absolute value of the color temperature difference ΔT is less than or equal to the allowable range ΔTerror.

The allowable range ΔTerror is a value such as 500K. The value of the permissible range ΔTerror is set in advance from a detection error caused by an individual difference of the image pickup sensor in the image pickup unit, a degree of entry of ambient light by a mounting position of the image pickup unit, and the like. The value of the permissible range ΔTerror is not limited to a fixed value, but may be a variable value because the allowable error varies depending on the average color temperature of all the image pickup units or the color temperature of a specific image pickup unit. It doesn't matter.

As a result of this determination, if the color temperature difference ΔT is within the allowable range ΔTerror, the process proceeds to step S130. If the color temperature difference ΔT is not within the allowable range ΔTerror, the process proceeds to step S140.

<Step S130>
The MPU 130 determines the average color temperature Tave of T1, T2, T3, T4 as a color temperature common to the right-eye main imaging unit 100R, the left-eye main imaging unit 100L, the right sub imaging unit 110R, and the left sub imaging unit 110L. The average color temperature Tave can be calculated according to the following formula.

Tave=AVE (T1, T2, T3, T4)
Here, AVE(x1, x2,..., Xn) is a function that returns the average value of x1 to xn. For example, when T1=3500K, T2=3550K, T3=3650K, and T4=3900K, the average color temperature Tave=3650K.

<Step S140>
The MPU 130 determines that the color temperature difference ΔT exceeds the allowable range ΔTerror, and it is difficult to treat the color temperatures of all the image pickup units equally, so that one of the T1, T2, T3, and T4 is set according to the standard for the right eye. The color temperature common to the image pickup unit 100R, the left-eye main image pickup unit 100L, the right sub image pickup unit 110R, and the left sub image pickup unit 110L is selected. In the present embodiment, the standard color temperature is determined in advance, and the color temperature closest to the standard color temperature among T1, T2, T3, and T4 is selected as the right eye main imaging unit 100R, the left eye main imaging unit 100L, and the right sub imaging unit 110R. , The left sub imaging unit 110L is selected as a common color temperature. For example, when the standard light source is D55, the standard color temperature is 5500K, and T1=3500K, T2=3550K, T3=3650K, and T4=3900K, the color temperature closest to the standard color temperature among T1, T2, T3, and T4. Is T4 (3900K), T4 is selected as a color temperature common to the right-eye main imaging unit 100R, the left-eye main imaging unit 100L, the right sub-imaging unit 110R, and the left sub-imaging unit 110L.

Here, the reason why the color temperature closest to the standard color temperature is selected will be described. First, the reason for providing standard ambient light is that it is often used for ambient light when the head-mounted display device 200 is used, because it is difficult to maintain the WB accuracy at any color temperature among a plurality of color temperatures. This is to set a color temperature that will probably be, and to design for the highest WB accuracy at that color temperature. Since the color balance when the WB correction is performed at the standard color temperature of 5500K serves as a design reference, after the WB correction, the WB state at the standard color temperature of 5500K is brought close to all in any color temperature environment. Means that.

For example, assuming that T1=3500K, T2=3550K, T3=3650K, and T4=3900K, all of the color temperatures have warmer-color-based values than the standard color temperature of 5500K, and the average color temperature Tave calculated in step S130. It can be seen from =3650K that the color is more yellow than red. FIG. 9 illustrates the tint at the color temperature. Here, if the average color temperature Tave=3650K is a correct value as the actual ambient light, after the WB correction, it exactly matches the WB condition at the standard color temperature of 5500K as intended. However, if the detected average color temperature Tave=3650K is a value that is deviated from red in comparison with the actual ambient light, WB correction with this is overcorrected, and as a result, it is deviated to a cold blue system. It becomes WB condition. At this time, when the WB-corrected image becomes a cold color system even though the actual ambient light is a warm color system, there is a problem that the correction error is recognized as a large human visual characteristic. Therefore, in order to prevent such overcorrection of WB correction, when the color temperature difference is large, a WB correction is performed by selecting a color temperature close to the standard color temperature of 5500K and performing WB correction. It can be set to a value that is close to the actual color of the ambient light and that is offset on the warm color side. For the above reason, even if the deviation amount of the WB condition at the standard color temperature of 5500K is larger than when the WB correction is overcorrected to become a cold color system, if the warm color system is the actual ambient light, Since it is recognized that the correction error is smaller, the processing aiming at this effect is performed in step S140. Although the effect and reason of step S140 have been described in the case where the detected color temperature is the warm color system, conversely, even in the case of the cold color system in which the detected color temperature is higher than the standard color temperature of 5500 K , the same reason is applied. Problems due to overcorrection can be avoided.

<Step S150>
The MPU 130 uses the color temperature determined in step S130 or the color temperature selected in step S140 as the common color temperature for the right-eye main imaging unit 100R, the left-eye main imaging unit 100L, the right sub-imaging unit 110R, and the left sub-imaging unit 110L. Set as (common color temperature).

Here, the reason why the common color temperature is set for all the image pickup units and the unified WB correction is performed will be described. As described above, the reason why the color temperatures of the plurality of image pickup units are different is that if different ambient light is detected due to the arrangement of the image pickup units, or if there is a detection error due to individual differences in the image pickup sensors, a common color temperature is set. If the WB correction is performed using the color temperature detected in each image pickup unit without using it, there arises a problem that the image picked up by each image pickup unit has a different tint. In particular, the head-mounted display device 200 is mounted in a case where WB detection is performed at a user's arbitrary timing called one push, instead of WB control called AWB that automatically detects WB at short fixed time intervals. When the user moves from the place where the WB is detected, there is a problem that the tint is different between the image pickup units and a great discomfort is given to the user. For the above reasons, it is necessary to perform uniform WB correction by using a common color temperature for all imaging units. Further, the reason why the color temperature is used for the uniform WB correction in all the image pickup units is to cope with the case where the characteristics of the luminance value are largely different among the image pickup units such as the type of the image pickup sensor. That is, by calculating the color temperature of the ambient light by applying the inverse characteristic to the luminance value having different characteristics for each image pickup unit in the detection of the ambient light, this color temperature becomes an index that can be commonly used by the image pickup units.

<Step S161>
The MPU 130 performs the process according to the flowchart of FIG. 7B to obtain the WB correction value used in the WB correction for the image captured by the right-eye main image pickup unit 100R using the common color temperature, and the obtained WB correction The value is sent to the right-eye main imaging unit 100R.

<Step S162>
The MPU 130 performs the process according to the flowchart of FIG. 7B to obtain the WB correction value used in the WB correction for the image captured by the left-eye main image pickup unit 100L by using the common color temperature, and the obtained WB correction The value is sent to the left-eye main imaging unit 100L.

<Step S163>
The MPU 130 performs the process according to the flowchart of FIG. 7B to obtain the WB correction value used in the WB correction for the image captured by the right sub imaging unit 110R using the common color temperature, and the obtained WB correction The value is sent to the right sub imaging unit 110R.

<Step S164>
The MPU 130 performs the process according to the flowchart of FIG. 7B to obtain the WB correction value used in the WB correction for the image captured by the left sub imaging unit 110L using the common color temperature, and the obtained WB correction The value is sent to the left sub imaging unit 110L.

Hereinafter, a case will be described where the process according to the flowchart of FIG. 7B is executed in step S161. In this case, the MPU 130 obtains a WB correction value for the right-eye main imaging unit 100R and sends it to the right-eye main imaging unit 100R. When executing the processing according to the flowchart of FIG. 7B in step S162, the MPU 130 obtains the WB correction value for the left-eye main imaging unit 100L and sends it to the left-eye main imaging unit 100L. Further, when the process according to the flowchart of FIG. 7B is executed in step S163, the MPU 130 obtains the WB correction value for the right sub imaging unit 110R and sends it to the right sub imaging unit 110R. Further, when executing the processing according to the flowchart of FIG. 7B in step S164, the MPU 130 obtains the WB correction value for the left sub imaging unit 110L and sends it to the left sub imaging unit 110L. S190>
The MPU 130 obtains a WB correction value for R, a WB correction value for G, and a WB correction value for B from the common color temperature. At that time, the MPU 130 refers to the table illustrated in FIG. In FIG. 10, the horizontal axis represents the color temperature and the vertical axis represents the correction gain value (WB correction value). The table in FIG. 10A corresponds to the image sensor (X type) used in the main camera, and the table in FIG. 10B is the image sensor (Y type) used in the sub camera. Is a table corresponding to. However, when executing the processing according to the flowchart of FIG. 7B in steps S161 and S162, the table of FIG. 10A is referred to, and the flowchart of FIG. 7B is executed in steps S163 and S164. When executing the processing according to, the table of FIG. 10B is referred to.

In FIGS. 10A and 10B, the RGB correction gain values are R/G (1010, 1050) and B/G (1020, 1060) with respect to the horizontal axis color temperature with G as a reference. Therefore, for example, when the color temperature T=3650K, the correction gain value in the image sensor (X type) is (R, G, B)=(0.9252, 1.0, from the table of FIG. 10A). 1.9633), and the correction gain value in the image sensor (Y type) is calculated as (R, G, B)=(0.7857, 1.0, 1.5678) from the table of FIG. To be done.

In this way, R/G and B/G corresponding to the common color temperature are specified with reference to the table of FIG. 10A or FIG. 10B, and R when G=1.0 is specified. , B values are specified. The values of R, G, and B thus specified become the WB correction value for R, the WB correction value for G, and the WB correction value for B, respectively.

<Step S192>
The MPU 130 outputs the R WB correction value, the G WB correction value, and the B WB correction value obtained in step S190 to the WB correction unit 502 included in the image processing unit 113 in the right-eye main imaging unit 100R. Send out. Thus, the WB correction unit 502 receives the WB correction value for R, the WB correction value for G, and the WB correction value for B received from the MPU 130, and with respect to the captured image generated by the color interpolation unit 501. By performing processing such as gain correction according to these WB correction values, the achromatic color balance is adjusted, and thereby WB correction for the captured image is realized.

In this way, the WB control process according to the present embodiment makes it possible to perform unified WB correction on the images captured by the main camera and the sub cameras.

In the present embodiment, in step S180, the average pixel value of each of R, G, and B is calculated and sent to the MPU 130, but in step S181, the average pixel value of B and the average pixel value of R are calculated. Since the processing is performed using, and the average pixel value of G is not used, the average pixel value of G does not have to be obtained.

Further, in the present embodiment, the case where the color components of the image are the R component, the G component, and the B component has been described, but other color components may be used. For example, the color components in the Lab color space may be the target. This embodiment can be similarly applied.

Further, in the present embodiment, the WB correction is performed on the image captured by each of the main camera and the sub camera, but the image provided to the user's eyes, that is, the image captured by the main camera is WB corrected, WB correction may not be performed on an image that is not provided to the user's eyes, that is, an image captured by the sub camera.

<Modification 1>
In the first embodiment, the head-mounted display device 200 includes two main cameras (a right-eye camera and a left-eye camera) for capturing a visual field of a user who wears the head-mounted display device 200 on his/her head. Camera) and one or more sub-cameras that capture images used to detect the position and orientation, and have a total of three or more cameras. However, as shown in FIG. 11, a camera 22R that also serves as the right eye main imaging unit 100R and a right sub imaging unit 110R, and a camera 22L that serves as both the left eye main imaging unit 100L and the left sub imaging unit 110L, It is also possible to have two cameras.

In this case, the image captured by the camera 22R is used as an image to be combined with the image in the virtual space when generating the image in the mixed reality space presented to the right eye of the user wearing the head-mounted display device 200. It is also used to detect the position and orientation of the camera 22R. The image captured by the camera 22L is used as an image to be combined with the image in the virtual space when generating an image in the mixed reality space presented to the left eye of the user wearing the head-mounted display device 200. , Is also used to detect the position and orientation of the camera 22L.

In the case of such a head-mounted display device 200, the color temperature common to the cameras 22R and 22L is acquired by the same process as the above process, and the color temperature for the camera 22R is acquired from the acquired common color temperature. The WB correction value and the WB correction value for the camera 22L are obtained, and the image captured by the camera 22R is WB corrected using the WB correction value for the camera 22R, and the image captured by the camera 22L is used the WB correction value for the camera 22L. WB correction can be performed.

<Modification 2>
In the first embodiment, the head-mounted display device 200 includes two main cameras (a right-eye camera and a left-eye camera) for capturing a visual field of a user who wears the head-mounted display device 200 on his/her head. (Camera) and two sub-cameras (right sub-camera and left sub-camera) for capturing an image used for detecting the position and orientation, and the averaging operation is performed assuming that the camera has a total of four eyes. The color temperature was calculated with a configuration in which different image sensor types are mixed. However, the common color temperature is calculated by dividing the combinations of the image sensors of the same type, and the calculated common color temperature for each type is used to make a common WB for each type of camera. It may be corrected. That is, the WB correction control of the present embodiment may be performed independently for each type of image sensor and between a plurality of cameras having the same type of image sensor.

<Modification 3>
In the first embodiment, if the color temperature difference ΔT is within the allowable range ΔTerror, the process proceeds to step S130. If the color temperature difference ΔT is not within the allowable range ΔTerror, the process proceeds to step S140. , And. However, if the color temperature difference ΔT is within the allowable range ΔTerror, the process proceeds to step S140, and if the color temperature difference ΔT is not within the allowable range ΔTerror, the process may proceed to step S130. By doing so, it is effective when the error in the WB correction becomes small.

<Modification 4>
In the first embodiment, in step S140, the color temperature closest to the standard color temperature among T1, T2, T3, and T4 is set to the right eye main imaging unit 100R, the left eye main imaging unit 100L, the right sub imaging unit 110R, the left sub imaging unit. The common color temperature is selected for the imaging unit 110L. However, in step S140, it suffices that any desired color temperature can be selected. Therefore, various methods are conceivable as the method of determining the color temperature common to each image pickup unit.

For example, among T1, T2, T3, and T4, the color temperature closest to the median of these may be selected. That is, any one may be selected based on the statistic obtained from T1, T2, T3, T4. Further, the color temperature having a predetermined value may be selected for the head-mounted display device 200.

<Modification 5>
In the first embodiment, in step S130, the average color temperature Tave of all (all) of T1, T2, T3, and T4 is set to the right eye main imaging unit 100R, the left eye main imaging unit 100L, the right sub imaging unit 110R, the left sub imaging unit. The color temperature is determined to be common to the imaging unit 110L. However, not all of T1, T2, T3, and T4, but an average color temperature of a part thereof may be set to Tave. For example, the average value of all T1, T2, T3, T4 is calculated, and the average value of the remaining color temperatures of T1, T2, T3, T4 excluding the color temperature having the largest difference from the average value is calculated. It may be Tave.

[Second Embodiment]
In the first embodiment, the head-mounted display device 200 has been described as having the image pickup unit groups having different types of image pickup sensors, but the image pickup sensor may have the same type of image pickup unit group. Absent. In such a configuration, the difference in the color temperature detection result between the image pickup units due to the individual difference of the image pickup sensor is a major problem. Therefore, in the present embodiment, WB correction control is performed to reduce the WB correction error between the image pickup units due to the individual difference of the image pickup sensor.

The differences from the first embodiment will be mainly described below, and the same as in the first embodiment unless otherwise stated below.

As shown in FIG. 12, the head-mounted display device 200 according to the present embodiment has a camera 23R that also serves as a right-eye main imaging unit 100R and a right sub-imaging unit 110R, and a left-eye main imaging unit 100L and a left eye. It has two cameras, a camera 23L which also serves as the application of the sub-imaging unit 110L, and both cameras are of the same type (Y type) as the image sensor.

In such a head-mounted display device 200, the process according to the flowchart of FIG. 13 is performed instead of the process according to the flowchart of FIG. In FIG. 13, the same process steps as those shown in FIG. 1 are designated by the same step numbers, and the description of the process steps will be omitted or briefly given.

<Step S201>
The camera 23R and the MPU 130 obtain the color temperature T1 of the entire image captured by the camera 23R or a partial region thereof by performing the process according to the flowchart of FIG. 7A.

<Step S202>
The camera 23L and the MPU 130 obtain the color temperature T2 of the entire image captured by the camera 23L or a partial region thereof by performing the process according to the flowchart of FIG. 7A.

In step S110, in the first embodiment, four color temperatures (T1, T2, T3, T4) were targeted, whereas in the present embodiment, there are two color temperatures T1 and T2, so ΔT is simply It is obtained by calculating ΔT=|T2-T1|. For example, when the color temperature T1 of the camera 23R is 3600K and the color temperature T2 of the camera 23L is T2 = 3950K, the color temperature difference ΔT is 350K. As described above, the causes of the difference in color temperature between the left and right cameras are individual differences among the cameras due to variations in the performance of the image sensor and alignment (mechanical tolerance) between the image sensor and the camera lens.

If the color temperature difference ΔT is within the allowable range ΔTerr at step S120, the process proceeds to step S130 as in the first embodiment, but if the color temperature difference ΔT is not within the allowable range ΔTerror. Processing proceeds to step S240.

<Step S240>
The MPU 130 selects the color temperature acquired for one of the cameras 23R and 23L, which is predetermined as a priority camera that is considered to have higher color temperature detection accuracy.

Here, a phenomenon in which the detection accuracy of color temperature varies and a method of determining a priority camera will be described with reference to FIG. FIG. 14A is a diagram showing data created to design the color temperature detection table of the image sensor (Y type). 1401 to 1403 are average pixel values of R obtained by WB detection (processing in step S180) by one camera having an image sensor (Y type) when the color temperature of ambient light is changed from 2500 to 9500K ( It is a curve showing the transition of the average brightness value), the average pixel value of G, and the average pixel value of B. Here, when similar data is created using another camera equipped with a Y-type image sensor, the data does not become exactly the same as in FIG. This is the individual difference of the camera.

FIG. 14B shows a color temperature detection table 801 in which the WB ratio is plotted on the horizontal axis and the color temperature is plotted on the vertical axis. In order to take into account individual differences among cameras, a plurality of cameras (all mounted) are shown. The image sensor is prepared from the data of FIG. 14A created for each Y type). For example, the average pixel value of R and B in the entire camera is calculated from the average pixel value of R and B in each camera, and the WB ratio (average pixel value of B in the entire camera/R of the entire camera) is calculated for each color temperature. The color temperature detection table 801 can be created by calculating the average pixel value. Of course, other methods can be considered as the method of creating the color temperature detection table 801. The error bar in the figure indicates the range of variation of the maximum and minimum WB ratios in the measured data.

In this embodiment, the color temperature detection table 801 is used when acquiring the color temperature of each of the cameras 23R and 23L. At this time, the WB ratios of the cameras 23R and 23L vary within this error bar range. Therefore, for example, in the case of a camera combination having a variation in which the camera 23R is the maximum and the camera 23L is the minimum at a certain color temperature, acquisition is made for each camera. The difference in color temperature is the worst. As a method to fundamentally solve this problem, it is conceivable to set the color temperature detection table that is accurately adjusted for each individual by measuring the total number when assembling the camera. Since it takes a lot of man-hours, it is not realistic to measure all of them at the time of assembly. However, it is realistic to measure the color temperature of a specific ambient light when assembling the camera, for example, under only one condition of the standard color temperature of 5500K. Therefore, it is possible to estimate to some extent how much the measured assembled camera deviates from the representative color temperature detection table 801 in the +/− direction.

FIG. 15 is an accurate color temperature detection table 1501 of the camera 23R when the color temperature of the entire range is shaken, and an accurate color temperature detection of the camera 23L when the color temperature of the entire range is shaken, as shown in FIG. The table 1502 is stacked. As described above, it is troublesome to create the color temperature detection tables 1501 and 1502 by shaking the color temperature of the entire range, but to obtain the WB ratio at the standard color temperature of 5500K in this example is to obtain the color range of the entire range. It is less troublesome than creating the color temperature detection tables 1501 and 1502 by changing the temperature. Therefore, when the standard color temperature is set to 5500K, the WB ratio of each of the cameras 23R and 23L at the color temperature=5500K is obtained, and the WB ratio closer to the WB ratio corresponding to 5500K in the color temperature detection table 801 is obtained. Is determined as the above-mentioned priority camera. In the case of FIG. 15, the WB ratio corresponding to 5500K in the color temperature detection table 1501 is smaller than the WB ratio corresponding to 5500K in the color temperature detection table 1502 by the WB ratio corresponding to 5500K in the color temperature detection table 801. Since it is close, the camera 23R is determined as the priority camera.

The process of determining the priority camera of the cameras 23R and 23L described above is performed before the head-mounted display device 200 is shipped, and the information indicating the determined priority camera is the head-mounted display device 200. It will be registered in the internal memory 131.

Further, the above description is for the case where the standard color temperature is 5500K, but if the color temperature used as the standard color temperature changes, the target to be determined as the priority camera may also change. However, if the color temperature to be used as the standard color temperature is determined in consideration of the environment in which the head-mounted display device 200 is used, a desirable camera corresponding to the color temperature can be determined as the priority camera. ..

Further, instead of simply setting the camera closest to the WB ratio of the color temperature detection table 801 in the standard color temperature as the priority camera, the average of the color temperature detection table 801 in the vicinity of the standard color temperature or higher is displayed. The camera closest to the WB ratio may be determined as the priority camera. That is, when the color temperature in the environment in which the head-mounted display device 200 is used is assumed, a camera that can obtain a desired WB ratio at the assumed color temperature (closer to the color temperature detection table 801 in the above example) is the priority camera. If it can be determined as, there are various possible methods for the determination.

As a result, even if the camera has a large deviation from the typical color temperature detection table 801, by assembling the head-mounted display device 200 in combination with a better camera, relatively stable WB control is achieved. It becomes possible to do.

Then, in step S150, the MPU 130 sets the color temperature determined in step S130 or the color temperature selected in step S240 as the color temperature common to the cameras 23R and 23L (common color temperature).

<Step S261>
The MPU 130 performs the process according to the flowchart of FIG. 7B to obtain the WB correction value used in the WB correction for the image captured by the camera 23R using the common color temperature, and the obtained WB correction value is used in the camera. Send to 23R.

<Step S262>
The MPU 130 performs the process according to the flowchart of FIG. 7B to obtain the WB correction value used in the WB correction for the image captured by the camera 23L using the common color temperature, and the obtained WB correction value is used in the camera. Send to 23L.

As described above, according to the present embodiment, it is possible to perform the WB control for reducing the WB correction error between the cameras due to the individual difference of the image sensor.

<Modification 1>
In the second embodiment, the head-mounted display device 200 has the same type of image sensor, but the head-mounted display device 200 has the same type of image sensor or a different type of image sensor. For example, the WB control of the second embodiment is used between cameras having the same type of image sensor and the WB control of the first embodiment is used between cameras having different types of image sensors. The above embodiments may be used in combination.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

130: MPU 100R: Right eye main imaging unit 100L: Left eye main imaging unit 110R: Right sub imaging unit 110L: Left sub imaging unit

Claims (17)

An acquisition unit that acquires the color temperature of each of the first captured image captured by the first imaging device and the second captured image captured by the second imaging device different from the first imaging device. ,
Determination means for determining an average color temperature of a part or all of the color temperatures acquired by the acquisition means as a color temperature common to the first image pickup device and the second image pickup device;
An image processing apparatus comprising: a control unit that controls color balance adjustment in the first captured image and the second captured image based on the color temperature determined by the determination unit. The determining means is
For each of the first captured image and the second captured image, the difference between the maximum color temperature and the minimum color temperature of the color temperatures acquired by the acquisition unit is calculated, and whether the difference is within an allowable range. The image processing apparatus according to claim 1, wherein the common color temperature is determined based on the. The image processing apparatus according to claim 2 , wherein a process of determining the common color temperature by the determining unit is different depending on whether the difference is within an allowable range or not. The determining means is
When the difference is within the allowable range, an average color temperature of a part or all of the color temperatures acquired by the acquisition unit for each of the first captured image and the second captured image is set to the common color. Decided as temperature,
If the difference is not within the allowable range, a part of the color temperature acquired by the acquisition unit for each of the first captured image and the second captured image is partially or entirely set as a standard color temperature in advance. The image processing apparatus according to claim 3 , wherein the color temperature closest to the set color temperature is determined as the common color temperature. The determining means is
When the difference is within the allowable range, an average color temperature of a part or all of the color temperatures acquired by the acquisition unit for each of the first captured image and the second captured image is set to the common color. Decided as temperature,
If the difference is not within the allowable range, one color temperature obtained from the color temperatures acquired by the acquisition unit for each of the first captured image and the second captured image is determined as the common color temperature. The image processing device according to claim 3 , wherein One color temperature obtained from the color temperature acquired by the acquisition unit for each of the first captured image and the second captured image is the same for each of the first captured image and the second captured image. The image processing apparatus according to claim 5 , wherein the color temperature is determined based on the color temperature statistic acquired by the acquisition unit. The determining means is
If the difference is within the allowable range, the average color temperature of a part or all of the color temperature acquired by the acquisition unit is determined as the common color temperature,
When the difference is not within the allowable range, the color temperature acquired by the acquisition unit is the same as the color temperature acquired by the acquisition unit with respect to the imaged image captured by a predetermined image capturing device of the first image capturing device and the second image capturing device. The image processing apparatus according to claim 3 , wherein the color temperature is determined as The average color temperature of a part of the color temperatures acquired by the acquisition unit for each of the first captured image and the second captured image is the average color temperature for each of the first captured image and the second captured image. The image processing apparatus according to claim 4 , wherein the color temperature acquired by the acquisition unit is the average color temperature of the remaining color temperatures obtained by removing the color temperature having the largest difference from the average color temperature of the color temperatures. The acquisition unit obtains an average pixel value for each color component of the captured image for each of the first captured image and the second captured image, and uses the average pixel value to determine a color temperature in the captured image. the image processing apparatus according to any one of claims 1 to 8, characterized in that to obtain. The acquisition unit obtains the average pixel value of the R component and the average pixel value of the B component of the captured image for each of the first captured image and the second captured image, and (average pixel value of the B component). The image processing apparatus according to claim 9 , wherein the color temperature corresponding to /(average pixel value of R component) is acquired. The image processing apparatus according to any one of claims 1 to 10, characterized in that said a first head-mounted display device having the imaging device and the second imaging device display and Image processing device. Further, a derivation unit that derives the position and orientation of the first image pickup device,
Any one of claims 1 to 11, characterized in that it comprises a synthesizing means for synthesizing the first captured image a virtual object to be produced on the basis of the position and orientation of said derived first imaging device The image processing device according to 1. The image processing apparatus according to claim 12 , wherein the deriving unit derives the position and orientation of the first imaging device based on the second captured image. The type of sensor of the first imaging apparatus, an image processing apparatus according to any one of claims 1 to 13, characterized in that different from the types of sensors of said second imaging device. 15. The image processing apparatus according to claim 14 , wherein the sensor of the first imaging device is a rolling shutter system, and the sensor of the second imaging device is a global shutter system. The color temperature in each of the first captured image captured by the first captured image and the second captured image captured by the second captured image different from the first captured image is acquired,
An average color temperature of a part or all of the acquired color temperatures is determined as a common color temperature between the first imaging device and the second imaging device,
An image processing method, comprising: controlling color balance adjustment in the first captured image and the second captured image based on the determined color temperature. A computer program for a computer to function as each means of the image processing apparatus according to any one of claims 1 to 15.