JP4633023B2 - User interface device - Google Patents

Description

  The present invention relates to a user interface device that synthesizes and displays an image of an operator's hand photographed by a camera on an operation reception screen displayed on a display device.

2. Description of the Related Art Conventionally, there is known a user interface device that synthesizes and displays an operator's hand image captured by a camera on an operation reception screen displayed on a display device (for example, Patent Document 1 and Patent Document 2).
According to these techniques, the operator can perform various operations at hand as if he / she is operating his / her hand on the operation reception screen displayed on the display device.
Japanese Patent Laid-Open No. 10-269102 JP 2000-6687

  In a conventional user interface device that displays an image of an operator's hand photographed with a camera on an operation reception screen displayed on a display device, a normal camera is used for photographing an operator's hand. In an environment where the ambient light is dark, the operator's hand cannot be photographed well, and the hand cannot be displayed so that the operator can clearly see it. On the other hand, if an infrared camera is used to photograph the operator's hand, the hand can be displayed so that the operator can clearly see even in a dark environment such as in the nighttime in a car. Since the image of the hand is different from the shape of the hand familiar to the operator with visible light, there is a problem that the image becomes unnatural.

  Therefore, the present invention provides a user interface device that composites and displays an image of an operator's hand taken by a camera on an operation reception screen displayed on a display device, and can be clearly seen even in an environment where ambient light is dark, and It is an object to display an operator's hand in a natural form.

  In order to achieve the above object, the present invention provides a user interface device that synthesizes and displays an image of an operator's hand photographed by a camera on an operation reception screen displayed on a display device, and the operator's hand displays visible light. A camera that is photographed by using the camera, an infrared camera that photographs the hand of the operator using infrared light from the same photographing direction as the camera, and the operator photographed by the camera from an image photographed by the camera. An image of the operator's hand taken by the infrared camera from an image taken by the infrared camera, and larger than the hand image. Hand image generating means for generating an image representing the operator's hand at an enlargement ratio as an infrared hand image; a hand image on the infrared hand image; and a peripheral portion of the infrared hand image borders the hand image The synthesized image is Is constructed by including a display hand image output means for outputting as an image to be synthesized and displayed on the screen.

  According to such a user interface device, an image of an operator's hand photographed using visible light and a composite image obtained by fringing the periphery of the operator's hand photographed using infrared light are accepted. Output as a composite image on the screen. Here, even when the ambient light is dark and the operator's hand cannot be clearly imaged by visible light, the infrared light can capture the operator's hand relatively clearly. Therefore, according to this user interface device, even when the ambient light becomes dark, the contour of the image taken with the visible light of the displayed operator's hand is kept clear, and the visibility of the displayed hand image is low. As compared with the case where only the image of the hand photographed with visible light is displayed, it is maintained better. In addition, since the operator's hand is displayed using an image of a hand photographed using visible light, the hand expression by the display becomes natural and realistic.

  Here, such a user interface device is configured such that when the display hand image output means cannot capture the operator's hand clearly with the camera, the infrared hand image or the brightness of the infrared hand image is obtained. An image with a lowered level may be output as an image to be displayed on the operation acceptance screen.

In this way, if the ambient light is so dark that the hand can only be photographed with some visible light, the hand captured with infrared light can be displayed, so that even in such a case, the hand can be seen well. The operator's hand can be displayed on the screen.
Further, in the user interface device as described above, the hand image generation means estimates that there is a region of the operator's hand that could not be captured by the camera, based on a region of the hand obtained from the infrared hand image. In this case, the hand image may be generated by complementing an image representing the region of the hand to the image of the operator's hand taken by the camera.

In this way, even when an image that lacks part of the operator's hand is captured by a camera that uses visible light, the operation is performed in a more natural form that complements the missing part. An image representing a person's hand can be displayed.
The user interface device as described above may be mounted on an automobile.

  As described above, according to the present invention, in a user interface device that composites and displays an image of an operator's hand photographed by a camera on an operation reception screen displayed on a display device, it is clear even in an environment where ambient light is dark. The operator's hand can be displayed in a natural form.

Hereinafter, an embodiment of the present invention will be described taking application to an in-vehicle system mounted on an automobile as an example.
FIG. 1 a shows the configuration of the in-vehicle system according to this embodiment.
This in-vehicle system is a system mounted on an automobile, and as shown in the figure, a camera 1 that captures visible light, an infrared camera 2 that captures infrared light, an input panel 3, a hand image extraction interpolation unit 4, an infrared A hand image extraction unit 5, a hand image composition unit 6, a composition control unit 7, a processing device 8, an output image composition unit 9, and a display device 10 are provided.

  Here, as shown in FIG. 1b, the display device 10 is arranged in an instrumental panel in front of the driver so that the display surface faces the driver. The input panel 3 is a touch panel, for example, and is disposed in the center console next to the driver with the operation surface facing upward. The operation surface of the touch panel is colored black, for example. The camera 1 and the infrared camera 2 are arranged on the ceiling above the driver so as to photograph the driver's hand placed on the operation surface.

  Now, in such a configuration, the hand image extraction / interpolation unit 4 uses the discrimination method using the chroma key technique, the luminance threshold value, or the like in the image captured by the camera 1 in which the driver's hand is reflected. Is identified as a hand image region by the discrimination method using the, and hand image region information representing the shape and position of the hand image region is output to the synthesis control unit 7. Also, the image portion in the hand image area of the image taken by the camera 1 is cut out as a hand image 31 as shown in FIG. 3 a and output to the hand image composition unit 6. The hand image extraction / interpolation unit 4 also performs interpolation processing of the hand image 31 in accordance with an instruction from the composition control unit 7. Details of this interpolation processing will be described later.

  On the other hand, the infrared hand image extraction unit 5 uses a discrimination method using a chroma key technique or a luminance threshold value for an area where a driver's hand is reflected in an image taken by the infrared camera 2. An infrared hand image area is identified by the discrimination method, and infrared hand image area information representing the shape and position of the infrared hand image area is output to the synthesis control unit 7. Further, the image portion in the hand image area of the image taken by the infrared camera 2 is cut out as an infrared hand image 32 as shown in FIG. Here, the sensitivity of the infrared camera 2 is set so that a relatively high-intensity infrared hand image 32 is obtained with respect to the infrared intensity obtained for a standard human hand. Further, the infrared hand image 32 output in this way is set to be slightly larger than the hand image 31 cut out by the hand image extraction interpolation unit 4 from the image taken by the camera 1. This is because the infrared hand image 32 can be made slightly larger than the hand image 31 by setting the scale of the hand of the driver of the infrared camera 2 larger than that of the camera 1. Then, the infrared hand image extraction unit 5 enlarges the image portion in the hand image area of the image photographed by the infrared camera 2 at a predetermined enlargement ratio to obtain the infrared hand image 32, whereby the infrared hand image is obtained. It is also possible to make 32 larger than the hand image 31.

Next, the composition control unit 7 performs composition control processing.
FIG. 2 shows the procedure of this synthesis control process.
As shown in the figure, this process waits for the camera 1 and the infrared camera 2 to capture a new image (step 202). If the camera 1 captures a new image, it is determined whether it is the current night. Check (step 204). Whether it is the night or not is determined by the output of an optical sensor that detects the brightness of ambient light provided separately, the lighting state of the headlight of the vehicle, and the brightness of the hand image 31 output by the hand image extraction interpolation unit 4 Investigate based on level.

  If it is determined that it is not night (step 206), the hand image area indicated by the hand image area information output from the hand image extraction interpolation unit 4 and the infrared hand output from the infrared hand image extraction unit 5 An imaging missing area that is an area that has not been extracted as a hand image area even though the hand should be reflected in an image taken by the camera 1 from the infrared hand image area indicated by the image area information Is calculated (step 208).

  Then, it is checked whether or not there is a shooting missing region (step 210). If there is no shooting missing region, the hand image is displayed on the hand image compositing unit 6 on the infrared hand image 32 output from the infrared hand image extracting unit 5. The output interpolating unit 4 instructs the output image synthesizing unit 9 to output the composite image on which the hand image 31 output by the extraction interpolation unit 4 is output as an output image (step 214), and returns to the processing from step 202. Here, the hand image synthesizing unit 6 that has received this instruction, for example, the hand image extracting / interpolating unit 4 on the infrared hand image 32 output from the infrared hand image extracting unit 5 as shown in FIG. A composite image 33 is generated by combining the output hand image 31 so that the peripheral portion of the infrared hand image 32 borders the hand image 31, and the composite image 33 is output as the output image 34 shown in FIG. 3b2. The image is output to the image composition unit 9.

  Now, returning to FIG. 2, if it is determined in step 206 that the present is night, the brightness level of the infrared hand image 32 is reduced to the hand image composition unit 6 and output as the output image 34. The image composition unit 9 is instructed to output (step 216), and the process returns to step 202. Upon receiving this instruction, the hand image synthesizing unit 6 converts the luminance level of the infrared hand image 32 output from the infrared hand image extracting unit 5 into a low luminance level as shown in FIG. The output image 34 is output to the output image composition unit 9.

  As a result of the above processing, the output image 34 output from the hand image synthesis unit 6 is as shown in FIG. FIG. 3b shows changes in the output images b1, b2,..., B6 accompanying changes in ambient light from day to night. As shown in the drawing, as the ambient light gradually becomes darker, the portion of the hand image 31 in the output image 34 becomes gradually darker and unclear as shown in b1 to b5. However, the brightness of the portion of the infrared hand image 32 that borders the portion of the hand image 31 in the output image 34 is kept high, so even if the ambient light becomes dark, the hand image in the output image 34 The contour of 31 is kept clear, and the visibility of the hand represented by the output image 34 is maintained more clearly than when only the hand image 31 is used as the output image 34. Further, since the driver's hand is represented using the hand image 31 obtained by photographing the hand with the camera 1 in the output image 34, the output image 34 is a realistic representation of the hand in a natural manner.

  When the ambient light is reduced to such an extent that the camera 1 can only capture a hand image, the output image 34 is obtained by reducing the brightness of the infrared hand image 32 as shown in b6. In such a case, the visibility of the hand represented by the output image 34 is kept clear. In addition, by reducing the luminance of the infrared hand image 32 to be the output image 34, the output image 34 is less unnatural than when the infrared hand image 32 is directly used as the output image 34. can do. However, if it is determined in step 206 that the present is night, in step 216, the hand image composition unit 6 is caused to output the infrared hand image 32 as it is as the output image 34 to the output image composition unit 9. It may be.

Returning to FIG. 2, if it is determined that there is a shooting missing area (step 210), the hand image extraction interpolation unit 4 is instructed to interpolate the shooting missing area (step 212), and the above-described step 214 is performed. Proceed to the process.
Here, the hand image extraction / interpolation unit 4 that has received an instruction to interpolate the shooting missing region outputs the hand image 31 to the hand image combining unit 6 after performing the interpolation processing of the shooting missing region.
That is, for example, as shown in FIG. 4a, when a hand image 31 is extracted from an image captured by the camera 1 and an infrared hand image 32 is extracted from an image captured by the infrared camera 2, both images are extracted. As shown in the figure, the synthesized composite image 33 is unnatural with a part of the hand interrupted.

  Therefore, in the case where there is a shooting missing area, the composition control unit 7 determines the infrared hand image area obtained by combining the scales of both areas based on the hand image area information and the infrared hand image area information. An area that is not a hand image area is obtained as a shooting missing area 311 as shown in FIG. 4b, and the hand image extraction interpolation unit 4 is instructed to interpolate the shooting missing area 311. In the unit 4, the pixel value of the pixel in the shooting missing region 311 in the hand image 31 is interpolated as shown in FIGS. 4 c 1 and c 2, and the hand image 31 is output to the image composition unit.

  Here, FIG. 4 c 1 is an example in which the pixel value of the shooting missing area 311 is interpolated by linear interpolation of the pixel values of the hand image 31 around the shooting missing area 311, and FIG. 4 c 2 is a luminance gradient in the shooting missing area 311. Using the brightness gradient according to the brightness gradient in the corresponding region of the infrared image, the pixel value of the shooting missing region 311 of the hand image 31 is smoothed and the pixel value of the shooting missing region 311 of the hand image 31 and the surrounding pixel values are smooth. This is an example of interpolation so as to be continuous.

As a result, the synthesized image 33 obtained by synthesizing the hand image 31 and the infrared hand image 32 by the hand image synthesizing unit 6 is as shown in FIGS. 4d1 and d2, and is more unnatural than the synthesized image 33 shown in FIG. 4a. It will be reduced.
However, the processing in steps 208 to 212 is not performed, and when the shooting missing region 311 occurs, the generated composite image 33 as shown in FIG. Even in this case, compared to the case where the hand image 31 is used as it is as the output image 34, the output image 34 with reduced unnaturalness can be obtained.

The combination control process performed by the combination control unit 7 has been described above.
Now, returning to FIG. 1, the processing device 8 outputs a menu screen for accepting a driver's operation via the input panel 3 to the output image composition unit 9 and performs processing according to the driver's operation.
On the other hand, the output image synthesizing unit 9 synthesizes the output image 34 output from the hand image synthesizing unit 6 and displays it on the display device 10 on the menu screen.
Here, FIG. 5 shows a display example of the display device 10 performed in this way.
FIG. 5 a is an example in which the menu screen 50 and the output image 34 are synthesized by superimposing and displaying the output image 34 representing the driver's hand on the menu screen 50, and FIG. This is an example in which the menu screen 50 and the output image 34 are combined by translucently displaying the output image 34 representing the driver's hand and displaying it. Further, FIG. 5C shows the menu screen 50 and the output image 34 by displaying a menu screen 50 that is transparent except for the outlines of menu characters and menu buttons on the output image 34 representing the driver's hand. This is an example of synthesis.

Each position on the input panel 3 is associated with each position on the menu screen. The output image 34 and the menu screen are related to each other in the positional relationship between the output image 34 and the menu screen. To be matched with the positional relationship between the position on the input panel 3 and the position on the menu screen.
The embodiment of the present invention has been described above.

It is a figure showing composition and arrangement of an in-vehicle system concerning an embodiment of the present invention. It is a flowchart which shows the synthetic | combination control process which concerns on embodiment of this invention. It is a figure which shows the process example of the synthetic | combination control process which concerns on embodiment of this invention. It is a figure which shows the process example of the synthetic | combination control process which concerns on embodiment of this invention. It is a figure which shows the example of a display of the vehicle-mounted system which concerns on embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Infrared camera, 3 ... Input panel, 4 ... Hand image extraction interpolation part, 5 ... Infrared hand image extraction part, 6 ... Hand image composition part, 7 ... Composition control part, 8 ... Processing apparatus, DESCRIPTION OF SYMBOLS 9 ... Output image synthetic | combination part, 10 ... Display apparatus, 31 ... Hand image, 32 ... Infrared hand image, 33 ... Composite image, 34 ... Output image, 311 ... Image | photographing missing area.

Claims (5)

A user interface device that synthesizes and displays an image of an operator's hand photographed by a camera on an operation acceptance screen displayed on a display device,
A camera for photographing the operator's hand using visible light;
An infrared camera for photographing the operator's hand using infrared light from the same photographing direction as the camera;
Hand image generating means for generating, as a hand image, an image of the operator's hand photographed by the camera from an image photographed by the camera;
An image of the operator's hand photographed by the infrared camera from an image photographed by the infrared camera, and representing an image of the operator's hand with a larger magnification than the hand image. Hand image generating means for generating as
Display hand image output that outputs a composite image obtained by combining a hand image on the infrared hand image and a peripheral image of the infrared hand image so that the hand image borders the hand image on the operation reception screen. A user interface device. The user interface device according to claim 1,
The display hand image output means displays the infrared hand image or an image obtained by reducing the brightness level of the infrared hand image on the operation reception screen when the operator's hand cannot be clearly photographed by the camera. A user interface device characterized in that the image is output as an image to be synthesized and displayed. The user interface device according to claim 1 or 2,
The hand image generation means captures an image of the operator's hand that was not captured by the camera based on the hand region determined from the infrared hand image. A user interface device, wherein the hand image is generated by complementing an image representing an area of the hand to the image of the operator's hand. The user interface device according to claim 1, 2, or 3,
The user interface device is mounted on an automobile. A display hand image generation method for generating a display hand image representing a user's hand,
Photographing the user's hand with visible light using a camera;
Photographing the user's hand using infrared light from the same photographing direction as the camera using an infrared camera;
Generating an image of the user's hand captured by the camera as a hand image from an image captured by the camera;
From the image captured by the infrared camera, an image of the user's hand captured by the infrared camera, which represents the user's hand at a larger magnification than the hand image, is generated as an infrared hand image. And steps to
Generating the display hand image by synthesizing the hand image on the infrared hand image so that the peripheral portion of the infrared hand image borders the hand image. Method.