JP2003316510A - Display device for displaying point instructed on display screen and display program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a display device and display program capable of highly precisely detecting the movement of an indicated object, and efficiently displaying an indicated point on a display screen. <P>SOLUTION: The display device for displaying an indicated point on a display screen measured by a stereo camera configured of an at least two-lens camera based on the position of an indicated object indicated on the display screen is provided with a detecting means for detecting the distance of the indicated object from each of stereo cameras arranged in the periphery of the display screen and a selecting means for selecting the stereo camera whose distance from the indicated object is the shortest. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a display device and a display program for displaying an instructed point on a display screen, and more particularly, to an operator using a stereo camera composed of two or more cameras. The present invention relates to a display device and a display program capable of efficiently performing an input instruction operation on a display screen by detecting a movement (gesture) of a pointing object such as a hand or a stick.

[0002]

2. Description of the Related Art Conventionally, as a method for operating a menu screen or the like displayed on a display screen, a method using a touch panel method has been used in which an operator operates by directly touching the display screen.

A display screen for performing the above operation will be described with reference to the drawings.

FIG. 1 is a diagram showing an example of a conventional display screen.

In FIG. 1, the operator touches the selection area corresponding to the desired operation content from the screen selection area 11 on the menu screen (FIG. 1A) displayed on the display screen 10. Can switch to the next screen. For example, when the screen selection area 11-1 is selected, a channel selection screen as shown in FIG. 1B is displayed on the display screen 10, and the channel selection areas (12-1 to 12-1
You can select the channel (program) you want to watch from 2-9). When the screen selection area 11-2 is selected, a volume adjustment screen as shown in FIG. 1C is displayed, and the volume can be adjusted by the volume adjustment area 13. If the screen selection area 11-3 is selected, the screen shown in FIG.
The data broadcast screen can be viewed as shown in (d).

[0006] However, when watching a program to be watched, the viewer must come close to a distance where the screen can be touched, and when the display screen is large, it becomes very difficult to watch on the spot. Therefore, recently, a display device or the like having a display screen for recognizing and operating a movement (gesture) of a pointing object such as a hand or a stick operated by an operator is required.

The method of recognizing the gesture of the operator to operate the device has not been put into practical use at the research stage, but using two cameras for both eyes, for example, the depth direction of the operator's hand. Only the movement of the hand is separated from the data of the hand, the coordinate of the hand is calculated from the separated movement of the hand, and the distance between the position of the coordinate and the position of the display screen and the parameter coordinate is calculated to detect the hand gesture. Then, a method of operating the display device can be considered.

In another method, for example, when a gesture is detected using one camera, a color of a hand (for example, skin color) is detected to detect a movement of a hand in front of clothes. Alternatively, you can put a white marking on your hand to make it easier to separate it from other background images and detect the movement of the marking, or you can attach a magnetic sensor to your hand and generate a magnetic field with a coil to detect the movement. The method to do is used. In addition to gestures, a method of inputting audio on a display screen is also used.

[0009]

However, in the above case, although the position of the pointing object can be detected when it is within the detection range of the camera, if the position is outside the detection range, the instruction content is not detected. There was a problem that I could not be satisfied.

Further, even if the gesture is successfully detected,
Due to the small movements of the body, the icon of the menu button may not be stably indicated, or the person may be instructed even if he / she is not pointing when passing by the display device. Furthermore, since the detection accuracy of the camera in the depth direction is generally not so good, it is necessary to use another manual throw such as voice to determine the icon menu. Further, when a plurality of pointing objects are simultaneously within the detection range of the camera, it is impossible to determine which pointing object to select, and it is not possible to perform accurate pointing recognition.

Further, when the pointing object is within the detection range of the camera, there is a problem that the menu screen continues to be displayed even if it is not operated, and the entire screen cannot be enjoyed.

The present invention has been made in view of the above-mentioned problems, and detects a movement (gesture) of a pointing object with high accuracy by using a stereo camera including two or more cameras, thereby improving efficiency. It is an object of the present invention to provide a display device and a display program capable of displaying a designated point on a display screen.

[0013]

In order to solve the above problems, the present invention employs means for solving the problems having the following features.

According to the first aspect of the present invention, the point designated on the display screen is based on the position of the pointing object designated on the display screen measured by the stereo camera including the two-lens camera. In the display device that displays
Place multiple single-lens cameras around the display screen,
A detection unit that detects a distance between an eye camera and the pointing object, and a selection that selects two single-lens cameras that form a pair of the stereo cameras based on the distance between the pointing object obtained by the detection unit. And means.

A stereo camera is composed of two left and right cameras corresponding to the left and right eyes of a human, but recently, a system using more cameras called multiview is seen from the camera side. It is often used because it can solve the problem of occlusion in which an opaque object in the foreground obscures an object at a distance.

According to the first aspect of the present invention, it is possible to select the single-lens camera and the stereoscopic camera corresponding to the position of the pointing object. Further, by this, the gesture of the pointing object can be recognized with high accuracy, and the points can be efficiently displayed on the display screen.

According to a second aspect of the invention, when the distance between the pointing object and the single-lens camera is within a predetermined distance, the selecting means selects the two units having the shortest distance and is outside the predetermined distance. In the case of, there is a feature that two units at both ends are selected.

According to the second aspect of the present invention, the optimum camera can be selected according to the position of the pointing object and the detection accuracy of the single-lens camera.

According to the third aspect of the present invention, the instruction is given on the display screen based on the position of the pointing object which is pointed to the display screen, which is measured by the stereo camera including two or more cameras. In a display device that displays points, arrange multiple stereo cameras around the display screen,
The present invention is characterized by further comprising detection means for detecting the distance of the pointing object from each of the stereo cameras, and selection means for selecting a stereo camera having the shortest distance from the pointing object in the detected distances.

According to the third aspect of the invention, it is possible to select a stereo camera that detects a gesture according to the position of the pointing object, so that highly accurate gesture detection can be performed. In addition, points can be efficiently displayed on the display screen.

According to a fourth aspect of the present invention, the detection means performs detection when the pointing object is within a preset speed range and / or within a preset forward, backward, leftward and rightward movement range. It is characterized by

According to the fourth aspect of the present invention, the detecting means can give the same nonlinearity as the movement of the pointing object without moving the pointer, and can perform highly accurate gesture detection.

The invention described in claim 5 is characterized in that the detecting means detects the position of the pointing object when the pointing object has a convex shape.

According to the fifth aspect of the present invention, even if the pointing object other than the convex shape exists within the detection range of the stereo camera, it can be ignored, and the position of the pointing object can be detected with high accuracy. it can. Further, this enables highly accurate gesture detection of the pointing object.

According to a sixth aspect of the present invention, there is provided operator position measuring means for measuring the position of the operator who operates the pointing object, and the detecting means is the operator measured by the measuring means. It is characterized in that the position of the pointing object is detected in a detection range based on the position.

According to the sixth aspect of the invention, since the optimum detection range can be set according to the position of the operator, the position of the pointing object can be detected with high accuracy. Further, this enables highly accurate gesture detection of the pointing object.

According to a seventh aspect of the invention, the detecting means detects the position of the pointing object closest to the display screen when a plurality of pointing objects are detected by the stereo camera. To do.

According to the seventh aspect of the present invention, it is possible to identify one of the plurality of pointing objects detected by the stereo camera. Further, this enables highly accurate gesture detection of the pointing object.

In the invention described in claim 8, when the stereo camera detects a plurality of pointing objects, the detecting means detects the position of the pointing object having the maximum or minimum movement amount and / or speed amount. It is characterized by detecting.

According to the invention described in claim 8, it is possible to identify one of the plurality of pointing objects detected by the stereo camera. Further, this enables highly accurate gesture detection of the pointing object.

The invention described in claim 9 is characterized in that it has a switching means for switching the setting on the display screen when the pointing object to be detected by the stereo camera is located outside the detection range.

According to the invention of claim 9, it is possible to easily switch the setting on the display screen.

According to a tenth aspect of the present invention, there is provided a deciding means for deciding an instruction at the pointer position on the display screen when the pointing object to be detected by the stereo camera is located outside the detection range. And

According to the tenth aspect of the present invention, it is possible to easily determine the instruction at the pointer position on the display screen.

According to the eleventh aspect of the present invention, the instruction is given on the display screen based on the position of the pointing object pointed to the display screen measured by the stereo camera including two or more cameras. In a display device that displays points, a plurality of stereo cameras are arranged around a display screen, and a recognition unit that recognizes a face of an operator who operates the pointing object using a predetermined stereo camera among the stereo cameras, Based on the detection means for detecting the position of the pointing object using the stereo camera having the shortest distance to the pointing object, and the operator's recognition step recognized in the recognition step, the operator-related information is output. It has a control means for controlling so as to perform.

According to the eleventh aspect of the present invention, it is possible to provide the operator with an optimum display screen by recognizing the face of the operator. Further, this enables highly accurate gesture detection of the pointing object.

According to a twelfth aspect of the present invention, the detection means detects the position of the pointing object based on the movement detection speed and / or the detection position range corresponding to the operator recognized by the recognition means. It is characterized by performing.

According to the twelfth aspect of the invention, since it can be set for each operator, the position of the pointing object can be detected with high accuracy. Further, this enables highly accurate gesture detection of the pointing object.

The invention described in claim 13 is the point designated on the display screen based on the position of the pointing object which is designated on the display screen measured by the stereo camera including the two-lens camera. In a program included in a display device for displaying, a plurality of single-lens cameras are arranged around a display screen, and a detection process of detecting a distance between each single-lens camera and the pointing object, The computer is caused to execute a selection process of selecting two single-lens cameras that form a pair of the stereo cameras based on the distance of the designated object.

According to the thirteenth aspect of the present invention, it is possible to select a single-lens camera and a stereo camera corresponding to the position of the pointing object. Further, by this, the gesture of the pointing object can be recognized with high accuracy, and the points can be efficiently displayed on the display screen.

According to the fourteenth aspect of the present invention, the instruction is given on the display screen based on the position of the pointing object pointed to the display screen measured by the stereo camera composed of two or more cameras. In a program included in a display device that displays points, a plurality of stereo cameras are arranged in the periphery of a display screen, a detection process of detecting the distance of the pointing object from each of the stereo cameras, and the detected distance, The computer is caused to execute a selection process of selecting a stereo camera having the shortest distance from the pointing object.

According to the fourteenth aspect of the invention, it is possible to select a stereo camera that detects a gesture according to the position of the pointing object, so that highly accurate gesture detection can be performed. In addition, points can be efficiently displayed on the display screen.

In the invention described in claim 15, the instruction is given on the display screen based on the position of the pointing object which is instructed to the display screen, which is measured by a stereo camera including two or more cameras. In a program included in a display device that displays points, a plurality of stereo cameras are arranged around a display screen, and a face of an operator who operates the pointing object is recognized by using a predetermined stereo camera among the stereo cameras. Recognition processing, detection processing for detecting a pointing object using a stereo camera having the shortest distance to the pointing object, and information related to the operator based on the recognition step of the operator recognized in the recognition step. And causing the computer to execute a control process for controlling to output.

According to the fifteenth aspect of the present invention, it is possible to provide the operator with an optimum display screen by recognizing the face of the operator. Further, this enables highly accurate gesture detection of the pointing object.

[0045]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention recognizes the movement (gesture) of a pointing object that points to a display screen by using a plurality of stereo cameras each of which has two or more cameras installed around the display screen. In this case, the main object is to select a stereo camera according to the position of the pointing object and perform gesture detection, thereby performing highly accurate gesture detection and efficiently performing pointing on the display screen.

Next, an embodiment of the present invention will be described with reference to the drawings. In order to easily explain the content of the present invention, the operator's hand is used as the pointing object.

FIG. 2 shows a first embodiment of the display device according to the present invention.

The display device 21-1 shown in FIG.
-1, a camera 1, a camera 2, and a camera 3 are configured.

FIG. 3 is a block diagram showing an example of the display device shown in FIG. The display device in FIG.
The camera 1, the camera 2, the camera 3, and the image processing unit 23
-1, a calculation unit 24-1, a storage unit 25-1, and a control unit 26-1.

In the first embodiment, three cameras are installed in the frame portion of the display device 21-1, matching is performed from the three single-lens cameras, and two cameras to be used as stereo cameras are selected. As a result, the pointing direction of the pointing object such as the operator's hand or stick and the entire movement of the hand are extracted and output as a device control signal.

First, when a hand is photographed by the camera 1, the camera 2 and the camera 3, the image processing unit 23-1 obtains the difference between the cameras 1, 2 and 3 respectively. Extract the parallax image. Where LPF
It is possible to obtain a rough parallax image of an object from the difference between the left and right images that have passed through the (low-pass filter).
A fine parallax image such as a fold of cloth can be obtained from the difference between the left and right images that have passed through the HPF (high-pass filter). According to the present invention, a rough parallax image of the entire hand or the like may be extracted from the background, and the LPF may be used to save the image memory. If the camera parameters such as the interval between the cameras, the angle of view, the resolution, etc. are known, the calculation unit 24-1 uses the binocular parallax image output from the image processing unit 23-1 to determine the actual target object. Calculate the distance.

The storage unit 25-1 stores, as data, an algorithm for separating numerical values and hand movements used for gesture detection. For example, in the menu screen shown in FIG. 1 (FIG. 1A), when the operator's gesture points to a portion of the screen selection area 11, a preset moving speed of the hand (for example, 50 cm / sec) And a preset range (for example, 10c in the screen selection area 11).
m) As above, when the operator's hand does not move, by not moving the operator's finger pointing position from the selected area, parameters for enabling stable gesture recognition even if the hand shakes, and the camera 10% to 30% (5 to 15 cm in the case where the detection range of the camera is 50 cm) or more, the objects that are not convex in the depth direction are the fields of view of the camera 1, the camera 2, and the camera 3. Parameters such as parameters that are not mistakenly recognized as a gesture even when an object other than the hand passes by are ignored, even if the object enters or passes by and is ignored.

The calculation unit 24-1 is the storage unit 25-described above.
Using the data of 1, the moving part of the hand is separated from the background and the body, and the amount of motion is output as a control signal. Control unit 26-
1 selects two cameras based on the control signal input from the calculation unit 24-1 and the data of the stereo pair most suitable for the gesture detection acquired from the storage unit 25-1. Outputs control signals related to the camera.

Here, an example of the selection process of the control unit 26-1 will be described using a flowchart.

FIG. 4 is an operation flowchart of an example of the camera selection process in the control unit.

In FIG. 4, when the camera detects the operator, the coordinates of the position of the hand are measured from the distance between the camera and the hand (S11), and the stereo pair to be used is selected from the position of the hand. For example, the position of the operator's hand is displayed on the display device 21-.
If it is close to 1 and to the left of the center, a stereo pair is selected using camera 1 and camera 2 (S12). If the position of the hand is close to the display device 21-1 and to the right of the center, the camera 2 and the camera 3 are selected (S1).
3). Further, when the operator's hand is farther than the display device 21-1, the cameras 1 and 3 at both ends are selected and the parallax of the stereo camera is increased to improve the detection accuracy. The determination as to whether the operator's hand is close to the display device 21-1 is set and stored in the storage unit according to the size of the display screen 22-1, the installation intervals of the cameras, and the detection accuracy of the cameras. ing.

As a result, it is possible to improve the detection accuracy of the operation contents of the operator, and to attach the operator's screen to the display screen 22-1 without touching the display screen 22-1 with only hand gestures. The operation can be performed easily. In addition, by selecting the camera, it is possible to detect the gesture in a wide range and easily display the point on the display screen with high accuracy.

The block configuration of the display device in the above-described first embodiment is not limited to this, and the display device 21 is not limited thereto.
-1 may be equipped with four or more cameras.

Next, a second embodiment of the present invention will be described.

FIG. 5 shows a second embodiment of the display device according to the present invention.

The display device 21-2 shown in FIG.
-2, a stereo camera 4, a stereo camera 5, a stereo camera 6, and a stereo camera 7.

In the display device 21-2 of FIG. 5, one pair of left and right stereo cameras is pre-configured and installed at the four corners of the frame portion of the display device 21-2. By using a close stereo camera, the movement (gesture) of the hand can be detected with high accuracy. Also,
FIG. 6 is a block diagram of an example of the display device of FIG. The display device 21-2 includes a stereo camera 4, a stereo camera 5, a stereo camera 6, a stereo camera 7, an image processing unit 23-2, a calculation unit 24-2, a storage unit 25-2, and a control unit. And a portion 26-2.

In FIG. 6, in the image processing section 23-2,
The binocular parallax image of each stereo camera (4 to 7) is detected.
The calculation unit 24-2 outputs the movement amount as a control signal based on the data from the storage unit 25-2. Control unit 26-
In 2, the operator selects which stereo camera to use to detect the movement of the operator's hand based on the data obtained from the storage unit 25-2, and outputs a control signal for the stereo camera.

Here, the stereo camera selection criterion is, for example, a stereo camera capable of detecting the hand of the operator operating the screen from a position of the camera to a predetermined position (for example, a position of about 50 cm). , By using the stereo camera closest to the hand, it is possible to detect the hand motion (gesture) with high accuracy. It is also possible to measure a shift angle or the like with respect to the front direction of the stereo camera and select a stereo camera having a small shift angle. The installation position and the number of stereo cameras are not limited to this.

Since the position of the hand pointing the screen moves, the hand detection process is periodically performed according to the position of the moved hand, and the stereo camera closest to the hand is selected.

Next, the contents stored in the above-mentioned storage unit 25 for detecting the movement of the operator's hand with high accuracy will be described in detail.

FIG. 7 is a diagram showing an example of the first detection criterion in the present invention.

In FIG. 7, as described above, when the operator's gesture points to a portion of the screen selection area 11, a preset hand movement speed (for example, 50 cm /
Seconds), and when the operator's hand does not move within a preset range (for example, 10 cm in the screen selection area 11),
Even if the operator's hand has moved a little from the set range, it is set so that it is not judged to be moving.
By providing so-called non-linearity, stable gesture detection can be performed.

In FIG. 7, the actual movement of the hand is indicated by the solid line, and the control signal indicating the movement of the hand output as a control signal from the control unit 26 is indicated by the dotted line. As shown in FIG. 7, by not detecting the movement of the hand in the set speed range and the movement range including the front, rear, left, and right, even if the operator's hand is shaking, the stable pointer display can be achieved. It can be carried out.

The above-mentioned predetermined speed is about 5
It has been known that highly accurate gesture detection can be performed by not performing gesture detection when the operator's hand is not moving and the error is about 10 cm or more as a predetermined range at 0 cm / sec.

When one or a plurality of operators detect a plurality of hands with the stereo camera at the same time, the respective motion amounts are measured and the hands to be detected are selected based on the motion amounts. Here, the above contents will be described with reference to the drawings.

FIG. 8 is a diagram showing an example of the second detection criterion in the present invention. In FIG. 8, for example, hand A and hand B
If both are operating on the display screen at the same time, the amount of movement of each hand is calculated and compared. As a result of comparison, when the hand with the larger movement amount is the detection target, the control signal is output with the hand A as the gesture detection target because the hand A has the larger movement amount. The comparison process described above is performed by the control unit 26.
Will be done at. Further, the detection condition is not limited to this, and the one having a smaller motion amount may be the detection target.

In general, when making a gesture that points to the display screen, when a person passes by in front of the background or the display screen, it may be erroneously detected. Therefore, the depth amount (the amount of forward and backward movement with respect to the stereo camera) is calculated from the input of the stereo camera, and if the depth amount is less than or equal to a preset value (threshold value), gesture detection is not performed. The detection accuracy can be improved. Here, the above contents will be described with reference to the drawings and the flowchart.

FIG. 9 is a diagram showing an example of the third detection criterion in the present invention. Further, FIG. 10 is an operation flowchart of an example of the detection reference process in FIG. 9.

In FIG. 10, when the video signal from the stereo camera which has already been set is input (S21), the calculation of the depth amount is performed by the calculation unit 24 (S22). The calculated depth amount is output to the control unit 26, and the control unit 26 compares the depth amount from the calculation unit 24 with the threshold value of the depth amount stored in the storage unit 25 (S23), and the threshold value. In the above case (YES in S23), gesture detection is performed (S24). As a result, as shown in FIG. 9, when the depth amount is equal to or larger than the threshold value, the gesture detection target is set, and when the depth amount is smaller than the threshold value, the gesture is ignored, so that the accuracy of the hand movement can be improved. In addition,
As the range of the above-mentioned threshold value, for example, 10% to 30% (the detection range is 50 c
If m, 5 to 15 cm) is set.

Further, when detecting the depth of the hand, the detection range of the hand is adaptively changed according to the position of the operator, so that highly accurate gesture detection can be performed regardless of the distance between the operator and the display screen. It can be carried out. The above contents will be described with reference to FIG. 11 showing an example of the fourth detection criterion in the present invention.

As shown in FIG. 11, for example, time t1.
When the position of the operator's body is at the depth position of A at t2,
The detection range of the operator's hand in the stereo camera is A1
Position A2. In this way, by measuring the depth amount of the body and measuring the detection range of the hand accordingly,
Even if the operator moves to a position having a different depth and gives an instruction to the display device, it is possible to detect the motion of the hand with high accuracy.

Further, when a plurality of hands are operated at the same time, the depth amount is measured as described above in addition to selecting the hand to be the gesture detection target according to the hand movement amount shown in FIG. By comparing, the target operator's hand can be selected. The above contents will be described with reference to FIG. 12 showing an example of the fifth detection criterion in the present invention.

As shown in FIG. 12, for example, when a gesture with two hands (hand A, hand B) is performed, the calculation unit 24 calculates the depth amount of each and the control unit 26.
Uses the calculation result to make the hand (hand B) having the smallest depth amount in the horizontal direction (front of the display device) the hand for gesture detection. The gesture detection hand may be the one having the largest depth amount in the horizontal direction.

The detection reference as described above is stored in the storage unit 25, and the control unit 26 selects a stereo camera or detects the position of the hand based on the detection reference stored in the storage unit 25 to increase the detection standard. Accurate gesture detection can be performed, and the point corresponding to the position of the hand can be accurately displayed on the display screen 22.

Even when the operator's hand is within the detection range of the stereo camera, the operator concentrated on the displayed screen may be in a state of doing nothing. In that case,
When the movement of the operator's hand is within the gesture detection range and no operation has been performed for a certain period of time, the menu screen is switched to non-display. After that, for example, when the hand moves out of the detection range and enters the gesture detection range again, the menu screen is displayed.

The above processing contents will be described with reference to a flowchart.

FIG. 13 is a flow chart of an example of the switching process according to the present invention. In FIG. 13, first, when the movement of the operator's hand is detected (S31), it is determined whether or not it is within the gesture detection range using the detection reference stored in the storage unit 25 (S32). If the hand is within the detection range in S32 (YES in S32), it is determined whether or not the instruction operation is performed by the operator for a predetermined time or more (S3).
3). If the operator's hand is out of the detection range in S32 (NO in S32), the process ends. S3
The constant time in 3 is stored in the storage unit 25 as the last operated time, and the control unit 26 compares the times.

In S33, if the operation has not been performed for a certain period of time (YES in S33), the menu screen of the display device 21 is hidden (S34). In S33, when the instruction operation is performed by the operator within a certain time (NO in S33), the menu screen is displayed (S).
35). As a result, the operator can view the display screen without bothering the menu screen.

The above-mentioned switching processing is not limited to the above, and the display screen may be switched when the operator's hand moves out of the detection range of the stereo camera, for example. The switching process described above is performed by the control unit 26.

Further, an embodiment of the present invention in the case where the deciding operation method when the operator decides an operation is performed by a gesture will be described with reference to a flowchart.

FIG. 14 is an operation flowchart of an example of the operation determining process according to the present invention. In FIG. 14, first, when the movement of the operator's hand is detected (S41), it is determined whether or not the movement of the hand is on the screen selection area 11 in FIG. 1 (a) (S42). Note that, here, it suffices to have a hand on the selection area, for example, the channel selection area 12 in FIG. 1B.

In S42, if it is on the screen selection area 11 (YES in S42), it is next determined whether or not the hand which is the gesture detection target of the operator is outside the detection range (S).
43). In S43, for example, when the operator moves his / her hand to a position outside the detection range (for example, put on the knee), the hand is outside the detection range (Y in S43, Y
ES), assuming that the decision operation has been performed, the display screen 22
The process corresponding to the selected area is performed (S44). In addition,
In S42, if it is not on the screen selection area 11 (S4
2, NO), or if the hand is within the detection range (S
In NO, the process ends here. Thereby, the decision operation can be easily performed.

The above-mentioned contents are not limited to the determination operation, but the depth amount of the hand is measured in S43, and if there is a movement of a predetermined depth amount or more, the operator gives a pointing instruction. It may be determined that there has been, and the determination operation of S44 may be performed.

The above-described determination operation process is executed by the control unit 26.
Will be held in.

Furthermore, the operator can be specified by extracting the face portion of the operator using the outputs of one or two of the already set stereo cameras, and the operation can be performed. Personal information of a person is stored in advance, and by using the personal information, for example, specific channel settings, hand detection position ranges, and movement detection speed settings corresponding to each individual can be changed.

For example, when a specific operator is recognized from the operator's face or the like by using a stereo camera and a television is operated based on the personal information, the channel is locked so that a certain channel cannot be operated, or the operator does not operate. If you come in front of the television, you can set it to the channel that is broadcasting the program that the person likes or is most relevant to. Furthermore, the "gesture detection speed threshold", "gesture detection position threshold", or "gesture input range" preset for the operator by face recognition
By setting such as, it is possible to perform highly accurate gesture detection adapted to the operator.

Next, an example of the display device for performing the above face recognition will be described with reference to the drawings.

FIG. 15 is a block diagram showing an example of a display device for recognizing a face image according to the present invention. Note that FIG.
5 already has two cameras, one for the right eye and one for the left eye.
It is assumed that the cameras of one stereo pair are set.

The display device shown in FIG. 15 has a left-eye camera 31.
A right-eye camera 32, a face image processing unit 33, a face image recognition unit 34, an image processing unit 23-3, and a calculation unit 24-3.
And a storage unit 25-3 and a control unit 26-3.

Now, a method of recognizing a face by the stereo camera will be described. As a typical method, there is a method in which one of the stereo cameras is used to photograph the face of the operator and the face of the operator is matched with a prepared face template. Since it requires a huge amount of calculation time to perform matching using an original image with high resolution, methods such as extracting rough information through the LPF for matching, extracting only edges, and using face color and hair color are available. It is often used.

Further, as a more statistical method, a face image from multiple directions is photographed in advance, the feature amount is extracted by multivariate analysis, the feature amount is compared with a newly input image, and obtained by multivariate analysis. Method of recognizing the face image with the coordinates closest to the specified coordinates, or the method of obtaining the difference between the left and right cameras using a stereo camera, creating a depth direction map of the face image, and specifying the face image from the previously captured pattern. Etc. As the face image recognition method used in the present invention, various methods other than the methods mentioned above can be used.

First, the left-eye camera 31 and the right-eye camera 3
The video imaged from 2 is output to the image processing unit 23-3 and the face image processing unit 33. The image processing unit 23-3 extracts the binocular parallax image as described above. Face image processing unit 33
Then, the face part of the operator is extracted using the above-described method. The face image recognition unit 34 identifies the operator's face based on the face data registered in advance. Information of the specified operator is output to the calculation unit 24-3, and the calculation unit 24-3 detects the gesture of the hand with respect to the data input from the image processing unit 23-3, and sets or locks the channel.
Alternatively, a control signal for displaying a selection screen suitable for the individual on the display screen is simultaneously output. Further, it is possible to set parameters and the like for smoothly performing gesture input according to an individual.

According to the present invention, a face recognizable display device is provided as shown in FIG. 15, so that the face of the operator is specified, the operator switches to the favorite channel, and the broadcast list of the favorite program is displayed. It can have functions such as the thing.

For example, if the operator is a minor, it is possible to prevent the channel from being operated for a program that is highly stimulating such as for adults or for violence. Further, the channel display can be changed according to the recognized individual preference. Ordinary TV, 1 to 12 channels of ground wave,
Alternatively, although BS digital channels 1 to 9 icons are displayed, CS (Communication S) can be used for operators who like movies and specific sports.
It is possible to display the selection area only for the channels of the related genres that can be received, including the channels of the Atellite) and cable television.

This function can be applied not only to the display device but also to the recording / reproducing device, the computer device and the interactive display, by organizing the selection areas not used by the operator,
Since only the selection area to be used can be provided, it is possible to design a neat and large selection area, and it is possible to provide an environment in which the operator can easily input a gesture.

As described above, in the embodiment shown in FIG. 15, the fact that the gesture detection camera simultaneously captures the face of the operator is used to recognize the face of the operator and to perform an operation for gesture input. It is possible to improve the sex. In addition,
In the present invention, a stereo camera for face recognition and a stereo camera for gesture detection may be separated by a display device having a plurality of stereo cameras. In that case, it is preferable that the stereo camera located at the shortest distance from the operator's hand detect the gesture and the second closest stereo camera recognizes the face.

Further, when a stereo camera is installed in a device such as a television, lighting, and an air conditioner in a house or a room, and when the movement of the operator's hand is input, the stereo camera closest to the hand is selected, and the television is selected. Can be operated, the air conditioner can be operated, the lighting can be operated, the door lock can be opened and closed, and the like.

In the above description, an example in which gesture detection is performed targeting the movement of the operator's hand has been described, but the present invention is not limited to this. For example, in order to give the operator's foot or a specific instruction. A gesture can be detected by using a stick or the like as a pointing object.

The operator's gesture may be detected by combining a plurality of the present embodiments.

Based on the position of the pointing object pointing on the display screen measured by the stereo camera as described above,
The display device for displaying the designated point on the display screen has been described, but a display program for displaying the designated point on the display screen is generated based on the position of the pointing object, and based on the program. It is also possible to cause the computer to execute the processing in each block on the display device described above.

The display program described above is a CD-
By storing it in a storage medium such as a ROM and installing it in a computer, highly accurate gesture detection can be easily executed. Further, this makes it possible to easily display the points on the display screen based on the operation of the operator.

As another method, a highly accurate gesture detection can be executed by obtaining the display program in a file format from the communication interface and the network and installing it in the computer.

As described above, according to the present invention, a stereo camera composed of two or more cameras is used to detect the gesture of the operator with high accuracy and to display the motion of the operator on the display screen. Based points can be easily displayed. In addition, this enables efficient and highly accurate input operation to the display device.

A display device and a display program capable of detecting the movement (gesture) of the pointing object with high accuracy by using the stereo camera and efficiently displaying the point pointed on the display screen are provided. Can be provided.

Further, a television device using the display device and the display program of the present invention, a television recording / recording device,
It is possible to realize a playback device, a computer device having a television display function, and an interactive display.

[0112]

According to the present invention, the movement of the pointing object can be detected with high accuracy. Further, this makes it possible to efficiently display the designated point on the display screen.

[Brief description of drawings]

FIG. 1 is a diagram showing an example of a conventional display screen.

FIG. 2 is a first embodiment of a display device according to the present invention.

FIG. 3 is a block configuration diagram of an example of the display device of FIG.

FIG. 4 is an operation flowchart of an example of a camera selection process in the control unit.

FIG. 5 is a second embodiment of the display device according to the present invention.

FIG. 6 is a block configuration diagram of an example of the display device of FIG.

FIG. 7 is a diagram showing an example of a first detection criterion in the present invention.

FIG. 8 is a diagram showing an example of a second detection criterion in the present invention.

FIG. 9 is a diagram showing an example of a third detection criterion in the present invention.

FIG. 10 is an operation flowchart of an example of detection reference processing in FIG.

FIG. 11 is a diagram showing an example of a fourth detection criterion in the present invention.

FIG. 12 is a diagram showing an example of a fifth detection criterion in the present invention.

FIG. 13 is a flowchart of an example of a switching process according to the present invention.

FIG. 14 is an operation flowchart of an example of operation determination processing according to the present invention.

FIG. 15 is a block diagram showing an example of a display device for recognizing a face image according to the present invention.

[Explanation of symbols]

1,2,3 camera 4, 5, 6, 7 stereo camera 10 Display screen 11 Screen selection area 12 channel selection area 13 Large / small adjustment area 23 Image Processor 24 arithmetic unit 25 memory 26 Control unit 31 Left-eye camera 32 Right eye camera 33 Face image processing unit 34 Face image recognition unit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁷ identification code FI theme code (reference) G06T 7/60 180 G06T 7/60 180B (72) Inventor Toshihiro Shimizu 1-10-11 Kinuta, Setagaya-ku, Tokyo No. Japan Broadcasting Corporation Broadcasting Technology Research Institute (72) Inventor Shigezumi Kuwashima 2-10-12 Fuji, Kokubunji city, Tokyo F-term in View Plus Co., Ltd. (reference) 5B057 BA02 BA26 CC01 CH08 CH11 CH18 DA07 DA16 DB03 DC02 DC32 5B087 AA09 CC26 CC33 5L096 CA05 FA66 HA04 HA07 JA11

Claims (15)

[Claims] 1. A display device for displaying an instructed point on the display screen based on the position of an instructing object instructing the display screen measured by a stereo camera including a two-lens camera, The single-lens camera is placed around the display screen and
A detection unit that detects the distance between the eye camera and the pointing object, and a selection that selects two single-lens cameras that form a pair of the stereo cameras based on the distance between the pointing object obtained by the detection unit. And a display device. 2. The selecting means selects two units having the shortest distance when the distance between the pointing object and the single-lens camera is within a predetermined distance, and when the distance is out of the predetermined distance, the two distances of both ends are selected. The display device according to claim 1, wherein a base is selected. 3. A display device for displaying an instructed point on the display screen based on a position of an instructing object instructing the display screen, which is measured by a stereo camera including two or more cameras. Detecting means for arranging a plurality of stereo cameras around the display screen and detecting the distance of the pointing object from each of the stereo cameras, and selecting a stereo camera having the shortest distance from the pointing object in the detected distances. A display device comprising: 4. The detection means performs detection when the pointing object is within a preset speed range and / or within a preset forward, backward, leftward, and rightward movement range. 1 to 3
The display device according to item. 5. The display device according to claim 1, wherein the detecting unit detects the position of the pointing object when the pointing object has a convex shape. . 6. An operator position measuring means for measuring a position of an operator who operates the pointing object, wherein the detecting means comprises:
The display device according to any one of claims 1 to 5, wherein the position of the pointing object is detected in a detection range based on the operator position measured by the measuring unit. 7. The detection means detects the position of the pointing object closest to the display screen when a plurality of pointing objects are detected by the stereo camera. The display device according to any one of 1. 8. The detecting means detects the position of the pointing object having the maximum or minimum movement amount and / or speed amount when a plurality of pointing objects are detected by the stereo camera. Item 8. The display device according to any one of items 1 to 7. 9. The method according to claim 1, further comprising a switching unit that switches settings on the display screen when a pointing object to be detected by the stereo camera is located outside a detection range. The display device according to item. 10. The determination means for determining an instruction at the pointer position on the display screen when the pointing object to be detected by the stereo camera is located outside the detection range. The display device according to any one of items. 11. A display device for displaying an instructed point on the display screen based on the position of an instructing object instructing the display screen, which is measured by a stereo camera including two or more cameras. A plurality of stereo cameras are arranged around the display screen, and a recognition unit that recognizes the face of the operator who operates the pointing object using a predetermined stereo camera among the stereo cameras, and the distance between the pointing object and the recognition object is the shortest. A detection means for detecting the position of the pointing object using the stereo camera, and a control means for controlling to output information related to the operator based on the recognition step of the operator recognized in the recognition step. A display device having. 12. The detection unit detects the position of the pointing object based on the movement detection speed and / or the detection position range corresponding to the operator recognized by the recognition unit. 11. The display device according to item 11. 13. A display device is provided which displays an instructed point on the display screen based on a position of an instructing object which is instructed to the display screen measured by a stereo camera including a two-lens camera. In the program, multiple single-lens cameras are placed around the display screen, and
A detection process for detecting a distance between an eye camera and the pointing object, and a selection for selecting two single-lens cameras forming a pair of the stereo cameras based on the distance between the pointing object obtained by the detecting means. A program that causes a computer to execute processing. 14. A display device is provided which displays an instructed point on the display screen based on a position of an instructing object instructing the display screen measured by a stereo camera including two or more cameras. In the program, a plurality of stereo cameras are arranged around the display screen, and a detection process of detecting the distance of the pointing object from each of the stereo cameras, and in the detected distance, the distance from the pointing object is the shortest. A program for causing a computer to execute selection processing for selecting a stereo camera. 15. A display device is provided, which displays an instructed point on the display screen based on a position of an instructing object instructing the display screen measured by a stereo camera including two or more cameras. In the program, a plurality of stereo cameras are arranged around a display screen, and a recognition process of recognizing a face of an operator who operates the pointing object by using a predetermined stereo camera among the stereo cameras; Detection processing for detecting a pointing object using a stereo camera having the shortest distance, and control processing for controlling to output information related to the operator based on the recognition step of the operator recognized in the recognition step. A program for causing a computer to execute and.