JP2013161406A - Data input device, display device, data input method, and data input program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable comfortable operation regardless of the level of mastery of an operation method, thereby preventing malfunction.SOLUTION: A first position detection unit acquires first positional information representing the position of part of body for each user included in a video captured by an imaging apparatus, a user information analysis unit determines the user on the basis of the first positional information, and detects user information including information representing the shape of part of body for the user included in the video captured by the imaging apparatus, and a control unit executes processing corresponding to the user information detected by the user information analysis unit.

Description

  The present invention relates to a data input device, a display device, a data input method, and a data input program.

Various devices and methods have been proposed as user interfaces for operating devices such as computers and game machines. In particular, game machines have been proposed that detect a user's movement (motion capture) and operate the device with the posture of the user's entire body.
For example, the interface device described in Patent Document 1 recognizes the shape and movement of an object in an image captured by a CCD (Charge Coupled Device) camera, and displays the recognized shape and movement of the object on a display. When the user performs an operation such as a hand shake toward the CCD camera, the user's operation is displayed on the display screen of the display of the apparatus. At this time, the user can select a virtual switch or the like displayed on the display screen with an arrow cursor icon displayed by the operation.
The video display device described in Patent Literature 2 identifies a user, detects the position and operation of the user, determines a user who is a viewing target of the display video based on the detected position and operation of the user, and corresponds to the user. To control applications that display video.

JP 2004-78977 A JP 2009-87026 A

  However, in the interface device described in Patent Document 1, it may be impossible to intuitively recognize what operation the user performs and what operation can be performed. For example, a method for confirming an input operation (for example, an operation that substitutes for an operation input such as a mouse click), a method for moving a displayed video content (for example, an operation that substitutes for an operation input such as a mouse drag), or the like An input method corresponding to the above operation may not be easy for the user. As a method for confirming the input operation corresponding to the click operation, for example, there is a method in which the user stops his / her hand for a longer time than a predetermined time. In this case, the user needs to know in advance that the position of the hand is fixed in order to confirm the input operation. If the predetermined time is too long, the user may not be able to perform a comfortable operation.

  Further, in the video display device described in Patent Document 2, when the user operates, it may not be possible to determine whether the operation is an operation intended for an operation input or an operation intended for another purpose. When a plurality of users view the video displayed on the video display device at the same time, an operation input unintended by the other user may be performed due to an operation by one user. Further, when a plurality of users operate at the same time and each video display apparatus accepts an operation input, an operation may be instructed by instructing processing for conflicting operation inputs. This may hinder viewing of the video.

  The present invention has been made in view of such a problem, and can be operated comfortably regardless of the level of mastery of the operation method, thereby suppressing the occurrence of malfunction.

(1) The present invention has been made to solve the above-described problem, and one aspect of the present invention is a first that represents the position of a part of the body for each user represented in an image captured by the imaging device. A first position detection unit that acquires position information; a user that determines information based on the first position information; and a user that includes information representing a shape of a part of the user's body shown in an image captured by the imaging device A data input device comprising: a user information analysis unit that detects information; and a control unit that executes processing corresponding to the user information detected by the user information analysis unit.

(2) Another aspect of the present invention is the above-described data input device, wherein the user information analysis unit obtains second position information where another part different from a part of the body for each user is located. A second position detector configured to perform a process corresponding to the user information for a user having a predetermined relationship between the first position information and the second position information.

(3) Another aspect of the present invention is the data input device described above, wherein the part of the body is a hand.

(4) Another aspect of the present invention is the data input device described above, wherein the other part different from the part of the body is at least one of a face or eyes.

(5) Another aspect of the present invention is the data input device described above, wherein the imaging device is installed at a position higher than the height of the face or eyes.

(6) Another aspect of the present invention is the above-described data input device including a display unit that displays an image captured by the imaging device, and the imaging device is installed at a position higher than the display unit. It is characterized by.

(7) Another aspect of the present invention is the above-described data input device, wherein the imaging device includes a plurality of imaging units provided at different positions, and the first position detection unit includes the plurality of imaging units. First position information including distance information is acquired based on an image captured by the imaging unit, and the user information analysis unit determines a user based on the first position information acquired by the first position detection unit. It is characterized by.

(8) Another aspect of the present invention is the above-described data input device, in which the user information analysis unit executes a process executed after a predetermined time has elapsed from the start of detection of the user information and the body of the user. And a display unit for displaying a guide image representing a part of the shape and the relationship.

(9) Another aspect of the present invention is the above-described data input device, wherein the control unit is configured so that the predetermined time elapses after the user information analysis unit detects the user information. When the process corresponding to the user information detected by the user information analysis unit is executed, the display unit does not display the guide image.

(10) Another aspect of the present invention is the above-described data input device, wherein the user information analysis unit estimates feature information representing the feature of the user based on an image captured by the imaging device, and The display unit displays a guide image having a different display mode according to the feature information.

(11) Another aspect of the present invention is a display device including the above-described data input device.

(12) According to another aspect of the present invention, in a data input method in a data input device that inputs data based on an image captured by an imaging device, the data input device is a user represented by an image captured by the imaging device. A first process of acquiring first position information representing a position of a part of the body, and the data input device discriminates a user based on the first position information, and the image captured by the imaging device A second step of detecting user information including information representing a shape of a part of the represented user's body, and a third step of executing a process corresponding to the detected user information by the data input device; It is a data input method characterized by having.

(13) Another aspect of the present invention is the above-described data input method, wherein the second process acquires second position information where another part different from a part of the body for each user is located. And the third step executes a process corresponding to the user information for a user having a predetermined relationship between the first position information and the second position information. .

(14) According to another aspect of the present invention, the position of a part of the body for each user represented in the video captured by the imaging device is input to the computer of the data input device that inputs data based on the video captured by the imaging device. A first procedure for obtaining first position information representing the information, a user is determined based on the first position information, and information representing a shape of a part of the body of the user represented in an image captured by the imaging device A data input program for executing a second procedure for detecting user information including the third procedure for executing processing corresponding to the detected user information.

(15) Another aspect of the present invention is the above-described data input program, wherein the second procedure acquires second position information where another part different from a part of the body for each user is located. The third procedure is characterized in that a process corresponding to the user information is executed for a user having a predetermined relationship between the first position information and the second position information.

  According to the present invention, it is possible to operate comfortably regardless of the level of mastery of the operation method, and to prevent malfunctions.

It is a conceptual diagram showing the usage condition of the display apparatus which concerns on embodiment of this invention. It is a top view showing the positional relationship of a user and a display apparatus. It is a block diagram showing the structure of the display apparatus which concerns on this embodiment. It is a conceptual diagram showing an example of a left image and a right image. It is a conceptual diagram showing the example of an image block. It is a conceptual diagram showing the positional relationship of an imaging surface. It is the schematic showing the structure of the user information analysis part which concerns on this embodiment. It is a conceptual diagram showing an example of the operation start detection range. It is a conceptual diagram showing the other example of the operation start detection range. It is a conceptual diagram showing an example of a guidance image. It is the schematic showing the timing which the control part which concerns on this embodiment performs a process. It is a flowchart showing the data input process which concerns on this embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a conceptual diagram showing how the display device 10 according to this embodiment is used.
In FIG. 1, a display device 10 is a device that displays video, such as a television receiver, a digital signage (electronic signage) device, or a video conference device. The display device 10 includes an imaging device 11 in the center of the lower front side, and includes a display unit 12 so as to cover most of the front surface.
The imaging device 11 is, for example, a stereo camera that captures an image in the front direction. The imaging device 11 includes, for example, imaging units 110a and 110b that capture images at positions separated from each other in the left-right direction. The imaging units 110a and 110b are each a camera unit. The display unit 12 is, for example, a display that displays an image. The display device 10 includes a speaker (not shown) that outputs sound.

  The operator 13 is a user who operates the display device 10. The operator 13 faces the front of the display device 10 and takes a predetermined posture, for example, a hand gesture or a gesture (gesture). The user information analysis unit 201 (FIG. 3) built in the display device 10 acquires user information representing the posture taken by a part of the body of the operator 13 represented in the image captured by the imaging device 11. The user information includes, for example, a hand shape such as a pointing hand, a fist, and information indicating how to move the hand. The display device 10 executes processing (function, operation) corresponding to the user information acquired via the imaging device 11. Thereby, the operator 13 can operate the processing of the display device 10 by the shape of the hand such as a pointing hand, the fist, and how to move it.

  The display device 10 is preset with an operable region that is set in advance based on its own position and that accepts an operation by the operator 13. In the display device 10, as an operable region, for example, an upper limit (for example, 3 m) of an operable distance that is a distance in the forward direction from the center of the display device 10 to the position of the hand of the operator 13 is set. . However, the left-right direction of the operable region can be set within the range of the viewing angle of the imaging device 11, for example. In this case, setting in the left-right direction is not necessary. The display device 10 does not accept an operation by the operation disabled person 14 who is a user away from the upper limit of the operable distance. The process for determining the user who receives the operation will be described in detail later.

FIG. 2 is a plan view showing the positional relationship between the user and the display device 10.
In FIG. 2, the top and bottom represent the back and front directions of the display device 10, respectively. The positional relationship among the operator 13, the inoperable person 14, and the display device 10 shown in FIG. 2 is the same. Here, in FIG. 2, the operator 13 is located on the front face of the display device 10 at a position where the distance from the display device 10 is shorter (closer) than the upper limit of the operable distance. On the other hand, the operation disabled person 14 is located in front of the display device 10 at a position where the distance from the display device 10 is longer (distant) than the upper limit of the operable distance.

  By setting the operable distance (the upper limit of the operable distance) as described above, the control unit 22 (FIG. 3) can perform an operation or an operation for an opportunity for a plurality of users to perform an operation at the same time or for a purpose other than the operation. Limits the opportunity to input an image representing an action that is mistakenly recognized (for example, a hand gesture by a passerby in the case of digital signage placed near the road). For example, even when the display device 10 is used by a plurality of users at the same time, such as a digital signage device installed in a public place, processing that is not intended by the user who views the video is avoided.

Although the example shown in FIG. 1 shows the case where the imaging device 11 is installed in the lower front side of the display device 10, it is not limited to this. For example, the imaging device 11 may be installed on the upper front side of the display device 10 or may be installed at a position away from the display device 10.
The imaging device 11 may be installed at a position higher than the height of the face of the operator 13, in particular, the height of the eyes. For this purpose, the height of the imaging device 11 is determined in advance in consideration of the height of the floor on which the operator 13 is located and the average height of a person. In addition, when the display device 10 is installed at a relatively low position such as a floor surface, the imaging device 11 may be installed at a position higher than the display unit 12.
Thereby, the imaging device 11 can capture an image representing the body of the operator 13 from a position higher than the face of the operator 13, and the operator 13 can change the shape of the hand such as a pointing hand, a fist, and how to move it. Can be prevented from being masked. Therefore, the control unit 22 can stably execute processing such as operator identification, face position detection, and operation detection performed using the face image of the operator 13. These processes will be described later.

[Configuration of display device]
Next, the configuration of the display device 10 according to the present embodiment will be described.
FIG. 3 is a block diagram illustrating the configuration of the display device 10 according to the present embodiment.
The display device 10 includes a data input device 2a or a display control device 2b. The data input device 2a includes an imaging device 11, an image processing device 20, an information DB 21 (database), and a control unit 22.
The imaging device 11 generates a video signal representing the captured video and outputs the generated video signal to the image processing device 20. The image processing device 20 obtains operator information representing the operator determined based on the video signal input from the imaging device 11, obtains first spatial information representing a position where a part of the operator's body is located, and operates. The user information representing the shape taken by a part of the person's body is acquired. The image processing device 20 outputs the acquired operator information, first spatial information, and user information as detection information to the display control device 2b.

The display control device 2b includes an information DB 21, a control unit 22, and a display unit 12.
The information DB 21 stores display information to be displayed in response to an operation input based on a video signal representing the video of the operator 13. The display information is, for example, a video signal representing video content, text information representing news, content information representing content received from the network, and a guide image signal representing a guide image (operation guide). Details of the guide image will be described later.

  The control unit 22 extracts first spatial information and user information from the detection information input from the image processing device 20. When the position of the operator 13 represented by the extracted first spatial information is within the predetermined operable region, the control unit 22 performs processing corresponding to the extracted user information. Here, for example, the control unit 22 determines whether or not the distance of the operator 13 represented by the first spatial information is smaller than the upper limit of the operable distance set in advance. Processing corresponding to user information includes, for example, display of guidance images, display of video content, information retrieval from the network, storage of video content and news related to the searched information, display of stored information, etc. This is processing related to various video displays.

The control unit 22 stores information instructed to be stored in the information DB 21 as display information. The control unit 22 reads display information instructed to be displayed from the information DB 21, and outputs a video signal representing the read display information to the display unit 12. The control unit 22 stops outputting the display information instructed to stop.
The display unit 12 displays the video signal input from the control unit 22 as a video. As a result, the video content or news related to the video content selected by the operator 13 and the operation, or the guidance image is displayed.
Thereby, the display control device 2b executes processing for selecting content represented by the user information included in the detection information input from the image processing device 20, and processing for displaying the selected content.

Next, a more detailed configuration of the data input device 2a will be described.
The imaging device 11 includes imaging units 110a and 110b. The imaging units 110 a and 110 b generate a video signal representing the captured video and output the generated video signal to the image processing device 200. The imaging unit 110a outputs the generated video signal to the user information analysis unit 201. The imaging units 110a and 110b are, for example, cameras that include an optical system including a lens that collects light incident from a subject and an imaging element that converts the collected light into an electrical signal. The imaging devices included in the imaging units 110a and 110b are, for example, CCD (Charge Coupled Device) elements and CMOS (Complementary Metal Oxide Semiconductor) elements.

The image processing apparatus 20 includes a distance calculation unit 200 and a user information analysis unit 201.
The distance calculation unit 200 receives video signals from the imaging units 110a and 110b. The distance calculation unit 200 calculates distance information representing the distance from the imaging device 11 to the subject (for example, the operator 13) based on each input video signal using, for example, a stereo matching method.

[Calculation of distance information]
Here, a calculation method of distance information using a block matching method which is a kind of stereo matching method will be described. In the stereo matching method, the parallax value of the video imaged by the imaging units 110a and 110b is calculated as the distance value. In the following description, an image included in a video captured by the imaging unit 110a and an image at a certain time point is referred to as a left image. An image included in the video imaged by the imaging unit 110b and the image at that time is called a right image.

In the stereo matching method, a right image block that is an area corresponding to a left image block that is a partial area of the left image is searched. Here, a left image and a right image captured simultaneously will be described as an example.
FIG. 4 is a conceptual diagram illustrating an example of the left image and the right image.
FIG. 4 shows a left image 40 on the left side and a right image 41 on the right side.
The distance calculation unit 200 sets a left image block (window) 400 centered on the target pixel in the left image 40. A total of nine squares in the left image block 400, three in the horizontal direction and three in the vertical direction, each represent a pixel. The horizontal distance from the right end of the left image 40 to the right end of the left image block 400 in FIG. 4 is L pixels (distance corresponding to L pixels). L is 1 or an integer greater than 1.

In the right image 41, the distance calculation unit 200 sets, as an initial value, a right image block 410 that has the same vertical coordinate as the left image block 400 and has a right end at L + d2 from the right end of the right image 41. . d2 is an integer value set in advance and represents the maximum parallax value. The size and shape of the right image block 410 are the same as those of the left image block 400.
The distance calculation unit 200 calculates an index value between the left image block 400 and the right image block 410. The distance calculation unit 200 shifts the position of the right image block 410 until the right edge of the right image block 410 from the right edge of the right image 41 becomes L pixels, and calculates an index value at each shifted position. The distance calculation unit 200 determines the right image block 410 at a position corresponding to the left image block 400 based on the calculated index value. For example, when an SAD (Sum of Absolute Difference) value is used as the index value, the right image block 410 at a position where the SAD value is minimized is determined. This position is an attention point corresponding to the attention pixel in the left image 40. The absolute value for the horizontal coordinate difference between the target point and the target pixel is parallax. The distance calculation unit 200 executes the respective pixels included in the left image 40, and displays disparity information (also referred to as a disparity map or a disparity map) indicating disparity values for each pixel included in the image captured by the image capturing unit 110a. Generate as The parallax has a relationship that increases as the distance from the imaging device 11 to the subject decreases, and decreases as the distance increases. The distance calculation unit 200 outputs the generated distance information to the user information analysis unit 201.

The disparity map is a grayscale bitmap image having a disparity value represented by an integer value represented by a predetermined number of bits (for example, in the case of 8 bits, a minimum value 0 to a maximum value 255) for each pixel. It is. The distance calculation unit 200 converts the parallax into a distance in the subject space from the imaging device 11 to the subject based on a camera parameter such as a baseline length that is an interval between the imaging unit 110a and the imaging unit 110b. Distance information representing the distance may be generated. Therefore, the distance calculation unit 200 may generate a bitmap image (depth map) obtained by converting the converted distance into a gray scale instead of the distance information indicating the parallax value for each pixel as the distance information.
Note that the imaging units 110a and 110b may be arranged at different coordinate values in the vertical direction, and calculate parallax using captured images that represent images captured by each. In that case, the distance calculation unit 200 may search the corresponding image block by shifting the image block in the image captured by the other image sensor in the vertical direction with reference to the image block in the image captured by either of the imaging units 110a and 110b. Good.

  The distance calculation unit 200 uses, for example, Expression (1) when calculating the SAD value.

In Expression (1), x _i is a pixel value for each pixel of, for example, green (G) included in the left image block 400. 8 is an example of the number of pixels included in one image block. The arrangement of the pixels corresponding to the pixel values x _{0 to} x ₈ is the order from the left end to the right end for each row as shown on the left side of FIG. 5, and from the top row to the bottom row between the rows. x _ai is a pixel value for each pixel included in the right image block 410. The arrangement of the pixels corresponding to the pixel values x _{a0 to} x _a8 is in the order from the left end to the right end for each row as shown on the right side of FIG. 5, and from the top row to the bottom row between rows.

The index value is not limited to the SAD value. Any other index value, such as an SSD (Sum of Squared Difference) value or DP (Dynamic Programming), may be used as long as it represents the correlation between the pixel value included in the left image block 400 and the pixel value included in the right image block 410. It may be a value.
The window size which is the size of the left image block 400 and the right image block 410 is not limited to 3 pixels in the horizontal direction × 3 pixels in the vertical direction as described above. For example, it may be larger than the above, such as 5 pixels in the horizontal direction × 5 pixels in the vertical direction, 9 pixels in the horizontal direction × 9 pixels in the vertical direction, and the center coordinate is the point of interest as in 4 pixels in the horizontal direction × 4 pixels in the vertical direction. The position may be shifted from the position. Further, the direction of shifting the right image block 410 is not limited from the left side to the right side as described above, and may be from the right side to the left side. The pixels included in the left image block 400 and the right image block 410 are not limited to the signal value of the green (G) pixel, but may be a signal value of another color, for example, a red (R) pixel. It may be a pixel signal value based on the above color system or any combination thereof.

  In the block matching method described above, the coordinates of the left image 40 and the corresponding coordinates of the right image 41 are shifted in the left-right direction, not shifted in the vertical direction, and the epipolar lines of the left image 40 and the right image 41 match. Was assumed. The reason why the optical axes of the imaging units 110a and 110b are arranged in parallel as described above is to match epipolar lines (also called auxiliary lines). In order to match the epipolar lines, based on the camera parameters of the imaging units 110a and 110b acquired in advance by the distance calculation unit 200, the image signals captured so that the optical axes of the left image 40 and the right image 41 are parallel are coordinated. You may perform the process to convert. The process of performing the coordinate transformation is called rectification or shift correction. The distance calculation unit 200 generates distance information after performing this process.

As shown in FIG. 6, the epipolar lines are lines 56 and 57 where the epipolar plane 53 and the imaging surfaces 54 and 55 included in each of the two imaging units 110 a and 110 b intersect. The epipolar plane 53 is a plane that passes through the three focal points 50 and 51 of the lens provided in each of the two imaging units 110a and 110b and the feature point 52 in the object space.
When the imaging units 110a and 110b are arranged so that the optical axes thereof are parallel, the epipolar lines 56 and 57 are horizontal lines having the same vertical coordinate in the left image 40 and the right image 41, respectively.

[Analysis of user information]
Next, the configuration of the user information analysis unit 201 according to the present embodiment will be described.
FIG. 7 is a schematic diagram illustrating the configuration of the user information analysis unit 201 according to the present embodiment.
The user information analysis unit 201 includes a face detection unit 30, an eye position detection unit 31, a hand position detection unit 32, a hand shape / fingertip position detection unit 33, a target position detection unit 35, a feature information analysis unit 37, and an operator determination unit 39. The detection information output unit 41 is provided.

[User face detection]
The face detection unit 30 detects an area representing an image of the operator's face represented by the video signal input from the imaging unit 110a. The face detection unit 30 generates two-dimensional face area information representing the two-dimensional coordinates of the representative points (for example, the center of gravity) in the detected face area and the two-dimensional coordinates of the upper end, the lower end, the left end, and the right end of the area. . The face detection unit 30 extracts the distance value related to the pixel of the two-dimensional coordinate represented by the two-dimensional face area information from the distance information input from the distance calculation unit 200. The face detection unit 30 converts the distance value corresponding to the above-described two-dimensional coordinates into three-dimensional coordinates in the object space, and generates three-dimensional face position information.
In order to detect a face area, the face detection unit 30 extracts, for example, pixels in a range of color signal values representing a preset face color (for example, skin color) from the input image signal.

Note that the face detection unit 30 may include a storage unit that previously stores a grayscale (monochrome) image signal representing a human face. Therefore, the face detection unit 30 calculates a correlation value between the grayscale image signal read from the storage unit and the input image signal for each image block including a plurality of pixels, and the calculated correlation value is greater than a predetermined threshold value. A large image block is detected as a facial region.
In addition, the face detection unit 30 calculates a feature amount (for example, Haar-Like feature amount) based on the input image signal, and performs a predetermined process (for example, an Adaboost algorithm) based on the calculated feature amount. The face area may be detected. The method by which the face detection unit 30 detects the facial region is not limited to the above-described method, and any method may be used as long as the facial region is detected from the input image signal.
The face detection unit 30 outputs a face image signal representing the detected face image to the feature information analysis unit 37 and the eye position detection unit 31. The face detection unit 30 outputs the generated 3D face position information and 2D face area information to the operator determination unit 39. The face detection unit 30 outputs the generated three-dimensional face position information to the detection information output unit 41 as part of the detection information.

[Eye position detection]
The eye position detection unit 31 detects an eye region from the facial image represented by the facial image signal input from the face detection unit 30. The eye position detection unit 31 calculates two-dimensional eye position coordinates that are representative points (for example, centroid points) of the detected eye region. The eye position detection unit 31 extracts the distance value at the pixel located at the detected eye position coordinates from the distance information input from the distance information calculation unit 200. The eye position detection unit 31 converts the set of the calculated two-dimensional eye position coordinates and the extracted distance value into three-dimensional eye position coordinates in the object space, and generates three-dimensional eye position information. The eye position detection unit 31 outputs the three-dimensional eye position information representing the calculated three-dimensional eye position coordinates to the attention position detection unit 35 and the operator determination unit 39. The eye position detection unit 31 outputs an eye region signal representing the detected image of the eye region and two-dimensional eye position information representing the calculated two-dimensional eye position coordinates to the operator determination unit 39.

In order for the eye position detection unit 31 to detect an eye region, for example, a storage unit that stores a template image of an eye that has been captured in advance is provided. The eye position detection unit 31 may use a template matching method that reads an eye template image from the storage unit and collates the read template image with the input facial image signal. In addition, the eye position detection unit 31 calculates the positional relationship of eyes on a predetermined face (for example, the pre-measured face area and the positions of both eyes) among the facial areas represented by the input facial image signal. The eye region may be detected using the eye position information that is represented. Further, the eye position detection unit 31 calculates a feature amount (for example, Haar-Like feature amount) based on the input face image signal, and performs a predetermined discrimination process (for example, an Adaboost algorithm) based on the calculated feature amount. ) To detect the eye region.
The method of detecting the eye region by the eye position detection unit 31 is not limited to the above-described method, and any method may be used as long as the method detects the eye region from the face image signal.
The eye position detection unit 31 may output an eye area signal representing the positions of the left eye and the right eye, or all of them, regardless of the center of gravity of both eyes as the eye area to be detected.

[Detection of hand position]
The hand position detection unit 32 detects a region representing the image of the operator's hand represented by the video signal input from the imaging unit 110a, and calculates the position of the detected hand.
In order to detect a region representing a hand image, the hand position detection unit 32 inputs, for example, a pixel within a color signal value range representing a color (for example, skin color) of a hand surface set in advance to an input video signal. Extract from The hand position detection unit 32 calculates a two-dimensional coordinate value of a representative point (for example, a center of gravity point) of an area representing the hand image detected as the hand position. The hand position detection unit 32 extracts a distance value corresponding to the calculated coordinate value from the distance information input from the distance calculation unit 200, and sets a set of distance values corresponding to the calculated two-dimensional coordinate value in the subject space. Three-dimensional hand position information is generated by converting into three-dimensional coordinates. The hand position detection unit 32 outputs a hand image signal representing the detected image of the hand region and hand position information representing the calculated two-dimensional coordinate value of the representative point to the hand shape / fingertip position detection unit 33. The hand position detection unit 32 outputs the hand position information to the operator determination unit 39.
Further, in order to detect the region representing the hand image, the hand position detection unit 32 represents the three-dimensional face position information input from the face detection unit 30 based on the distance information input from the distance detection unit 200. Extracting an image within a distance range represented by a start point and an end point in a predetermined depth direction with reference to the three-dimensional face position as a region representing a hand image from the video signal input from the imaging unit 110a Also good. The predetermined distance range is, for example, a range in front of the three-dimensional face position (on the display device 10 side). Thereby, it can prevent recognizing the hand of another person who is ahead or behind the operator as the hand of the operator.

[Detection of hand shape and fingertip position]
The hand shape / fingertip position detection unit 33 detects the shape of the hand based on the hand image signal input from the hand position detection unit 32 and the hand position information.
In order to detect the shape of the hand, the hand shape / fingertip position detection unit 33 performs edge extraction processing, for example, from the hand image signal to detect the contour portion of the hand. The hand shape / fingertip position detection unit 33 searches the detected contour portion for a projection portion having a predetermined radius of curvature (eg, 6-12 mm) as an image of the finger region. The hand shape / fingertip position detection unit 33 determines the presence or absence of the above-described protrusion in the search area having a predetermined radius from the representative point represented by the hand position information in the search, and changes the radius sequentially to make the search area concentric. Update to The hand shape / fingertip position detection unit 33 counts the number of fingers based on the detected finger area. The hand shape / fingertip position detection unit 33 detects the apex of the detected protrusion as two-dimensional coordinates representing the fingertip position of each finger. The hand shape / fingertip position detection unit 33 extracts the distance value of the pixel located at the two-dimensional coordinates of the determined fingertip from the distance information input from the distance calculation unit 200. The hand shape / fingertip position detection unit 33 generates a three-dimensional fingertip position information representing a three-dimensional coordinate in the subject space from the set of the extracted distance value and the two-dimensional coordinate at the fingertip. The hand shape / fingertip position detection unit 33 outputs the generated three-dimensional fingertip position information to the attention position detection unit 35. The hand shape / fingertip position detection unit 33 detects the finger image signal representing the detected finger region, the number information representing the number of fingers, and the two-dimensional fingertip position information representing the two-dimensional coordinates on the fingertip as detection information. Output to the output unit 41.

[Detection of attention position]
The attention position detection unit 35 is a position where the operator pays attention based on the three-dimensional eye position information input from the eye position detection unit 31 and the three-dimensional fingertip position information input from the hand shape / fingertip position detection unit 33. Detect the attention position. The attention position detection unit 35 calculates, for example, an intersection point where a straight line connecting the eye position represented by the three-dimensional position information and the fingertip position represented by the three-dimensional fingertip position information intersects the display device 10 as the attention position. The attention position detection unit 35 converts the calculated intersection (three-dimensional coordinates in the object space) into a two-dimensional image coordinate system for the image represented by the display device 10, and generates attention position information representing the converted coordinates. The attention position detection unit 35 outputs the generated attention position information to the detection information output unit 41 as a part of the detection information.

[Analysis of feature information]
The feature information analysis unit 37 generates feature information representing user attributes (for example, age, gender, facial expression) based on the face image signal input from the face detection unit 30. As characteristic information (age information) representing age, for example, information not only representing a specific age but also a predetermined age group (for example, teens, 20s, childhood, youth, youth, later years) Good. The characteristic information (gender information) representing gender is information representing male or female. The feature information (expression information) representing a facial expression is information representing, for example, whether or not you are laughing. The facial expression information may include a smile level indicating the degree of smile.

  In order to generate age information and gender information, the feature information analysis unit 37 stores, for example, a facial image signal representing an image of a human face whose age is known or an image of a human face whose sex is known. A storage unit. The feature information analysis unit 37 calculates an index value between the detected facial region image and the facial image signal read from the storage unit, and determines age information or gender information based on the calculated index value. For example, a method may be used in which similarity is calculated as an index value, and age information or gender information that maximizes the similarity is determined. In addition, a method may be used in which a Gabor feature amount is calculated as an index value, and age information or gender information is determined using an SVM (Support Vector Machine, support vector machine).

  In order to generate facial expression information, the feature information analysis unit 37 represents, for example, the contour information of each component (for example, eyes and mouth) of the image information representing the image of a smiling human and the position of the component. A storage unit that stores location information in advance is provided. The feature information analysis unit 37 generates the contour information of each component and the position information of the component from the detected facial region image, and collates the contour information and the position information read from the storage unit.

Thereby, the feature information analysis unit 37 estimates feature information representing age, sex, or facial expression. The method for estimating the feature information is not limited to the above, and any method may be used as long as it estimates the age, sex, or facial expression from the image information representing the face.
The feature information analysis unit 37 outputs the generated feature information to the detection information output unit 41 as part of the detection information.

[Determination of operator]
Next, an operator determination process will be described.
Based on the three-dimensional face position information input from the face detection unit 30, the operator determination unit 39 determines a user who is located in a predetermined operable distance area among users who have detected a face area as an operator. (See FIGS. 1 and 2). The operator discriminating unit 39 discriminates, as an operator, a user whose face is located at a position where the distance represented by the three-dimensional face position information is shorter than the predetermined upper limit of the operable distance, for example. . Thereby, an operator is discriminated from the users whose face detection unit 30 has detected the face. On the other hand, the operator discriminating unit 39 discriminates a user who is located outside a predetermined operable area as an inoperable person.

  The operator determination unit 39 determines whether a part of the user's body (for example, a hand) located in the operable area is located in the operation start detection range. The operation start detection range is a defined area that is included in the operable area and is narrower than the operable area. The operator determination unit 39 determines an operation start detection range based on the two-dimensional face position information input from the face detection unit 30 and the two-dimensional eye position information input from the eye position detection unit 31. As a result, it is detected that the operation from the operator has been started, and at the same time, it is possible to avoid receiving operations from a plurality of people at the same time and to accept the operation of only one operator.

Next, an example of the operation start detection range will be described.
FIG. 8 is a conceptual diagram illustrating an example of the operation start detection range.
FIG. 8 shows the front of the operator 13 on the left side and the left side of the operator 13 on the right side. The operation start detection area 133 is, for example, a line segment representing the height of both eyes connecting the left eye 131-1 and the right eye 131-2 included in the image of the operator 13 represented by the video signal input from the imaging device 11. This is a region including 134. That is, the region is sandwiched between a line segment 135 parallel to the line segment 134 and in contact with the upper end of the face 132, and a line segment 136 parallel to the line segment 134 and in contact with the lower end of the face 132. Here, the operator determination unit 39 calculates an operation start detection region based on the two-dimensional face region information input from the face detection unit 30 and the two-dimensional eye position information input from the eye position detection unit 31.

Returning to FIG. 7, when the representative point represented by the hand position information input from the hand position detecting unit 32 is within the range of the operation start detection area, the operator determining unit 39 locates the hand at the representative point, The user whose face is located in the operation start detection area is determined as the operator 13. In that case, the operator discriminating unit 39 outputs an operation start signal indicating that the operation has started based on the detection information relating to the operator to the detection information output unit 41. That is, the operator discriminating unit 39 determines the user who moves his / her hand to the same height as the face as the operator 13 by defining the operation start detection area in this way.
When the representative point represented by the hand position information input from the hand position detector 32 is outside the range of the operation start detection area, the operator determination unit 39 determines that the operator 13 who has placed his hand at the representative point It is determined that the operation has been completed. In that case, the operator discriminating unit 39 outputs an operation end signal indicating that the operation has ended based on the detection information relating to the operator to the detection information output unit 41. In other words, the display device 10 is connected to the operator 13 during a period from when the operator determination unit 39 outputs an operation start signal for a certain operator 13 to the detection information output unit 41 until an operation end signal is output. An operation input based on the determined user's hand shape is received. Even if the representative point represented by the hand position information for another user is within the range of the operation start detection region, the display device 10 does not accept an operation from another user during that time.

  The operator discriminating unit 39 checks the presence / absence of another user located in the operable area, and when it is determined that there is another user, as described above, the other user who is determined to be present is the operator 13. Determine whether or not. When there are a plurality of other users, the operator determination unit 39 determines one user whose representative point represented by the hand position information is closest to the center of the operation start detection area as the operator 13. Thereby, the display device 10 accepts an operation input from only one new operator 13 and does not accept an operation input from two users at the same time.

  In FIG. 8, the operator is taking a horizontal posture (for example, standing) with respect to the bottom surface. However, the operator does not always take such a posture. For example, the operator may lie on the bottom surface. Even in such a case, the operator determination unit 39 can make a stable determination regardless of the operator's posture by determining the operation start detection range and determining the operator 13 as described above. Detection can be avoided. This will be described next.

FIG. 9 is a conceptual diagram illustrating another example of the operation start detection range.
In the example shown in FIG. 9, the operator 13 is lying on the bottom surface 137. At this time, the angle θ formed by the line segment 134 connecting the left eye 131-1 and the right eye 131-2 and the bottom surface 137 is an angle closer to 90 ° than 0 ° (for example, 20 °).
In this case, the operator determination unit 39 determines a line segment 134 that connects the left eye 131-1 and the right eye 131-2 based on the two-dimensional eye position information input from the eye position detection unit 31. Based on the two-dimensional face area information input from the face detection unit 30, the operator determination unit 39 touches the upper end of the face and is parallel to the line segment 134, and touches the lower end of the face and is parallel to the line segment 134. Minute 136 is defined. The operator determination unit 39 determines an area between the line segment 135 and the line segment 136 as an operation start detection range 133. Thus, the operator determination unit 39 determines the operation start detection range 133 based on the position of the face that is a part of the body of the operator 13, and sets the positional relationship with the hand that is the other part of the body related to the operation input. Based on this, an operator who receives the operation is determined.

  In the above description, the operator determination unit 39 has been described with respect to the example in which the operation start detection range is determined based on the two-dimensional position information and the two-dimensional face area information. However, the present embodiment is not limited thereto. The operator determination unit 39 may determine the operation start detection range based on the three-dimensional position information representing the three-dimensional coordinates in the object space and the three-dimensional face area information. In this case, the operator discriminating unit 39 starts the operation start detection range in which the hand position on the subject space represented by the three-dimensional hand position information is in the depth direction (operation start detection start position (depth direction) in FIG. 8). It may be determined whether or not the operator accepts the operation depending on whether it is also included in the start detection area (depth direction). Thereby, it is possible to avoid erroneous detection of the relationship between the operation start detection range and the hand position in consideration of the coordinates in the depth direction.

Returning to FIG. 7, detection information is input to the detection information output unit 41 from the face detection unit 30, the feature information analysis unit 37, and the operator determination unit. When an operation start signal is input from the operator determination unit 39, the detection information output unit 41 outputs detection information of the operator related to the input operation start signal to the control unit 22. When the operation end signal is input from the operator determination unit 39, the detection information output unit 41 ends the output of the operator detection information related to the input operation end signal.
Note that the user information analysis unit 201 is not limited to the above-described method and feature amount, and may detect information related to the user's feature and the operation instructed by the user based on the input video signal.

[Guidance image display]
The control unit 22 reads the guide image signal from the information DB 21 and outputs the read guide image signal to the display unit 12. The display unit 12 displays a guide image based on the guide image signal input from the control unit 22.
The guidance image is an image including an image representing the shape of the hand for each type of processing performed by the display device 10. That is, the guidance image indicates that the operator can execute one of the displayed hand shapes and instruct the display device 10 to perform a corresponding process. The hand shape includes not only a stationary shape but also a hand motion that changes with time.
The time when the control unit 22 outputs the guidance image signal to the display unit 12 is, for example, the time when the detection information is input from the detection information output unit 41. This time corresponds to a time when the operator 13 raises his / her hand in the operation start detection area (see FIGS. 8 and 9).

Here, an example of the guidance image according to the present embodiment will be described.
FIG. 10 is a conceptual diagram illustrating an example of a guidance image.
In FIG. 10, the rectangular outer frame represents the display area of the screen of the display unit 12. A rectangle indicated by a broken line below the display unit 12 represents the guide image 80. The guide image 80 includes images 81-84 representing hand shapes or actions for each of the four types of processing. A character string representing the content of the corresponding process is displayed immediately above the images 81-84. For example, the image 81 represents that both hands are extended and reciprocated to the left and right, and a character string indicating power-off such as “turn off the power” is displayed immediately above the image 81. The image 82 represents moving up and down while grasping all fingers of both hands, and displaying an electronic program guide (EPG, Electronic Program Guide) “View the program guide” directly above the image 82. A character string is displayed. The image 83 represents that the index finger and middle finger of both hands are raised upward and moved in the depth direction, and a character string representing that the volume is increased is displayed immediately above the image 83. . The image 84 represents that the forefinger and middle finger of both hands are raised downward and moved in the depth direction, and a character string representing that the volume is set to “reduce sound” is displayed directly above the image 84. . That is, even the operator 13 who is not familiar with the operation can grasp the shape and movement of the hand to be taken in order to perform the intended operation by viewing the guidance image.
In addition, the guide image is not limited to this, and may be a guide image composed of a moving image (animation) that visually represents the shape and motion of the hand. For example, the image 81 may be displayed in the guide image as a moving image (animation) that moves both hands open and reciprocates left and right.

[Operation by hand shape recognition]
The control unit 22 extracts three-dimensional face position information, finger image signals, number information, two-dimensional fingertip position information, attention position information, and feature information from the detection information input from the detection information output unit 41. Hand shape information representing the shape of the hand is configured to include the finger image signal, the number information, and the two-dimensional fingertip position among the extracted information. The hand shape information may be information representing a temporary or stationary hand shape, that is, a posture, or may be information representing a hand shape that changes over time, that is, an action.
The control unit 22 includes a storage unit that stores in advance hand shape information in association with each type of process to be operated. The control unit 22 reads the hand shape information from the storage unit, and collates the read hand shape information with the input hand shape information. The control unit 22 executes processing corresponding to the hand shape information that has been successfully verified. For example, in the case where the input hand shape information represents moving up and down while grasping all fingers of both hands as shown in the image 82, the control unit 22 reads the program guide information from the information DB 21 and reads it out. The program guide information is output to the display unit 12.
The shape of the hand described above is not limited to that illustrated in FIG. 10, and may be specified as an instruction for the type of processing of the display device 10. For example, the shape may be such that the number of fingers being stretched, the direction of the hand, the tip of the thumb and the tip of the index finger are in contact. In the present embodiment, the shape of the other part of the body is not limited to the shape of the operator's hand.

[Control of guide image display timing]
For an operator who is familiar with the operation, it may be troublesome to perform the operation after the guidance image 80 is displayed. For example, the control unit 22 may not display the guidance image when the detection information is input from the detection information output unit 41. However, the control unit 22 displays a guide image when the verification of the hand shape information read from the storage unit and the input hand shape information fails. Accordingly, it is possible to prompt the operator to confirm the operation content for performing a desired process when the verification fails.
Moreover, the control part 22 may display a guidance image at a different timing for each operator. For example, I.D. There is a case where a guide image is displayed (I. guide image display). In addition, there is a case where detection information is received from the beginning without displaying a guidance image for an operator who is proficient in the operation (II. Direct operation).

Next, timing when the control unit 22 according to the present embodiment performs processing will be described.
FIG. 11 is a schematic diagram illustrating the timing at which the control unit 22 according to the present embodiment performs processing.
In FIG. 11, the horizontal axis represents time, and the vertical axis represents I.D. Guide image display, II. Represents a direct operation. The left end of the horizontal axis represents the time point when the detection information is first input by the control unit 22.
I. Regarding the guide image display, the control unit 22 waits for the output of the guide image signal until the predetermined time T1 elapses after the detection information is first input. The control unit 22 outputs a guide image signal to the display unit 12 when the time T1 has elapsed. When the detection information is input from the detection information output unit 41, the control unit 22 executes a process corresponding to the input detection information, and the display unit 12 erases the displayed guide image.
II. For direct operation, the control unit 22 waits for detection information from the detection information output unit 41 after detection information is first input. When the detection information is input from the detection information output unit 41, the control unit 22 executes a process corresponding to the input detection information.

In order to distinguish the timing at which the control unit 22 displays the guidance image, for example, the user information analysis unit 201 includes a user recognition unit, recognizes the operator, and determines the proficiency level for each recognized operator. Also good. Here, the user recognition unit includes a storage unit that stores a facial image signal for each operator in advance, and recognizes an operator that is collated with the facial image signal input from the face detection unit 30. When executing the process corresponding to the detected information by the recognized operator, the control unit 22 outputs a process execution signal indicating that the process has been executed to the user recognition unit. The user recognizing unit accumulates the number of times the process execution signal is input from the control unit 22, and outputs the number of times information indicating the accumulated number of times to the control unit 22 through the detection information output unit 41. When the number of times represented by the number information input from the user recognition unit does not exceed a preset number (for example, 30 times), the control unit 22 determines that the operator is not proficient and displays a guidance image. indicate. That is, the control unit 22 performs the above-described I.D. Processing is performed at the timing in accordance with the guide image. When the number of times represented by the number information input from the user recognition unit exceeds a preset number, the control unit 22 determines that the operator has mastered and displays a guidance image. That is, the control unit 22 performs the above II. Processing is performed at the timing in accordance with the direct operation. In the present embodiment, the higher the skill level of the operator, the shorter the time T1 until the guidance image is displayed.
Thereby, the guidance image is not displayed for the operation input by the operator who is familiar with the operation, and the corresponding processing is performed. Therefore, in the display device 10 according to the present embodiment, it is possible to comfortably perform an operation input regardless of whether or not the operation is proficient.

  The information DB 21 stores a plurality of types of guidance image signals in association with predetermined distance sections, and the control unit 22 guides corresponding to the distance sections including the distance represented by the extracted three-dimensional face distance information. An image signal may be read out. Then, the control unit 22 outputs the control-read guidance image signal to the display unit 12, and the display unit 12 displays the guide image represented by the guide image signal input from the control unit 22. For example, as the distance of the operator 13 from the display device 10 is longer, the number of types of processing included in one guide image is decreased, and the screen area (number of pixels) representing operations for each processing is increased. Is stored in the information DB 21. Further, a guide image signal representing an image in which characters for displaying the processing contents are enlarged may be stored. In this way, the operator 13 can clearly grasp the operation content that is displayed greatly even when the distance from the display unit 12 is increased.

  The information DB 21 stores a plurality of types of guide image signals in association with feature information, and the control unit 22 reads the guide image signal corresponding to the extracted feature information and displays the read guide image signal on the display unit 12. May be output. For example, as the age group of the operator 13 is higher, the number of types of processing included in one guidance image is decreased, and a guidance image signal representing an image with an increased screen area (number of pixels) representing an operation for each processing. Is stored in the information DB 21. Moreover, you may memorize | store the guidance image signal showing the image which made the character which displays the processing content larger than another age group. In this way, even an older operator 13 can clearly grasp the operation content. In addition, an image guidance signal representing an image whose volume adjustment step size is larger than that of other age groups is stored in the information DB 21, and the control unit 22 may perform processing related to the volume adjustment based on the step size. . Thereby, it is possible to increase the volume adjustment amount for the operator 13 with a higher age than other age groups for convenience.

For example, when the gender of the operator 13 is female, a guide image signal whose background is expressed in a warm color system such as red or pink, or a guide image signal expressed by a character in an animated movie may be stored. . In this way, even if the operator 13 is a woman, the display device 10 can be operated with a sense of familiarity.
In addition, for the age group of the operator 13 who is a child (for example, 10 years old or younger), a guide image signal representing characters for displaying the processing contents in hiragana or a guide image signal representing a character in an animated movie is displayed. It may be memorized. In this way, even if the operator 13 is a child, the displayed operation content can be grasped, and the display device 10 can be operated with a sense of familiarity.
Further, for the age group of the operator 13 who is a child or an elderly person (for example, 60 years old or older), a guide image signal representing the processing content as a moving image may be stored. Thereby, even a child or an elderly person can view and intuitively grasp the moving image on which the operation method is displayed.

  The control unit 22 avoids an erroneous operation by not detecting a minute hand shape change that is not intended by the operator 13. For this purpose, the control unit 22 determines that the change has been detected when the shape of the hand changes from a predetermined distance or position threshold at predetermined time intervals, and then checks the above-described hand shape information. Do. Here, the control unit 22 increases the distance threshold as the distance represented by the extracted three-dimensional face distance information is longer. Accordingly, as the distance from the display device 10 is longer, the control unit 22 cannot accept the change unless the change in the hand shape by the operator 13 is increased. As a result, as the distance from the display device 10 is longer, the operator 13 is prompted to perform an operation input with a larger change in hand shape, and the influence of the resolution of the imaging device 11 can be reduced.

The control part 22 may make the threshold value of the distance for detecting the change of a hand shape smaller than the other age group with respect to the age group of the operator 13 who is a child. Thereby, even the child whose hand size is not sufficiently developed can comfortably operate the display device 10. Moreover, the control part 22 may lengthen the time interval for detecting, so that the age group of the operator 13 is high. Thereby, even the elderly person whose operation | movement is slow can operate the display apparatus 10 comfortably.
As a result, in this embodiment, the display of the guidance image and processing related to other operations are changed for each feature information representing the feature of the operator, so that a comfortable operation can be realized regardless of the feature of the operator. it can.

[Process flow]
Next, data input processing according to the present embodiment will be described.
FIG. 12 is a flowchart showing data input processing according to the present embodiment.
(Step S900) The imaging units 110a and 110b each capture a video and output the captured video signal to the distance calculation unit 200. The imaging unit 110a outputs the video signal to the face detection unit 30 and the hand position detection unit 32 of the user information analysis unit 201. Thereafter, the process proceeds to step S901.
(Step S901) The distance calculation unit 200 calculates the distance from the imaging device 11 to the operator based on the video signals input from the imaging units 110a and 110b, for example, using a stereo matching method, and the calculated distance information. Is generated. The distance calculation unit 200 outputs the generated distance information to the face detection unit 30, the eye position detection unit 31, the hand position detection unit 32, and the hand shape / fingertip position detection unit 33 of the user information analysis unit 201. Then, it outputs to step S902.

(Step S902) The face detection unit 30 detects an area representing an image of the face of the operator represented by the video signal input from the imaging unit 110a. The face detection unit 30 generates two-dimensional face area information based on the detected face area. The face detection unit 30 extracts the distance value related to the pixel of the two-dimensional coordinate represented by the two-dimensional face area information from the distance information input from the distance calculation unit 200. The face detection unit 30 converts the distance value corresponding to the above-described two-dimensional coordinates into three-dimensional coordinates in the object space, and generates three-dimensional face position information. The face detection unit 30 outputs a face image signal representing the detected face image to the feature information analysis unit 37 and the eye position detection unit 31. The face detection unit 30 outputs the generated 3D face position information and 2D face area information to the operator determination unit 39. The face detection unit 30 outputs the generated three-dimensional face position information to the detection information output unit 41 as part of the detection information.

  The eye position detection unit 31 detects an eye region from the facial image represented by the facial image signal input from the face detection unit 30. The eye position detection unit 31 calculates eye position coordinates based on the detected eye region. The eye position detection unit 31 extracts the distance value at the pixel located at the detected eye position coordinates from the distance information input from the distance information calculation unit 200. The eye position detection unit 31 converts the set of the calculated two-dimensional eye position coordinates and the extracted distance value into three-dimensional eye position coordinates in the object space, and generates three-dimensional eye position information. The eye position detection unit 31 outputs the three-dimensional eye position information representing the calculated three-dimensional eye position coordinates to the attention position detection unit 35 and the operator determination unit 39. The eye position detection unit 31 outputs an eye region signal representing the detected image of the eye region and two-dimensional eye position information representing the calculated two-dimensional eye position coordinates to the operator determination unit 39. Thereafter, the process proceeds to step S903.

(Step S903) The hand position detection unit 32 detects an area representing the image of the operator's hand represented by the video signal input from the imaging unit 110a, and calculates a two-dimensional coordinate value representing the detected position of the hand. The hand position detection unit 32 extracts a distance value corresponding to the calculated coordinate value from the distance information input from the distance calculation unit 200, and sets a set of distance values corresponding to the calculated two-dimensional coordinate value in the subject space. Three-dimensional hand position information is generated by converting into three-dimensional coordinates. The hand position detection unit 32 outputs a hand image signal representing the detected image of the hand region and hand position information representing the calculated two-dimensional coordinate value of the representative point to the hand shape / fingertip position detection unit 33. The hand position detection unit 32 outputs the hand position information to the operator determination unit 39.

  The hand shape / fingertip position detection unit 33 detects the shape of the hand based on the hand image signal input from the hand position detection unit 32 and the hand position information. The hand shape / fingertip position detection unit 33 searches for an image of the finger area based on the detected hand shape, and counts the number of fingers. The hand shape / fingertip position detection unit 33 detects the fingertip position of each finger as a two-dimensional coordinate, and extracts the distance value of the pixel located at the detected two-dimensional coordinate from the distance information input from the distance calculation unit 200. . The hand shape / fingertip position detection unit 33 generates a three-dimensional fingertip position information representing a three-dimensional coordinate in the subject space from the set of the extracted distance value and the two-dimensional coordinate at the fingertip. The hand shape / fingertip position detection unit 33 outputs the generated three-dimensional fingertip position information to the attention position detection unit 35. The hand shape / fingertip position detection unit 33 detects the finger image signal representing the detected finger region, the number information representing the number of fingers, and the two-dimensional fingertip position information representing the two-dimensional coordinates on the fingertip as detection information. Output to the output unit 41. Thereafter, the process proceeds to step S904.

(Step S904) Based on the three-dimensional face position information input from the face detection unit 30, the operator determination unit 39 selects a user who is located in a predetermined operable region from among users who have detected a facial region. It is determined as The operator determination unit 39 determines the operation start detection range based on the two-dimensional face area information input from the face detection unit 30 and the two-dimensional eye position information input from the eye position detection unit 31. The operator determination unit 39 determines whether or not the operator's hand located in the operable region is located in the operation start detection range. Thereby, it is detected that the operation from the operator is started. If it is determined that the operation has started (step S904 Yes), the process proceeds to step S905. If it is determined that the operation has not been started (No in step S904), the process proceeds to step S900.
As described above, the operator determination unit 39 may determine the operation start detection range based on the three-dimensional position information representing the three-dimensional coordinates on the object space and the three-dimensional face area information. In this case, the operator discriminating unit 39 starts the operation start detection range in which the hand position on the subject space represented by the three-dimensional hand position information is in the depth direction (operation start detection start position (depth direction) in FIG. 8). You may make it discriminate | determine whether it is an operator who receives operation by whether it is included in a start detection area | region (depth direction). Thereby, it is possible to avoid erroneous detection of the relationship between the operation start detection range and the hand position in consideration of the coordinates in the depth direction.

(Step S905) The control unit 22 extracts a finger image signal, number information, and a two-dimensional fingertip position as hand shape information representing the shape of the operator's hand from the detection information input from the detection information output unit 41. Thereafter, the process proceeds to step S906.
(Step S906) The control unit 22 reads the hand shape information from the storage unit, and collates the read hand shape information with the input hand shape information. When the collation between the read hand shape information and the input hand shape information is successful (step S906 Yes), the control unit 22 proceeds to step S910. If the verification of the read hand shape information and the input hand shape information fails (No in step S906), the control unit 22 proceeds to step S907.

(Step S907) The control unit 22 determines whether or not the guide image output standby time (T1) has elapsed. When the guide image output standby time (T1) has elapsed (step S907 Yes), the process proceeds to step S908.
When the guide image output standby time (T1) has not elapsed (No in Step 907), the control unit 22 proceeds to Step S900.
(Step S908) After the predetermined time T1 has elapsed, the control unit 22 reads the guide image signal from the information DB 21, and outputs the read guide image signal to the display unit 12. The display unit 12 displays a guide image based on the guide image signal input from the control unit 22. Thereafter, the process proceeds to step S900.

(Step S910) When the guide image is displayed, the display unit 12 deletes the guide image. The control unit 22 executes processing corresponding to the hand shape information that has been successfully verified. This process is, for example, one of operations related to the function of the display device 10 described above. Further, an image signal representing an image (for example, a screen related to a user interface) displayed when performing the operation is read from the information DB, and the read image signal is output to the display unit 12. Thereafter, the process proceeds to step S911.

(Step S911) The control unit 22 determines whether or not the operation by the operator is finished. For example, the control unit 22 determines that the operation is completed when an operation input indicating power-off is input. If it is determined that the operation has not ended (No at step S911), the process proceeds to step S900. If it is determined that the operation has been completed (step S911 Yes), the data input process is terminated.

In the above description, the distance information indicating the distance to the operator's body is acquired based on the image signals input from the two imaging units 110a and 110b. It is not limited to. For example, one of the imaging units 110a and 110b is replaced with a distance measurement unit that acquires distance information to the subject by other methods, and the distance measurement unit inputs the acquired distance information to the user information analysis unit 201. May be.
This distance measuring unit uses, for example, a TOF (Time of Flight) method as a method for acquiring distance information. In the TOF method, for example, an LED (Light Emitting Diode) is used as a light source, and the arrival time from when a light beam is emitted until the reflected light reflected from the subject is received is measured. By measuring the arrival time for each predetermined divided area, it is possible to acquire distance information for each plane position in the object space. The light source is, for example, an infrared light emitting diode that emits infrared light that cannot be seen by human eyes, but is not limited thereto. For example, a laser light source capable of performing pulse irradiation may be used. Further, a phase modulator that can modulate the amplitude or phase of the light beam may be provided, and the arrival time may be calculated based on the phase difference between the emitted light and the reflected light.

  In the above description, the example in which the control unit 22 performs the type of processing corresponding to the hand shape information as the detection information input from the user information analysis unit has been described, but the present embodiment is not limited thereto. In the present embodiment, user information including information representing the shape of not only a hand but also other parts such as an arm, a neck, a head, and a torso may be used as a part of the body. In addition, the shape in the user information is not limited to a stationary shape, and may include an action and a posture that are changes in shape over time. As the user information, three-dimensional position information may be used instead of the two-dimensional position information.

As described above, according to the present embodiment, the first position information representing the position of a part of the body for each user represented in the video captured by the imaging device 11 is acquired. In addition, according to the present embodiment, the user is determined based on the first position information, and user information including information representing the shape of a part of the user's body shown in the image captured by the imaging device 11 is detected. Then, processing corresponding to the detected user information is executed. Therefore, even if a plurality of operators attempt an operation by changing the shape of the body, it is possible to perform a comfortable operation without causing the processes related to the plurality of operations to collide at the same time.
In addition, according to the present embodiment, the feature information representing the feature of the user is estimated based on the presence / absence of the display of the guidance image or the image captured by the imaging device at the timing when the process corresponding to the user information is executed. A guide image with a different display mode is displayed. Therefore, a comfortable operation can be performed for a user having various features.

Note that a part of the display device 10 in the above-described embodiment, for example, the distance calculation unit 200, the face detection unit 30, the eye position detection unit 31, the hand position detection unit 32, the hand shape / fingertip position detection unit 33, and the target position detection. The unit 35, the feature information analysis unit 37, the operator determination unit 39, the detection information output unit 41, and the control unit 22 may be realized by a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” is a computer system built in the display device 10 and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.
Moreover, you may implement | achieve part or all of the display apparatus 10 in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each functional block of the display device 10 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

DESCRIPTION OF SYMBOLS 10 ... Display apparatus, 2a ... Data input device, 11 ... Imaging device, 20 ... Image processing apparatus,
200: distance calculation unit, 201: user information analysis unit, 30 ... face detection unit, 31 ... eye position detection unit,
32 ... Hand position detection unit, 33 ... Hand shape / fingertip position detection unit, 35 ... Attention position detection unit,
37 ... feature information analysis unit, 39 ... operator discrimination unit, 41 ... detection information output unit,
2b ... display control device, 12 ... display unit, 21 ... information DB, 22 ... control unit

Claims (15)

A first position detection unit that acquires first position information indicating a position of a part of the body of each user represented in an image captured by the imaging device;
A user information analysis unit that determines a user based on the first position information and detects user information including information representing a shape of a part of the body of the user represented in an image captured by the imaging device;
A data input device comprising: a control unit that executes processing corresponding to user information detected by the user information analysis unit. The user information analysis unit includes a second position detection unit that acquires second position information in which another part different from a part of the body for each user is located,
The data input device according to claim 1, wherein the control unit executes processing corresponding to the user information for a user in which the first position information and the second position information have a predetermined relationship. The data input device according to claim 1, wherein the part of the body is a hand. The data input device according to claim 2, wherein the other part different from the part of the body is at least one of a face and an eye. The data input device according to claim 4, wherein the imaging device is installed at a position higher than a height of the face or eyes. A display unit for displaying video captured by the imaging device;
The data input device according to claim 5, wherein the imaging device is installed at a position higher than the display unit. The imaging device includes a plurality of imaging units provided at different positions,
The first position detection unit acquires first position information including distance information based on images captured by the plurality of imaging units,
The data input apparatus according to claim 6, wherein the user information analysis unit determines a user based on the first position information acquired by the first position detection unit. A display unit for displaying a guide image representing a relationship between a process to be executed and a shape of a part of the user's body after the user information analysis unit has passed a predetermined time from the start of detection of the user information; The data input device according to claim 1, wherein   When the control unit executes a process corresponding to the user information detected by the user information analysis unit before the predetermined time elapses after the user information analysis unit detects the user information, The data input device according to claim 8, wherein the display unit does not display the guide image. The user information analysis unit estimates feature information representing the features of the user based on the video captured by the imaging device,
The data input device according to claim 8, wherein the display unit displays a guide image having a different display mode according to the feature information.   A display device comprising the data input device according to claim 1. In the data input method in the data input device for inputting data based on the video imaged by the imaging device,
A first process in which the data input device acquires first position information indicating a position of a part of a body for each user represented in an image captured by an imaging device;
The data input device determines a user based on the first position information, and detects user information including information representing a shape of a part of the user's body represented in an image captured by the imaging device. And the process
The data input method includes: a third step of executing processing corresponding to the detected user information. The second process includes a fourth process of acquiring second position information where another part different from a part of the body for each user is located,
The data input method according to claim 12, wherein in the third step, a process corresponding to the user information is executed for a user having a predetermined relationship between the first position information and the second position information. . To the computer of the data input device that inputs data based on the image captured by the imaging device,
A first procedure for acquiring first position information representing a position of a part of a body for each user represented in an image captured by an imaging device;
A second procedure for determining a user based on the first position information and detecting user information including information representing a shape of a part of the body of the user represented in an image captured by the imaging device;
A third procedure for executing processing corresponding to the detected user information;
Data input program for executing The second procedure includes a fourth procedure for obtaining second position information where another part different from a part of the body for each user is located,
15. The data input program according to claim 14, wherein the third procedure executes a process corresponding to the user information for a user having a predetermined relationship between the first position information and the second position information. .

Images