JP2014081788A - Hand gesture recognition device and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology to recognize hand gestures of a plurality of users with high accuracy.SOLUTION: A hand gesture recognition device recognizes faces of users photographed with a camera to determine a hand gesture detection area unique to each user, and recognizes movement of a hand image in the hand gesture detection area as hand gesture. When the hand gesture recognition device pays attention to one of the hand gesture detection areas, for a hand image coming into the attention hand gesture detection area from the outside, deletes the image from the hand images in the attention hand gesture detection area, and recognizes hand gesture from the remaining hand images.

Description

  The present invention relates to a technique for recognizing an operator's hand gesture and performing a corresponding process.

  In recent years, a variety of three-dimensional sensing technologies have been developed, and an apparatus for recognizing an operator's hand movement as an instruction operation and performing an information operation has been developed. When the technology is used for home appliances such as a TV, it is possible to perform an operation from a remote location without a remote controller.

  However, in many homes, there are a plurality of people, and in a small space, they often sit next to each other. In view of this point, there is a document shown below that discloses a technique for handling a plurality of people's space gestures.

  In the gesture recognition method of Patent Document 1, a plurality of subjects are imaged by an omnidirectional visual sensor, and the image processing apparatus equally divides the person range into 3 × 3 in the image processing apparatus. Then, the image is divided into images including the respective subjects, and recognition is performed by handing over to a step of obtaining a time difference image from the divided images.

Japanese Patent No. 3784474

  However, in the gesture recognition method of Patent Document 1, recognition may be difficult in a situation where gesture hand movement interferes next to each other, such as when users are adjacent to each other.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide a technique for recognizing each user's hand gesture with high accuracy even when there are a plurality of users.

In order to solve this problem, for example, the hand gesture recognition device of the present invention has the following configuration. That is,
A hand gesture recognition device for recognizing a movement of a user's hand in an image captured in time series by an imaging means as a hand gesture,
Face detection means for detecting a face area of at least one user in the image picked up by the image pickup means;
A determining unit that determines a region having a preset relative position as a detection region for detecting a hand gesture for each of the face regions detected by the face detecting unit;
When paying attention to one of the detection areas determined by the determining means, an image of a hand that has detected that it has entered inside the target detection area from outside the target detection area from an image captured in time series, Deleting means for deleting from the focus detection area;
Hand gesture recognition means for recognizing a hand gesture in the focus detection area from a time-series hand image in the focus detection area that has been deleted by the deletion means.

  According to the present invention, even if there are a plurality of users, it is possible to recognize each user's hand gesture with high accuracy.

The figure showing the usage pattern in embodiment. The block diagram of the information equipment in an embodiment. The processing block diagram of the example of an embodiment. The figure showing the detected face area | region and the hand gesture start area | region of hand gesture. The conceptual diagram showing the relationship between a hand gesture and a hand gesture start area | region. The flowchart of a hand gesture start area | region determination process. The flowchart of a hand gesture start area | region adjustment process. The flowchart of the process which classifies gesture data. The conceptual diagram showing before the adjustment at the time of hand gesture start area | region adjustment processing, and after adjustment. The processing block diagram of 2nd Embodiment. The figure which shows the example of a display of 2nd Embodiment. The flowchart of a hand gesture start area | region display process. The processing block diagram of 3rd Embodiment. The figure which shows the processing content of 3rd Embodiment. The flowchart of a hand gesture image detection correction process. It is a flowchart of an interpolation process. The figure for demonstrating image deletion and an interpolation process. The figure for demonstrating image deletion and an interpolation process. The figure for demonstrating image deletion and an interpolation process.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 shows a usage pattern of the apparatus of the embodiment. In the figure, reference numeral 1 denotes a camera unit as an imaging means, which is installed on a TV screen and can capture a user's movement in an image captured in time series (a preset frame rate). Reference numeral 2 denotes a TV in which the device of this embodiment is incorporated. A hand gesture performed by the user is photographed by the camera unit, the movement of the hand is recognized as a command, and a channel is changed or a two-screen operation is performed instead of a TV remote control. 3 and 4 represent users sitting on the sofa. In the present embodiment, an example of operating the TV 2 with a hand gesture while two users are sitting is shown.

  FIG. 2 is a block diagram of the information processing apparatus according to the embodiment of the present invention. The apparatus includes a display unit 5, a camera unit 6, a hard disk, a CPU 8, a RAM 9, a ROM 10, and a bus 11 for connecting them. In addition, since the apparatus in the embodiment is characterized by the detection of the user's hand gesture and the processing contents thereof, the circuit for functioning as the TV 2 (for example, a circuit such as a tuner) is not shown in the drawing.

  The display unit 5 is a general display unit, which includes a liquid crystal display element, a liquid crystal control circuit, and a display memory, and is connected to the CPU 8 via the system bus 11. And then. In accordance with an instruction from the CPU 8, image display and character display are executed. The display unit 5 may be a general display device that can be digitally controlled.

  The camera unit 6 may be a general one and includes a sensor such as a CCD, a lens, and a captured image memory, and is connected to the CPU 8 via the system bus 11. Then, in response to an instruction from the CPU 8, the lens is zoomed, an image is taken, and the taken image is transferred to the RAM 9.

  The hard disk 7 is for recording recording data such as a TV program. The CPU 8 is connected to the RAM 9, the ROM 10, the hard disk 7, and the like via the system bus 11, and performs a processing operation by a program stored in the ROM 10. The RAM 9 is used as a work area for the CPU 8.

  FIG. 3 is a functional configuration diagram of the apparatus according to the present embodiment. Each function is realized by several hardware resources and a program executed by the CPU 8.

  The hand gesture image recording unit 12 includes a camera unit 6 and a processing program using the camera unit 6. The hand gesture image recording unit 12 detects a hand gesture image of a plurality of users and sends the hand gesture image to the hand gesture image control unit 14.

  The hand gesture start area determination unit 13 detects a face image from the image captured by the camera unit 6 and determines a hand gesture start area (detection area) for each user based on the position of the face image. Process. This process is performed for the number of detected users. When there are a plurality of hand gesture start areas and the hand gesture start areas are adjacent to each other, a process of adjusting the size of the hand gesture start area is performed based on the position of the adjacent hand gesture start area. Then, the determined hand gesture start area information is sent to the hand gesture image control unit 14. Hereinafter, a user who is a target for recognizing a gesture in the hand gesture start region of interest is referred to as a target.

  The hand gesture image control unit 14 performs a process of removing the image of the hand that has entered the area from the outside of the area and recorded the movement of the hand that has been detected within the detection area of interest. Do. Each user performs a hand gesture operation within each hand gesture start area. The hand gesture image control unit 14 performs a process of removing a noise image caused by a movement of a user's hand that is not a target of the region, disturbance light, or the like. The hand gesture image control unit 14 is determined to be a deleted image determination unit 14a that determines an image that has entered the target detection region from outside the target detection region (regardless of whether or not the target detection region has moved out of the region). The designated image deletion unit 14b is configured to delete an image. Details of the processing will be described later along a flowchart.

  The hand gesture recognition unit 15 extracts a feature amount from the photographed hand movement, performs matching with the feature amount of the hand gesture dictionary, and outputs the closest hand gesture command as a recognition result. For example, the operation of rotating the hand is registered in the gesture dictionary as a gesture command for changing the volume of the TV. For example, the volume is increased when the right rotation is recognized, and the volume is decreased when the left rotation is recognized. Also, for example, the up / down movement of the hand is registered as a gesture command, the TV channel number is added by the hand ascending motion, and the TV channel number is subtracted by the hand descending motion. By registering gesture commands in the gesture dictionary in this way, a device for operating the TV by hand is realized. The hand gesture recognition unit 15 performs control processing (channel change, volume change, etc.) according to the recognized hand gesture, and this processing itself depends on the control target device. In the present embodiment, the description is continued assuming that the control target device is a TV, but the present invention is not limited to this.

  FIG. 4 is a conceptual diagram showing a detected face area and a hand gesture start area of a hand gesture corresponding to the detected face area. Reference numeral 16 in the figure is a rectangular face area detected as the user A. Reference numeral 17 denotes a rectangular face area detected as the user B. Reference numeral 18 denotes a hand gesture start area (detection area of the user A gesture) of the hand gesture set based on the face area 16 of the user A. Reference numeral 19 denotes a hand gesture start area (user B gesture detection area) of a hand gesture set based on the face area 17 of the user B.

  FIG. 5A is a conceptual diagram showing the relationship between a hand gesture and a hand gesture start area. The same reference numerals as those in FIG. 4 have the same meaning. That is, reference numeral 18 denotes a hand gesture start area of the user A's hand gesture, and 19 denotes a hand gesture start area of the user B's hand gesture. Reference numeral 20 denotes the entire image area of the camera. 21 represents the start position of user A's hand gesture, and 22 represents the end position of user A's hand gesture. Reference numeral 23 denotes a start point of the user B's unconscious hand movement, and reference numeral 24 denotes an end point of the user B's unconscious hand movement. If a plurality of people are adjacent to each other in this way, the movement of the hand of the next person may get in the way.

  FIG. 5B is a conceptual diagram showing the relationship between the hand gesture and the hand gesture start area. FIG. 5A is a conceptual diagram when focusing on only the hand gesture start area of user A from FIG.

  FIG. 5C is a conceptual diagram after the hand gesture image data from outside the hand gesture start area is deleted from the state of FIG. Since the movement of the hand of user B shown in FIG. 5B from 23 to 24 enters from outside the hand gesture start area, the data is deleted by the processing unit 14 of FIG. The image data of only the hand gesture movements of the user A of 21 to 22 are used.

  FIG. 6 is a flowchart of the hand gesture start area determination process.

  In step S6-1, the work area and the like are secured and initialized in response to the activation of the apparatus, and the hand gesture start area determination process is started.

  In step S6-2, the hand gesture start area determination unit 13 detects the face of at least one user from the camera image. First, an image as shown in FIG. 4 is taken by the camera unit 6. By extracting an oval area of a certain size or larger from the photographed image, a face candidate area can be detected. A more probable face area can be detected by extracting image feature values from the detected area and matching the face dictionary. Face areas such as reference numerals 16 and 17 in FIG. 4 are detected. The rectangle information of the detected face area is passed to step S6-3.

  In step S6-3, the hand gesture start area determination unit 13 sets each hand gesture start area to a preset relative position from the rectangular area position and size information of the face and the rectangular area position and size of each face. decide. In the example of FIG. 4, the hand gesture start area 18 is determined from the position of the face area 16. For example, two face heights below the face area, the vertical length is the same as the face height, and the horizontal width is 1.5 times the horizontal of the face. This value is an example, and the actual movement of the hand gesture is determined by sampling a plurality of people. In addition, a learning process or the like may be entered, and if there are many misrecognitions, it may be automatically corrected.

  In step S6-4, the hand gesture start area determination unit 13 performs a process of sorting each hand gesture start area on the X axis. In the example of FIG. 4, when the face area 16 is detected after the face area 17 is detected, even if the user A comes after the user B, the hand gesture start area rectangle on the X axis By sorting, processing can be performed in the same order.

  In step S6-5, the hand gesture start area determination unit 13 performs processing for reading data of one hand gesture start area. For example, the example of FIG. 4 and the data of the face area 16 are read. Next, when coming to this step again, the data of the face area 17 is read.

  In step S6-6, the hand gesture start area determination unit 13 performs a hand gesture start area adjustment process. Details will be described with reference to the flowchart of FIG. For example, as shown in FIG. 9, the hand gesture start areas 23 and 24 set at the face position partially overlap. By adjusting it, it is set to a separated position as in the areas 25 and 26. If the position is away from the beginning, there is no need to adjust, so this process is skipped.

  In step S6-7, the hand gesture start area determination unit 13 checks whether the processing of all hand gesture start areas has been completed. If there are two face-detected users, two hand gesture start areas are checked, and if there are three users, three hand gesture start areas are checked.

  In step S6-8, the hand gesture start area determination process ends. The hand gesture start area determination unit 13 sends information on the determined hand gesture start area to the hand gesture image control unit 14.

  FIG. 7 is a flowchart of the hand gesture start area adjustment process. This process represents the detailed process of step S6-6 described above.

  In step S7-1, the hand gesture start area determination unit 13 starts a hand gesture start area adjustment process.

In step S7-2, the hand gesture start area determination unit 13 sets a hand gesture start area of the target hand gesture image. In the example of FIG. 9, the coordinate values Xmin, Xmax, Ymin, and Ymax of the rectangular area 23 are set.
In step S7-3, the hand gesture start area determination unit 13 sets an adjacent hand gesture start area. In the example of FIG. 9, the coordinates of the rectangular area 24 are set. In the description of this embodiment, two hand gesture start areas are targeted. However, in the case of three, when adjusting the middle data, the adjacent data on the left and right are set. adjust. Here, RXmin, RXmax, RYmin, and RYmax are set.

In step S7-4, it is checked whether Xmax> RXmin and Xmax <RXmax. If the condition is met, the process proceeds to step S7-5, and if not, the process proceeds to step S7-7. In the example of FIG. 9, since the area 23 is adjacent on the right side, the process proceeds to step S7-5 to correct the right frame.
When adjacent hand gesture start areas overlap in the X-axis direction, the overlap value is represented by DX.

  In step S7-5, the hand gesture start area determination unit 13 sets the horizontal width DX of the area. Here, DX = RXmin−Xmax. In step S7-6, a new value of Xmax is set. Let Xmax = Xmax-DX.

  In step S7-7, it is checked whether Xmin <RXmax and Xmin> RXmin. If the condition is met, the process proceeds to step S7-8, and if not, the process proceeds to step S7-10 to finish the process. In the example of FIG. 9, since the region 24 is adjacent on the left side, the process proceeds to step S7-8 and the left frame is corrected.

  In step S7-8, the hand gesture start area determining unit 13 sets the area width DX. DX = RXmin-Xmin. In step S7-9, a new value of Xmin is set. Xmin = Xmin + DX. Although the horizontal adjustment process has been described in this flowchart, in some cases, a vertical adjustment process may occur, but this is an adjustment process in which X is replaced with Y.

  In step S7-10, the hand gesture start area adjustment process ends.

  FIG. 8 is a flowchart of processing for selecting gesture data based on the hand gesture start area.

  In step S8-1, the hand gesture image control unit 14 starts a process of selecting gesture data based on the hand gesture start area.

  In the flowchart, the hand gesture start area is set, and this processing is called in a state where image data of the hand of the hand gesture is acquired by the camera unit 6.

  In step S8-2, the deleted image determination unit 14a performs processing for reading the image data in the hand gesture start area into the buffer area in time series. In the example of FIG. 5B, hand image data in the region 18 between the hand images 21 to 22 and the hand images 23 to 24 is read.

  In step S8-3, the deleted image determination unit 14a detects image data of a hand that has crossed the hand gesture start area from the outside to the inside. In the example of FIG. 5B, the hand images 21 to 22 are completed because they are completed in the region 18. On the other hand, since the hand images 23 to 24 cross the boundary region of the hand gesture start region from the outside to the inside, the data of the hand images 23 to 24 is detected.

  In step S8-4, the designated image deletion unit 14b deletes the image data of the hand detected in the previous step from the buffer. In the example of FIG. 5B, the detected image is deleted and the state shown in FIG.

  In step S8-5, the hand gesture recognition means 15 detects an image of a hand that does not move even after a predetermined time has elapsed from the time-series stored image data in the buffer. Then, a process of recognizing a series of movements of the hand until it stops after moving for a certain period of time as one action is performed. In the example of FIG. 5C, the hand image 21 stops at that position for a certain period of time, then moves to the right, slightly moves up, and stops at the position of the hand image 22. Yes.

  In step S8-6, the hand gesture recognition means 15 extracts the movement trajectory of the center coordinates of the time-series hand from the detected stop image. In step S8-7, the hand gesture recognition unit 15 extracts a feature amount from the trajectory data. For example, the trajectory of the hand images 21 to 22 is equally divided into 10 and converted into direction vectors. In the present embodiment, the direction vectorization is performed by dividing the normal direction angle 360 degrees into 8 or 16 equal parts.

  In step S8-8, the hand gesture recognition unit 15 performs a process of determining a gesture code by performing a matching process with the feature amount of the dictionary. If the input data is in the form of “L” and the direction vector data of “L” “1” is stored in the dictionary, the gesture code of “L” is determined.

  In step S8-9, this process ends.

  FIG. 9 is a conceptual diagram showing before and after adjustment in the hand gesture start area adjustment processing. Thus, by performing the processing of the present embodiment, the hand gesture start area can be determined for each user. In each hand gesture start area, the gestures are recognized by excluding the images of the hands entering and exiting from the outside of the area, so even if two adjacent people perform hand gestures, they can comfortably watch TV. Can be realized.

[Second Embodiment]
In the second embodiment, a mechanism for notifying the user of the area of the hand gesture start area and clearly indicating to the user that recognition of the hand gesture can be started when a hand is detected in the hand gesture start area will be described. In the first embodiment, the processing for setting a hand gesture region by internal processing and preventing erroneous recognition has been described. In the second embodiment, by clearly indicating the set area to the user, it is possible to adjust the position where the user performs the gesture, and it is possible to provide a user interface that is easier to understand and more accurate.

  In the second embodiment, by adding a hand gesture start area state display unit 27 to the configuration of the first embodiment, it is possible to notify the user of the hand gesture start area.

  FIG. 10 is a functional configuration diagram of the apparatus according to the second embodiment. The same reference numerals as those in the first embodiment are the same components, and the description thereof is omitted. The hand gesture start region state display unit 27 displays region information representing the hand gesture start region on the screen of the TV 2 in FIG. 1 in order to clearly indicate to the user the hand gesture start region determined by the hand gesture start region determination unit 13. .

  FIG. 11 is a display example of the second embodiment. The TV screen is displayed in two screens, the program of channel A being viewed by user A is displayed in area 28, and the program of channel B being viewed by user B is displayed in area 29. An area 30 indicates a user A hand gesture start area, and an area 31 indicates a user B hand gesture start area. Also in this embodiment, when it is detected that the hand has been stopped for a predetermined time or longer in the detection area, the process proceeds to hand gesture recognition processing. At that time, a predetermined mark (an image of a hand in the figure) is displayed in the frame as in the region 30 to notify the user that the gesture can be recognized at the current hand position. In this example, the program image and the hand gesture start area are shown at different positions, but it is of course possible to display the program image and a part of the image with an overlay. Further, an LED may be provided in the frame portion of the TV screen so that the start of hand gesture recognition is notified by blinking of the LED. A multicolor LED may be used to notify the recognition start, during recognition, and recognition end.

  FIG. 12 is a flowchart of the area display process of the hand gesture start area.

  In step S12-1, the hand gesture start area state display unit 27 starts an area display process of the hand gesture start area.

  In step S12-2, hand gesture start area information determined by the hand gesture start area determination unit 13 is read in the hand gesture start area adjustment processing. In step S12-3, the hand gesture start area state display unit 27 estimates the user position from the information of the camera unit 6. The current position of the user can be estimated from information about the shooting distance and face size from the camera information. In step S12-4, the hand gesture start area state display unit 27 determines the size according to the user position information, and displays a corresponding frame on the screen. The frame serves as a guide indicating the hand gesture start area to the user, and it becomes clear by which position the user recognizes the hand by holding the hand by looking at the guide frame.

  In step S12-5, the hand gesture start area state display unit 27 checks whether or not the all hand gesture start area guide display process has been completed. In the example of FIG. 11, the process ends when two guide frames are displayed. If three people are watching, the above process is repeated until three guide frames are displayed.

  In step S12-6, the camera unit 6 performs processing for reading the current image of the hand gesture start area.

  In step S12-7, the hand gesture start area state display unit 27 checks whether there is a hand in the area. If a skin-like hand-sized area is detected, the process proceeds to step S12-8; otherwise, the process proceeds to step S12-9. In this embodiment, when a skin color area in the size of a hand is detected, a matching process is performed with an image dictionary in which the hand shape is registered. Recognition can be prevented.

  In step S12-8, the hand gesture start area state display unit 27 displays the hand display in the hand gesture start area guide display. It is also possible to cut out flesh-colored pixels detected by photographing with a camera and display them.

  This process is a process of a part that is clearly shown when the user's hand exists in the hand gesture scan start area.

  In step S12-9, the hand gesture start area state display unit 27 checks whether the check of the hands in all hand gesture start areas has been completed. If all checks have been completed, the process proceeds to step S12-10; otherwise, the process proceeds to step S12-6. In the example of FIG. 11, this step ends when the check of the hands in the two hand gesture start areas ends. In step S12-10, the area display process of the hand gesture start area is terminated.

  By performing the processing in this way, the user can be informed of the hand gesture start area and an easy-to-use machine can be realized.

  As mentioned above, although an example of embodiment of this invention was explained in full detail, this invention can take the embodiment as a system, an apparatus, a method, a program, or a storage medium etc., for example. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

[Third Embodiment]
In the third embodiment, an overlap of the hand gesture target hand and another hand is detected, and an image disappearance portion due to the overlap with the other hand is corrected.

  In the first embodiment, a process of setting a hand gesture area for each user and preventing misrecognition that is confused with a gesture of a different user has been described. At this time, there is a case where the target hand gesture and the hand gesture to be deleted overlap. The hand gesture to be deleted when viewed from the camera may be above and under the target hand gesture. At this time, since the target gesture is hidden by the gesture made above and is not captured by the camera, if the upper hand gesture image is deleted, a blank portion exists and the flow of the lower hand gesture image is divided. . If the divided image is recognized as it is, it is different from the normal image, so that it cannot be recognized or a different recognition result is output. For example, even if it is a movement like “L”, if it is divided in the middle, it becomes two gestures “|” “_”. By detecting and correcting a blank portion resulting from the deletion, the moving image of the divided hand gesture image is reproduced and recognized. "|" "_" Is corrected to "L".

  By adding the overlap detection unit 32 and the hand gesture image correction unit 33 to the configuration of the first embodiment, a device capable of correctly recognizing a divided image due to overlap is realized.

  FIG. 13 is a processing configuration diagram of the third embodiment. The same reference numerals as those in the first embodiment denote the same components, and the description thereof is omitted.

  The overlap detection unit 32 detects that the target has fallen because the hand gesture that has entered from outside the frame, the hand gesture of the target, and the hand gesture by a different user's hand that has entered from outside the region intersect. Then, the position information of the gesture in the area at the time of detection (boundary contact point with the image area to be deleted) is sent to the hand gesture image correction unit 33. The hand gesture image correcting unit 33 corrects the hand gesture image that has disappeared by the hand of the adjacent person based on the four points of the boundary contact with the deleted image region. By correcting the lost hand gesture image, the two hand gesture images are connected to form a series of images.

  FIG. 14 is a process display image diagram of the third embodiment. FIG. 14A shows an example of a screen in a state where two gestures overlap. A hand gesture start area 18 in FIG. 14A is an area for detecting a target hand gesture, and a hand gesture start area 19 is an area for detecting a hand gesture of an adjacent user.

  FIG. 14B is a screen example in a state after executing the processing of the first embodiment and deleting the hand gesture image from outside the area. Here, since the target hand is hidden by the adjacent user's hand and there is a portion that is not imaged, the hand gesture image of the detection target user is divided into two.

  FIG. 14C shows an example of a screen after correcting the hand gesture image that has been divided by the processing of the third embodiment.

  FIG. 17 is a display example in which the screen of FIG. 14A is enlarged. The area where the hand moves is represented by the area of the width of the hand. The illustrated areas 35 and 36 are areas where the hand of the target hand gesture has moved. The area 34 is an area where the hand of the adjacent person has moved. Regions 35 and 36 are images in which the same hand has moved, but are divided by the hand of region 34.

  FIG. 18 is a screen example after the image of the hand of the person next to the area 34 is deleted from FIG. Four points 37, 37, 39, and 40 are boundary contacts with the deleted image area. A region 35 is a region from the start position, and a region 36 is a region on the end position side. Point 41 is the position when the hand was present before the hand was at the position of point 37. For example, if the shooting unit of the camera captures 30 frames per second, the position is 1/30 second before. An image of the next hand is estimated on the extension line connecting the points 41 and 37.

  The point 42 is a position when the hand was present before the hand was present at the position of the point 38. The point 43 is a position when the hand is present after the hand is present at the position of the point 39. The point 44 is a position when the hand is present after the hand is present at the position of the point 40.

  FIG. 15 is a flowchart of the hand gesture image detection process under the deleted hand gesture image. This process is called after the hand gesture image from outside the region is deleted by the process described in the first embodiment. FIG. 16 is a flowchart of a process for interpolating blank portions.

  In step S15-1, the overlap detection unit 32 starts a hand gesture image detection process below the deleted hand gesture image. Secure the work area and perform initialization. After deletion of the hand gesture image from outside the area described in the first embodiment, information on the deletion area is provided.

  In step S15-2, the overlap detection unit 32 performs processing for reading the area information of the deleted hand gesture image from the designated image deleting unit 14b. Since it is a moving image area of the hand, it can be considered as a trajectory having a certain width. In the example of FIG. 5, information on the area represented by the movement trajectory of the hand moving from the start point 23 to the end point 24 is passed in an XY coordinate point sequence.

  In step S15-3, the overlap detection unit 32 obtains a contact point between the movement trajectory from the target hand gesture start point image to the end point and the area of the deletion hand gesture image. When the target hand gesture and the deletion hand gesture image intersect, there is a contact point, and the point is obtained. If the target hand gesture and the deleted hand gesture image are separated from each other, this processing is not necessary and is not called. In the example of FIG. 18, the four points of the boundary contact point with the deleted image area and the coordinates thereof are defined as follows. A point 37 (x1, y1) on the left side of the contact point from the start point to the area of the deletion hand gesture image, a point 38 (x2, y2) on the right side, and a point 39 (x3 on the left side of the contact point from the end point to the area of the deletion hand gesture image , y3), right point 40 (the coordinates are (x4, y4).

  In step S15-4, the overlap detection unit 32 checks whether the area surrounded by the contact point from the start point and the contact point from the end point is blank. In the example of FIG. 18, it is checked whether the area surrounded by the points 37, 38, 39, and 40 is blank. If the image of the neighbor's hand is present in the shadow portion of the target image, the target hand gesture image is present as it is at the four points of boundary contact with the deleted image area.

  If it is blank, it is necessary to correct it and the process proceeds to step S15-5. If not blank, the process proceeds to step S15-8 and the process is terminated. If the image of the target hand gesture is above (near) as viewed from the camera unit 6 and the deleted hand gesture image has moved down (back), it is determined in this step that the image is not blank. If the image of the target hand gesture is at the bottom (back) as viewed from the camera, and the deleted hand gesture image has moved up (front), the movement of the hand that was at the bottom is not photographed, resulting in a blank space.

  In step S15-5, the hand gesture image correcting unit 33 sets four points of boundary contacts in the blank area. In the example of FIG. 18, XY coordinate points (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the points 37, 38, 39, and 40 are stored.

  In step S15-6, the hand gesture image correction unit 33 performs a process of interpolating the blank portion. Detailed processing will be described with reference to the flowchart of FIG. In step S15-7, the interpolated hand gesture image is output. The hand gesture image obtained by dividing the regions 35 and 36 shown in FIG. 18 is corrected, and a hand image is added to the region surrounded by the points 37, 38, 39, and 40 as shown in FIG. And 36 are output by correcting the hand images into one hand gesture image.

  If the gesture is recognized, it can be recognized as a “” ”gesture.If it is divided, it cannot be recognized correctly because it is divided into two“ _ ”and“ |. ”In step S15-8, the process ends, Release work area.

  Next, the processing content of step S15-6 will be described with reference to the flowchart of FIG.

  In step S16-1, the hand gesture image correction unit 33 starts a process of interpolating the blank portion. Secure and initialize the work area.

  In step S16-2, the hand gesture image correcting unit 33 stores the coordinates of the four points as the blank area information. In the case of the example of FIG. 18, the coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the points 37 to 40 are stored.

  In step S16-3, the hand gesture image correcting unit 33 detects the direction angles of (x1, y1) and (x2, y2) from the direction of the starting point of the hand, and from (x1, y1) and (x2, y2) Extend line to directional angle. The direction angle is obtained from several points before reaching (x1, y1) and (x2, y2) of the moving point every 1/100 second of the center point of the hand image. It can be assumed that the hand has moved in the same direction from there. In the example of FIG. 18, the position of the hand image 1/100 second before the shooting time of the hand image at the position of the point 37 is the position of the point 41. The extension of the straight line from the point 41 to the point 37 becomes an arrow as shown in the figure. The position of the hand image 1/100 second before the shooting time of the hand image at the position of the point 38 is the position of the point 42. The extension of the straight line in the direction from the point 42 to the point 38 becomes an arrow as shown in the figure.

  In step S16-4, the hand gesture image correcting unit 33 detects the direction angles of (x3, y3) and (x4, y4) from the end direction of the hand, and from (x3, y3) and (x4, y4) Extend line to directional angle. From (x3, y3) and (x4, y4), the direction angle is obtained from several points of the movement point per hundredth of the center point of the hand image in the direction of the end point of the hand, and the opposite direction is obtained. This is because it can be assumed that the hand has moved in the same direction from the opposite direction. In the example of FIG. 18, the position of the hand image 1/100 second after the shooting time of the hand image at the position of the point 39 (x3, y3) is the position of the point 43. The extension of the straight line from the point 43 to the point 39 becomes an arrow as shown in the figure. The position of the hand image after 1/100 second of the shooting time of the hand image at the position of the point 40 (x4, y4) is the position of the point 44. The extension of the straight line from the point 44 to the point 40 becomes an arrow as shown in the figure.

  In step S16-5, the hand gesture image correcting unit 33 connects the line from the start point direction and the line from the end point direction at an intermediate point. Then, an image of the hand at the positions (x1, y1) and (x2, y2) is copied on the line at the same time interval at the timing of the photographing time, and the hand gesture image is interpolated. In the example of FIG. 19, the extension line from the points 41 to 37 and the extension line from the points 43 to 39 are connected at the position of the point 45. The extension line from the points 42 to 38 and the extension line from the points 44 to 40 are connected at the point 46. In the area surrounded by the points 37, 38, 46, 40, 39, and 45, the hand image at the point 37 is copied at the same 1/100 second interval as the shooting timing. Draw at the position and interpolate the hand gesture image. In step S16-6, the process ends. The work area is released and the process proceeds to step S15-6.

  In this way, by performing processing, even if the target hand gesture and the hand gesture to be deleted overlap, a device that correctly recognizes the hand gesture image reproduced by interpolating the hidden image is realized.

  As described above, according to the present embodiment, even if there are a plurality of users, each user's hand gesture can be recognized with high accuracy. Furthermore, it is possible to further improve the recognition accuracy by detecting and adjusting the overlapping of adjacent areas of the hand gesture start areas determined from the position information of each user. In addition, by clearly indicating that the user's hand has been detected in the hand gesture scan start area, it is possible to notify the user that the gesture recognition can be started, and the operation becomes easy. Since the added part can be realized by software, it is not necessary to add another detection part, so that the cost is improved. Even a hand gesture image divided by another user's hand can be recognized by being corrected by the correction unit.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

A hand gesture recognition device for recognizing a movement of a user's hand in an image captured in time series by an imaging means as a hand gesture,
Face detection means for detecting a face area of at least one user in the image picked up by the image pickup means;
A determining unit that determines a region having a preset relative position as a detection region for detecting a hand gesture for each of the face regions detected by the face detecting unit;
When paying attention to one of the detection areas determined by the determining means, an image of a hand that has detected that it has entered inside the target detection area from outside the target detection area from an image captured in time series, Deleting means for deleting from the focus detection area;
Hand gesture recognition means for recognizing a hand gesture in the focus detection area from a time-series hand image in the focus detection area left after being deleted by the deletion means;
A hand gesture recognition device comprising: The region determining means includes
When the face detection means detects face areas of a plurality of users and the detection areas for the respective face areas overlap with each other, the face detection means includes means for adjusting the size of each detection area so that there is no overlap. Item 4. The hand gesture recognition device according to Item 1.   In the detection area determined by the area determination unit, when an image of a hand that does not move even after a preset time has been detected, the hand gesture is detected in the detection area including the image of the hand without movement. The hand gesture recognition device according to claim 1, further comprising an explicit means for clearly indicating that recognition is possible.   4. The hand gesture recognition apparatus according to claim 3, wherein the specifying means includes means for displaying a predetermined mark in a frame corresponding to the detection area in the image captured by the imaging means. The hand gesture recognition means includes:
As a result of deletion by the deletion means, when the image of the hand of the target that recognizes the hand gesture is divided, it includes means for interpolating the divided disappearance portion based on the coordinates of the divided boundary. The hand gesture recognition device according to any one of claims 1 to 4. A control method for a hand gesture recognition device for recognizing movement of a user's hand in an image captured in time series by an imaging means as a hand gesture,
A face detecting step for detecting a face region of at least one user in the image picked up by the image pickup means;
A determining step in which a determination unit determines a region of a preset relative position as a detection region for detecting a hand gesture for each of the face regions detected in the face detection step;
When the deletion means pays attention to one of the detection areas determined in the determination step, the hand that has detected that it has entered inside the target detection area from outside the target detection area from an image captured in time series A deletion step of deleting the image of the target detection area,
A hand gesture recognition step for recognizing a hand gesture in the target detection region from a time-series hand image in the target detection region, which is left after the deletion by the deletion step.
A method for controlling a hand gesture recognition device, comprising:   A non-transitory computer-readable storage medium storing a program for causing a computer having an imaging unit to execute each step according to the method according to claim 6 to cause the computer to function as a hand gesture recognition device.   A computer-readable storage medium storing the program according to claim 7.