JP6460862B2 - Gesture recognition device, system and program thereof - Google Patents

Description

  The present invention relates to a gesture recognition device, a system, and a program for the same, which can perform an interface operation of a personal computer or the like by performing a gesture that can be performed in a movable range of the person even if the person is physically handicapped. It is about.

  Conventionally, various interfaces have been developed so that a disabled person with a physical disability can use the Internet and mail using a computer. For example, a switch for a person whose fingertip moves, an expiration switch used for expiration, voice input software, and the like. However, the degree of obstacles and the range of motion vary from person to person, and the physical condition changes from day to day, so some parties have made the state worse by using an incompatible interface. For this reason, it is often necessary to develop an interface that is tailored to the individual's condition, and there is a problem that an interface that can be comfortably used by many people with disabilities cannot be provided sufficiently.

  In gesture recognition based on image data, gesture detection by a gesture recognition system using distance image data has been proposed. In Non-Patent Document 1, since there are individual differences in the size and movement of gestures, a gesture that can minimize individual differences is considered, and by making everyone a natural and intuitive gesture, A technique for performing illumination control by detection is disclosed.

Mitsunori Miki, 3 authors, “Control of Illumination by Gesture Detection Using Kinect”, 2012 Annual Conference of Japanese Society for Artificial Intelligence (26th), June 12, 2012, p.1-3

  However, since the technique disclosed in Non-Patent Document 1 is a gesture that can be easily recognized by the system side, it is necessary to perform a gesture according to the specification of the system side. However, handicapped people, for example, often do not bend 90 degrees after making their fingers straight when they make finger bending gestures, and their hands must always be placed in front. May not be possible. Furthermore, in the case of persons with disabilities, each symptom varies from person to person, and the parts where reproducible movement is possible, how to move, and the range of movement are different, and depending on the person, movements that are not intended to be reproducible There may be many involuntary movements. Accordingly, the same gesture as that required by a healthy person as requested by the system cannot be performed, so that there is a problem that it cannot be used as an interface when applied to a disabled person.

  The present invention has been made in view of the above-described problems of the prior art, and an object thereof is to provide a gesture recognition device capable of recognizing gestures of an unspecified number of physically disabled persons and performing interface control. .

  In view of the above object, the present invention is a gesture recognition device that recognizes a user's gesture based on distance image data captured by an imaging device and performs interface control associated with the recognized gesture on the interface device. An image capturing unit that captures distance image data output from the imaging device, and a region extraction that detects a predetermined part of the user each time the distance image data is captured and extracts a region where the predetermined part exists from the distance image A part detecting unit that detects a predetermined part from the extracted part region, and a part change detecting part that detects a change in coordinates of the detected predetermined part based on the detected predetermined part in each distance image data When the detected change in the predetermined part is equal to or greater than a predetermined value, the gesture recognition unit that recognizes that the gesture has been performed, and the gesture Once it is recognized that we have to provide a gesture recognition apparatus having an interface control unit that performs interface control associated with the gesture, the.

  The gesture recognition device further includes a gesture determination unit that determines a part associated with interface control and a gesture at the part from one or a plurality of candidate parts of the user, and the part region extraction unit includes one or a plurality of parts. A region where a candidate part exists is extracted from the distance image data, the part detection unit detects one or a plurality of candidate parts from the extracted part region, and the part change detection unit detects a change in the detected candidate part The gesture determination unit determines a part associated with the interface control and a change amount recognized as a gesture in the part based on the change amount detected by the part change detection unit. The gesture recognition unit includes the gesture determination unit. It may be configured to recognize that a gesture has been made when a change amount greater than or equal to the determined amount is detected at the determined site. . By providing the gesture determination unit, it is possible to determine the amount of change for gesture recognition in accordance with the symptom of each individual, so that gesture recognition can be performed with higher accuracy.

  Furthermore, the part region extraction unit extracts candidate regions, the part detection unit detects a rectangle that includes the extracted region, the part change detection unit detects a change in the rectangle, and the gesture determination unit Based on the change detected by the change detection unit, a rectangle that includes the region with the change is determined, and a part associated with the interface control and a change amount recognized as the gesture are determined. The gesture recognition unit is determined by the gesture determination unit. It may be configured to recognize that a gesture has been made when a change amount or more determined in the determined part is detected. Thereby, since gesture recognition can be performed without determining which body part, it is possible to reliably perform gesture recognition without having a gesture model for each body part.

  Furthermore, it has a parameter database that stores a threshold value for detecting a change, and the gesture determination unit stores the amount of change determined to be recognized as a gesture in the parameter database as a threshold value. The threshold value stored in the database may be read out, and when the change in the predetermined part is equal to or greater than the threshold value, the gesture may be recognized as a gesture.

  In addition, the gesture determination unit may determine a threshold value that is recognized as a gesture based on a change amount of the candidate part at a predetermined timing. Thereby, it is possible to prevent the involuntary movement, which is a movement that is not intended by the user, from being erroneously recognized as a gesture.

  The gesture determining unit determines a new threshold value to be recognized as a gesture based on a threshold value determined as a gesture in the past in the candidate part and a newly obtained change amount at a predetermined timing. Also good. Even a user whose symptoms change every day can reliably perform gesture recognition.

  The present invention also provides a gesture recognition program for recognizing a user's gesture based on distance image data and performing interface control associated with the recognized gesture, the distance image data output from the imaging device. An image capturing unit that captures a region, a region region extracting unit that extracts a region where a predetermined region of the user exists every time distance image data is captured, and a region detection that detects a predetermined region from the extracted region region A part change detecting unit that detects a change amount of the detected predetermined part based on the detected predetermined part in each distance image data, and the detected change of the predetermined part is a predetermined value or more, A gesture recognition unit that recognizes that a gesture has been performed, and an interface associated with the gesture when it has been recognized that a gesture has been performed. And characterized by causing a computer to function as a gesture recognition apparatus having an interface controller, a performing over scan control, provides a gesture recognition program.

  The present invention is also directed to an imaging device that captures distance image data, a user's gesture based on the distance image data captured by the imaging device, and interface control associated with the recognized gesture to the interface device. A gesture recognition system comprising: an image capturing unit that captures an image of a user's body, and the gesture recognition device includes an image capturing unit that captures distance image data output from the image capturing device; Each time the distance image data is captured, a region region extraction unit that extracts a region where a predetermined region of the user exists from the distance image, a region detection unit that detects a predetermined region from the extracted region region, and each distance image data A part change detecting unit for detecting a change amount of the detected predetermined part based on the detected predetermined part, and a detected predetermined part A gesture recognition unit that recognizes that a gesture has been performed when the change in position is equal to or greater than a predetermined value; and an interface control unit that performs interface control associated with the gesture when the gesture has been recognized. A gesture recognition system is provided.

  According to the present invention, since a region where a gesture operation can be performed is extracted, a gesture region is detected, and gesture recognition is performed based on the amount of change in motion. Even so, it is possible to reliably recognize the gesture and perform interface control.

  In addition, by providing a gesture determination mode and instructing the user to perform an action at a predetermined timing and moving his / her body, the user can reliably acquire a reproducible gesture that can be used for interface control. By storing the movement of the gesture as a change amount and recognizing the gesture, it is possible to provide a gesture recognition device that matches the individual symptoms of the user.

  Furthermore, in the gesture determination mode, by extracting the moving area and determining the amount of change in that area, it is possible to recognize the gesture without limiting where the moving part is. It can be performed. This eliminates the need to store detection parameters for each body part, and makes it possible to provide a gesture recognition device that is easily adapted to each individual user.

FIG. 1 is a block diagram showing an example of a gesture recognition system including a gesture recognition device according to the first embodiment of the present invention. FIG. 2 is a flowchart showing an example of a flow of interface control processing corresponding to a gesture recognized as recognition of a user's gesture motion performed in the gesture recognition device 20 according to the first exemplary embodiment of the present invention. FIG. 3 is a diagram illustrating an example of data storage contents of the parameter database 284 and the corresponding gesture database 286 stored in the storage unit 280. FIG. 4 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed by the gesture recognition device 20 of the present invention. FIG. 5 is a diagram illustrating an example of distance image data captured when the finger bending gesture is recognized. FIG. 6 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed by the gesture recognition device 20 of the present invention. FIG. 7 is an example of a screen on which the arm tracking process is performed from the distance image data by the particle filter when the arm swing is recognized. FIG. 8 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed in the gesture recognition device 20 of the present invention. FIG. 9 is an example of a screen on which a nose is extracted from the distance image data and a normal vector is calculated when recognizing a gesture of shaking the head. FIG. 10 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed by the tongue recognition operation performed in the gesture recognition device 20 of the present invention. FIG. 11 is an example of distance image data acquired when recognizing a tongue out gesture. FIG. 12 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition of the knee closing operation performed in the gesture recognition device 20 of the present invention. FIG. 13 is an example of distance image data captured to detect a knee-closed gesture. FIG. 14 is a block diagram showing an example of a gesture recognition system including a gesture recognition device according to the second embodiment of the present invention. FIG. 15 is a flowchart illustrating an example of a flow of processing for determining a user's gesture performed in the gesture recognition device 20 in the gesture determination mode according to the second embodiment of the present invention. FIG. 16 is an example of an instruction screen in the gesture instruction program according to the second embodiment of the present invention. FIG. 17 is a flowchart illustrating an example of a flow of processing for determining a user's gesture performed in the gesture recognition device 20 in the gesture determination mode according to the third embodiment of the present invention. FIG. 18 is an example showing data storage contents of the parameter database 284 and the corresponding gesture database 286 according to the third embodiment of the present invention. FIG. 19 is a flowchart illustrating an example of a flow of interface control processing corresponding to a gesture recognized as recognition of a user's gesture motion performed in the gesture recognition device 20 according to the third exemplary embodiment of the present invention. FIG. 20 is an example of a hardware configuration diagram of the gesture recognition device.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing an example of a gesture recognition system including a gesture recognition device according to the present invention. In FIG. 1, the gesture recognition device 20 is connected to an imaging device 10 and an interface 30. The imaging device 10 is a so-called stereo camera having two cameras, for example. Moreover, the camera which can acquire a three-dimensional distance image may be used. For example, the imaging device 10 images a user who is performing an operation used for interface control, such as an operation of moving a finger. The imaging device 10 acquires image data in time series by shooting and sends the image data to the gesture recognition device 20. The interface device 30 is an interface controlled by the gesture recognition device 20 and includes, for example, the button 310, the switch 320, and the alarm 330, but is not limited thereto, and may be a mouse, a keyboard, a touch panel, or the like.

  The gesture recognition device 20 includes an image capturing unit 210, a region region extraction unit 220, a region detection unit 230, a cache unit 240, a region change detection unit 250, a gesture recognition unit 260, an interface control unit 270, and a storage unit 280.

  The image capturing unit 210 captures, for example, a distance image obtained by capturing the user's operation input from the imaging device 10 in real time. The captured moving image data is distance image data having parallax data or distance data. Note that two image data may be acquired from the imaging device 10 that is a stereo camera, and the image capturing unit 210 may calculate the parallax. Although the parallax data is used, when the imaging apparatus is not a stereo camera but a camera having a distance meter, image data and distance data may be captured. The “distance image data” here is image data (one frame image) at each time point in the moving image data which is a time-series image. The image capturing unit 210 outputs the input image data and parallax data to the part detection unit 220.

  The part region extraction unit 220 extracts a region where the predetermined part of the user exists in each frame image data of the distance image data captured by the image capturing unit 210. The predetermined part of the user is a part of the body necessary for gesture recognition associated with the interface control, such as a finger, an arm, a head, a knee, and a shoulder. The part region extraction unit 220 is provided with a three-dimensional space (x, y, z) in which the part can exist in advance as a parameter, and searches for a predetermined part of the user in the space. Extract.

  The part detection unit 230 detects a region in which the part to be extracted is imaged in the three-dimensional space in the distance image data extracted by the part region extraction unit 220. The part detection unit detects, for example, a part to be extracted from the texture image of the extracted part region based on the color or shape. The extracted region data is output as three-dimensional coordinate data.

  The cache unit 240 is a memory that temporarily stores data (for example, three-dimensional coordinate data) of a part region detected by the part detection unit 230 from the distance image data. The part area data temporarily stored is read and used by the part change detection unit 250 in order to detect the movement of the predetermined part.

  The part change detection unit 250 detects a change in the predetermined part region detected by the part detection unit 230 in each frame image. Specifically, the movement of the predetermined part is detected by calculating the amount of change of the part region detected by the part detection unit 230 between the frame images. The amount of change is not limited to the change in the coordinate value of the three-dimensional coordinates, but also includes the change in the color area. The calculation of the change is not limited to between adjacent frame images, and may be performed at a predetermined frame interval such as an interval of 5 frames or an interval of 30 frames. The part change detection unit 250 calculates the value of the previous frame image data used in the change calculation by reading it from the cache unit 240.

  The gesture recognition unit 260 recognizes that a predetermined gesture has been performed when the amount of change of the predetermined part detected by the part change detection unit 250 is equal to or greater than a predetermined value. If it is determined that the position of the predetermined area has changed by a predetermined amount or more, it is determined that a gesture has been performed. The gestures include, for example, finger bending, arm swing, changing the direction of the head, sticking out tongue, and closing the knee. The gesture is not limited to these. The amount of change is determined by comparison between the steady state and the maximum movement. That is, it is determined by the size of the three-dimensional trajectory distance in the position change. Further, the amount of change is the rotation angle when the recognition target is the head, the angle between the back of the hand and the finger when the recognition target is a finger, and the size of the tongue region when the recognition target is the tongue. If the gesture recognition unit 260 determines that a gesture has been made, the gesture recognition unit 260 sends the determined gesture content to the interface control unit 270.

  The interface control unit 270 performs interface control associated with the gesture recognized by the gesture recognition unit 260. Specifically, the corresponding gesture database 286 in the storage unit 280 is read, interface control corresponding to a predetermined gesture content is read, and the interface 30 is controlled. For example, when a finger bending gesture is recognized, the interface control unit 270 reads the corresponding gesture database 286 and turns on an interface control corresponding to the finger bending, for example, a switch.

  The storage unit 280 includes a distance image database 282, a parameter database 284, and a corresponding gesture database 286. The distance image database 282 stores the distance image data captured by the image capturing unit 210 from the imaging device 10. The distance image stored in the distance image database is read by the part region extraction unit 220 as appropriate.

  The parameter database 284 detects a coordinate range of a distance image area where a predetermined part region is imaged, a color threshold value or coordinate value used for detecting the predetermined part, and a part change and recognizes it as a gesture. It stores the threshold value and so on. The parameters stored in the parameter database may be configured to be arbitrarily changed and set for each user.

  The correspondence gesture database 286 is a database that stores the contents of the interface control associated with the gesture operation in association with each other. When the gesture recognition unit 260 recognizes that a predetermined gesture has been made, the interface control unit 270 reads the corresponding gesture database 286, reads the interface control corresponding to the recognized gesture, and performs interface control.

  FIG. 2 is a flowchart showing an example of a flow of interface control processing corresponding to a gesture recognized as recognition of a user's gesture operation performed in the gesture recognition device 20 of the present invention.

  Part region extraction unit 220 detects a region of a predetermined part (step S201). The part region extraction unit 220 extracts a predetermined part region from the distance image data from the imaging device 10 acquired by the image capturing unit 210. The predetermined part region may be a body part that performs gesture recognition, for example, a body part such as a head, a hand, an arm, a knee, or a shoulder, in the distance image data captured by the user. Extract regions. Since the range in which the predetermined part in the distance image data can exist is determined from the positional relationship between the imaging device 10 and the user, the range is stored in the parameter database 284 as three-dimensional coordinate data (x, y, z). ing. The part region extraction unit 220 reads out the three-dimensional coordinate data stored in the parameter database 284 and extracts a predetermined part region from the distance image data.

  Next, the part detection unit 230 detects a predetermined part from the part region (step S202). The part detection unit 230, for example, based on parameters such as texture information, color information (for example, hue, saturation, brightness), and shape in the distance image data of the part region extracted by the part region extraction unit 220 Is detected. Which parameter is used to extract the region depends on the portion to be detected. The part detection unit 230 reads the parameter information from the parameter database 284 based on the corresponding body part and uses it for detection.

  The part change detecting unit 250 detects a coordinate change of the part (step S203). The part detection unit 230 detects a change in coordinates of the predetermined part between the distance image data based on the predetermined part detected in each distance image data. For example, the part change detection unit 250 calculates a coordinate change between the part region in the distance image data of the previous frame to be compared and the part region in the distance image data of the current frame, which are temporarily stored in the cache unit 240. Further, the part change detection unit 250 calculates the change amount and the time until the change amount is reached. In order to avoid misrecognition due to involuntary movements of disabled people, it is intended to exclude those whose movement speed is too slow or too fast.

  Next, the gesture recognition unit 260 determines whether or not the change in the part is equal to or greater than a predetermined value based on the change amount of the part change detected by the part change detection unit 250 (step S204). When performing the determination, the gesture recognition unit 260 reads the part change detection parameter corresponding to the corresponding part from the parameter database 284 and performs the determination based on the read parameter. For example, when the predetermined part is a finger and a finger bending gesture, the determination is made based on whether or not the change in the angle between the finger and the hand is a predetermined amount or more.

  If the change in the region is greater than or equal to the predetermined value (yes), the gesture recognition unit 260 recognizes that the gesture operation has been performed (step S205). If the change in the part is not greater than or equal to the predetermined value (no), the part change detection unit 250 continues to detect the change in the coordinate of the part (step S203), and the gesture recognition unit 260 determines whether the change in the part is greater than the predetermined value. Do. When the gesture recognition unit 260 recognizes that the gesture operation has been performed, the gesture recognition unit 260 outputs the recognized gesture operation to the interface control unit 270.

  The interface control unit 270 performs interface control associated with the acquired gesture operation (step S206). The interface control unit 270 refers to the corresponding gesture database 286 and performs interface control of the interface 30 based on the interface control content corresponding to the recognized gesture operation.

  Although a method for detecting one body part has been described here, a plurality of body parts may be detected from one distance image data. For example, in the case of distance image data in which the upper body is photographed, a plurality of gestures such as a finger bending gesture, a head swing gesture, and a tongue out gesture may be detected.

  FIG. 3 is a diagram illustrating an example of data storage contents of the parameter database 284 and the corresponding gesture database 286 stored in the storage unit 280.

  The parameter database 284 includes, for each body part, a parameter for extracting a part region from distance image data, a parameter for detecting a predetermined part, and a parameter for detecting a part change for recognizing a gesture. Is remembered. These parameters are different for each body part to be detected. Here, the three-dimensional extraction range is specified by the three-dimensional coordinate data as the part region extraction parameter. However, the present invention is not limited to this. It is the same.

  Here, texture information and color information are designated as finger part detection parameters, but the designated parameters differ depending on the body part. Details of parameters that differ depending on the body part will be described later. These parameters may be configured to be freely changeable to arbitrary values based on the range of motion of the user's body.

  The corresponding gesture database 286 stores the recognized gesture and the corresponding interface control content in association with each other. For example, when a gesture in which the head is swung to the right is recognized, it is stored that a click operation is performed with the mouse. In addition to this, an alarm sound is generated for arm swing and a switch press is added as an example in addition to finger bending. However, the present invention is not limited to this, and a key press on the keyboard may be associated with a certain gesture. The interface control unit 270 refers to the corresponding gesture database 286 based on the gesture content output by the gesture recognition unit 260, and performs interface control based on the associated interface control content.

  The details of the gesture recognition method and interface control in each body part will now be described with reference to a flowchart.

  FIG. 4 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed by the gesture recognition device 20 of the present invention.

  First, the part region extraction unit 220 detects a region where a hand exists, that is, extracts a part region in a three-dimensional space that becomes a hand region from the distance image data captured by the image capturing unit 210 (step S401). . Specifically, the part region extraction unit 220 reads a parameter set as a three-dimensional space in which the hand can exist in the distance image data from the parameter database 282, and extracts the region set as a parameter from the distance image data. Extract. In the parameter database, for example, a maximum value and a minimum value of three-dimensional coordinates are stored as a three-dimensional space in which a hand can exist.

  Next, the part detection unit 230 detects a finger and a hand to be detected from distance image data of an area where the hand is present (step S402). The part detection unit 230 detects a hand and a finger to be a gesture target from the distance image data of the part region output from the part region extraction unit 220 based on the texture information. Specifically, based on the texture information, since a mark is attached to a finger to be gestured with a colored finger sack or the like, a three-dimensional object having the colored color region is extracted. For the hand, the 3D object in the area closest to the finger is labeled, and the gesture target hand is extracted by recognizing the maximum label as the hand. When extracting a hand, it is also possible to simultaneously use a skin color area corresponding to the hand. For the finger and hand, the extracted hue area and skin color area are stored in advance in the parameter database 284 with the maximum value and the minimum value of the hue and brightness, and the part detection unit 230 reads out these values. Then, the finger and hand parts are detected.

  The part change detection unit 250 detects a change in the angle between the finger and the hand between the distance image data based on the part of the finger and hand detected by the part detection unit 230 from the distance image data (step S403). Specifically, the region change detection unit 250 stores the position of the finger and hand in the distance image data of one or several frames before stored in the cache unit 240 and the part of the finger and hand in the current distance image data. The finger bending change is detected from the position. In detecting the bending change of the finger, the moment of the finger and the wrist in each distance image data is calculated, and the biaxial angle is calculated from the moment. Then, the difference between the finger and hand angles is calculated.

  The gesture recognition unit 260 determines whether the change amount of the finger-hand angle is equal to or greater than a predetermined value based on the angle calculated by the region change detection unit 250 (step S404). The set threshold value that is a predetermined value is stored in the parameter database 284, and the gesture recognition unit 260 reads the threshold value and refers to it to make a determination. As a result of the determination, if it is equal to or greater than the predetermined value (yes), it is determined that the finger is bent (step S405). As a result of the determination, if the amount of change in angle is equal to or less than a predetermined value (no), it is determined that no finger bending gesture has been made, and the process returns to step S403. Detection is performed. When it is determined that the finger is bent (step S405), the gesture recognition unit 260 outputs to the interface control unit 270 that the finger bending gesture has been performed. In this way, by detecting the region of the hand and then detecting the finger part, even a person whose hand moves due to involuntary movement can reliably detect the finger movement and recognize the gesture Can do.

  Next, the interface control unit 270 performs interface control associated with finger bending according to the output of the gesture recognition unit 260 (step S406). Specifically, the interface control unit 270 refers to the corresponding gesture database 286 and performs interface control corresponding to finger bending on the interface 30. In this way, since the finger bending is determined by the amount of change regardless of where the hand or finger is located and bent from any angle, it is possible to perform gesture recognition of people having various characteristics.

  FIG. 5 is a diagram illustrating an example of distance image data captured when the finger bending gesture is recognized. Since the captured image data includes distance data, boundaries where the distances are greatly different are in a state of being white. A finger and a hand are detected from such distance image data.

  FIG. 6 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed by the gesture recognition device 20 of the present invention.

  First, the part region extraction unit 220 detects a region where an arm can exist, that is, extracts a part region in a three-dimensional space that becomes a forearm region from the elbow to the distance image data captured by the image capturing unit 210. (Step S601). Specifically, the part region extraction unit 220 reads, from the parameter database 282, a parameter set as a three-dimensional space in which the forearm and hand ahead of the elbow can exist in the distance image data, and uses the parameters from the distance image data as parameters. Extract the set area. The parameter database stores, for example, the maximum value and the minimum value of the three-dimensional coordinates as a three-dimensional space in which the forearm and the hand can exist.

  Next, the part detection unit 230 detects the arm to be detected from the distance image data of the region where the arm exists (step S602). Part detection unit 230 applies a particle filter to the distance image data of the part region output from part region extraction unit 220.

  The part change detection unit 250 detects a change in arm movement between the distance image data based on the arm part detected by the part detection unit 230 from the distance image data (step S603). Specifically, the part change detection unit 250 detects a change in the position of the arm by tracking a local feature using a particle filter. Then, the swing of the arm that moves greatly is tracked by determining the likelihood of the particle in the inter-frame difference of the distance image data. Then, the part change detection unit 250 detects the arm position change by estimating the arm state from the gravity center moving distance of the particle group.

  The gesture recognition unit 260 determines whether or not the arm swing detected by the part change detection unit 250 is greater than or equal to a predetermined value (step S604). That is, it is determined whether the change amount of the center-of-gravity movement distance of the arm particle group within a predetermined time is greater than or equal to a predetermined value. The set threshold value that is a predetermined value is stored in the parameter database 284, and the gesture recognition unit 260 reads the threshold value and refers to it to make a determination. As a result of the determination, if it is equal to or greater than the predetermined value (yes), it is determined that the arm has been swung (step S605). As a result of the determination, if the amount of change in the arm position is less than or equal to the predetermined value (no), it is determined that no arm swing gesture has been made, and the process returns to step S603, and the region change detection unit 250 continues to change the arm position. Detection is performed. If the gesture recognition unit 260 determines that the arm has been shaken (step S605), the gesture recognition unit 260 outputs to the interface control unit 270 that the arm swing gesture has been performed.

  Next, the interface control unit 270 performs interface control associated with arm swing according to the output of the gesture recognition unit 260 (step S606). Specifically, the interface control unit 270 refers to the corresponding gesture database 286 and performs interface control corresponding to arm swing on the interface 30.

  FIG. 7 is an example of a screen on which the arm tracking process is performed from the distance image data by the particle filter when the arm swing is recognized. The center image is the distance image data, the upper right image is an image obtained by applying a particle filter to the distance image data, and the lower right image is an image showing the centroid detected from the particle group at the fingertip. As described above, by applying the particle filter to the distance image data, the local feature of the arm is tracked, the center of gravity of the arm is detected, and the movement of the center of gravity is calculated to detect the swing of the arm.

  FIG. 8 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed in the gesture recognition device 20 of the present invention.

  First, the part region extraction unit 220 detects a region where the head can exist from the distance image data captured by the image capturing unit 210, that is, a part region in a three-dimensional space that becomes a head region above the neck. Is extracted (step S801). Specifically, the part region extraction unit 220 reads a parameter set as a three-dimensional space in which the head can exist in the distance image data from the parameter database 282, and a region set as a parameter from the distance image data To extract. In the parameter database, for example, a maximum value and a minimum value of three-dimensional coordinates are stored as a three-dimensional space in which the head can exist.

  Next, the part detection unit 230 detects the head to be detected from the distance image data of the region where the head is present, and then detects the nose (step S802). The part detection unit 230 detects the head to be a gesture target based on the texture image from the distance image data of the part region output from the part region extraction unit 220. Specifically, the shape of an elliptic sphere that can be the head is extracted based on the texture image. Next, the extraction range is labeled, and the object closest to the face is recognized as a face. The maximum and minimum coordinate values are stored in advance in the parameter database 284 for the ellipsoidal shape and face object to be extracted, and the part detection unit 230 extracts the face by reading out these values. To do. Next, the distance image data of the extracted face image is zoomed, rotated, etc., to normalize the position, and from the face image data, the point closest to the camera is extracted as the nose.

  Subsequently, the part detection unit 230 calculates a normal vector of the face from the nose part (step S803). Specifically, based on the detected position of the nose, the normal vector of the face (face orientation) is calculated based on the distance information of the region centered on the nose.

  The part change detection unit 250 detects a change in the direction of the normal of the face (step S804). The part change detecting unit 250 calculates the amount of change in the normal vector of the face in the distance image data one or several frames before stored in the cache unit 240 and the amount of change in the normal vector in the current distance image data. Detect orientation changes.

  The gesture recognition unit 260 determines whether or not the change in the direction of the normal vector detected by the region change detection unit 250 is greater than or equal to a predetermined value (step S805). That is, it is determined whether the amount of change in the direction of the normal vector is greater than or equal to a predetermined value. The determination may be made from the amount of change in direction within a predetermined time. For example, it may be determined by the amount of change that occurs when the neck is moved with the intention of moving to the right, left, or downward, with the face not moving as the initial position. The setting threshold value that is a predetermined value is stored in the parameter database 284 in accordance with each direction, and the gesture recognition unit 260 reads the threshold value and refers to it to make a determination. . As a result of the determination, if it is equal to or greater than the predetermined value (yes), it is determined that the head has been moved (step S806). As a result of the determination, if the amount of change in the direction of the normal is equal to or less than the predetermined value (no), it is determined that no gesture of shaking the head has been made, and the process returns to step S803. Change of direction is detected. If it is determined that the head is shaken (step S806), the gesture recognition unit 260 outputs to the interface control unit 270 that the gesture of shaking the head has been made.

  Next, the interface control unit 270 performs interface control associated with the head movement according to the output of the gesture recognition unit 260 (step S807). Specifically, the interface control unit 270 refers to the corresponding gesture database 286 and performs interface control corresponding to head movement on the interface 30.

  FIG. 9 is an example of a screen on which a nose is extracted from the distance image data and a normal vector is calculated when recognizing a gesture of shaking the head. In this way, after normalizing the face image and detecting the nose, the normal vector is calculated.

  FIG. 10 is a flowchart showing an example of the flow of interface control processing corresponding to gesture recognition and gesture recognition performed by the tongue recognition operation performed in the gesture recognition device 20 of the present invention.

  First, the part region extraction unit 220 detects a region where the head can exist from the distance image data captured by the image capturing unit 210, that is, a part region in a three-dimensional space that becomes a head region above the neck. Is extracted (step S1001). Since this step is the same as step S801 in recognizing the gesture motion of shaking the head, details are omitted.

  Next, the part detection unit 230 detects the head to be detected from the distance image data of the region where the head is present, and then detects the nose (step S1002). This processing is also the same as that in step S802 in recognizing the motion of shaking the head, and thus the description thereof is omitted.

  The part detection unit 230 detects the tongue area from the head area below the nose based on the nose part detected from the distance image data (step S1003). Specifically, based on the detected position of the nose, HSV color information (hue, brightness, saturation) of the head region below the nose is acquired, and the hue threshold set as the tongue color is set in the parameter database. Read from 284 and perform filtering. A region extracted by filtering is subjected to a labeling process to detect a region having a label of a predetermined size or more as a tongue.

  The part change detection unit 250 detects a change in the tongue region (step S1004). The part change detection unit 250 calculates the amount of change between the tongue region in the distance image data one or several frames before stored in the cache unit 240 and the tongue region in the current distance image data, and detects the change in the tongue region. To detect.

  The gesture recognition unit 260 determines whether or not the change in the tongue region detected by the region change detection unit 250 is greater than or equal to a predetermined value (step S1005). Moreover, you may determine by the tongue area | region becoming more than predetermined area and continuing for a fixed time. The set threshold value that is a predetermined value is stored in the parameter database 284, and the gesture recognition unit 260 reads the threshold value and refers to it to make a determination. As a result of the determination, if it is greater than or equal to a predetermined value (yes), it is determined that the tongue has been taken out (step S1006). As a result of the determination, if the change in the tongue area is equal to or smaller than the predetermined value (no), it is determined that no gesture for sticking out the tongue has been made, and the process returns to step S1004. Do. If it is determined that the tongue has been taken out (step S1006), the gesture recognition unit 260 outputs to the interface control unit 270 that a gesture for taking out the tongue has been made. In this way, the tongue region is extracted from the detection of the head region, so that even if there is an involuntary movement in the face movement, the head region movement can be reliably tracked. It becomes possible to detect gesture and perform gesture recognition.

  Next, the interface control unit 270 performs interface control associated with the gesture for sticking out the tongue according to the output of the gesture recognition unit 260 (step S1007). Specifically, the interface control unit 270 refers to the corresponding gesture database 286 and performs interface control corresponding to tongue out for the interface 30.

  FIG. 11 is an example of distance image data acquired when recognizing a tongue out gesture. The center of (a) is distance image data with the tongue out, and an image processed image is displayed on the right side. The upper right is the image data with the head detected, the middle right is the image data normalized after the head detection, and the lower right is the image data with the tongue region detected. The center of (b) is distance image data that does not stick out the tongue, and an image processed in the same manner as (a) is displayed on the right side. In this way, the color region corresponding to the color of the tongue is extracted, and the tongue recognition gesture recognition is performed based on the change of the color region.

  FIG. 12 is a flowchart showing an example of a flow of interface control processing corresponding to gesture recognition and gesture recognition performed in the gesture recognition device 20 of the present invention.

  First, the part region extraction unit 220 extracts detection of a region where both knees may exist from the distance image data captured by the image capture unit 210 (step S1201). Specifically, the part region extraction unit 220 reads a parameter set as a three-dimensional space in which the knee can exist in the distance image data from the parameter database 282, and reads the region set as a parameter from the distance image data. Extract. In the parameter database, for example, the maximum value and the minimum value of the three-dimensional coordinates are stored as a three-dimensional space in which the knee can exist.

  Next, the part detection unit 230 detects the knee head portions of both knees to be detected from the distance image data of the area where the knee exists (step S1202). The part detection unit 230 is the distance image data of the knee region output from the part region extraction unit 220, and estimates the knee head position of both knees by the hill-climbing method from the end side of each knee region. Is detected.

  The part change detection unit 250 detects a change in the position of both knees between the distance image data based on the position of the knee heads of both knees detected by the part detection unit 230 from the distance image data (step S1203). Specifically, the part change detection unit 250 calculates the position of the knee head part in the distance image data one or several frames before stored in the cache unit 240 and the positions of both knees in the current distance image data. Detect position change of both knees. In detecting the position change of both knees, the amount of change is calculated from the position coordinates of both knees in the respective distance image data. Specifically, here, since it is a gesture for closing the knee, the amount of change due to the approach of the position coordinates of both knees is calculated.

  The gesture recognition unit 260 determines whether or not the amount of change continues for a predetermined time or more based on the position change of both knees calculated by the region change detection unit 250 (step S1204). In addition, you may determine by the distance by the coordinate value of both knees approaching more than predetermined value. The set threshold value that is a predetermined value is stored in the parameter database 284, and the gesture recognition unit 260 reads the threshold value and refers to it to make a determination. As a result of the determination, if it is equal to or greater than the predetermined value (yes), it is determined that both knees are closed (step S1205). As a result of the determination, if the amount of change in the positions of both knees is equal to or less than a predetermined value (no), it is determined that no knee closing gesture has been made, and the process returns to step S1203. Change detection is performed. If it is determined that the knee is closed (step S1205), the gesture recognition unit 260 outputs to the interface control unit 270 that the knee closing gesture has been made.

  Next, the interface control unit 270 performs interface control associated with the knee closure according to the output of the gesture recognition unit 260 (step S1206). Specifically, the interface control unit 270 refers to the corresponding gesture database 286 and performs interface control corresponding to knee closure on the interface 30. Note that it is not always necessary to approach the knees until the knees are in close contact with each other, and it may be recognized that the knee-closing gesture has been performed when the knees are closer than the normal state. The position at which the knee is recognized as being closed can be arbitrarily set by a threshold value to be set. Here, the gesture of closing the knee is used, but gesture recognition may be performed by opening the knee.

  FIG. 13 is an example of distance image data captured to detect a knee-closed gesture. In this way, when recognizing the knee, the image is shot with the camera setting so that the knee is centered.

(Second embodiment)
In the first embodiment, by setting a parameter for recognizing a gesture for each part of the body that can be used for interface control and selecting which part is used, reproducible movement can be performed. A gesture recognition device capable of recognizing a gesture even when the site is different for each physically handicapped has been described. In the second embodiment, a gesture determination mode is provided before performing interface control by gesture recognition (gesture recognition mode), thereby accurately recognizing gestures according to the symptoms of individual disabled persons that change daily. A gesture recognition apparatus for performing the above will be described. Note that description of the same processing as in the first embodiment is omitted.

  FIG. 14 is a block diagram illustrating an example of a gesture recognition system including the gesture recognition device according to the second embodiment of this invention. In addition, about the structure which is common in FIG. 1, the same number is attached | subjected and description is abbreviate | omitted. In FIG. 15, the gesture recognition device 200 is connected to the imaging device 10 and the interface 30. The gesture recognition device 200 includes an image capture unit 210, a region region extraction unit 220, a region detection unit 230, a cache unit 240, a region change detection unit 250, a gesture recognition unit 260, an interface control unit 270, a storage unit 280, and a gesture determination unit. 290.

  The storage unit 280 includes a distance image database 282, a parameter database 284, a corresponding gesture database 286, and a gesture instruction program 288. The distance image database 282 stores distance image data captured by the image capturing unit 210 from the imaging device 10. The parameter database 284 includes a part region extraction parameter that is a coordinate range in distance image data in which a predetermined part region is imaged, a part detection parameter that is a color threshold value or coordinate value used to detect the predetermined part, A threshold for detecting a change in a part and recognizing it as a gesture is stored. In the second embodiment, at least a region detection parameter that is a threshold for detecting a region change and recognizing it as a gesture is a parameter determined by the gesture determination mode, and has a different value for each user. Is set. The threshold value, which is a change amount for recognizing as a gesture, is a value determined by the gesture determination unit 290, and is updated every time the gesture determination mode is executed. The correspondence gesture database 286 is a database that stores the contents of the interface control associated with the gesture operation in association with each other. Based on the determination of the gesture determination unit 290, for each user, the body part used for interface control and the content of the interface control are stored in association with each other.

  The gesture instruction program 288 is a program for instructing the user to move the moving body part at a predetermined timing, and is read by the gesture determination unit 290. The gesture instruction program 288 issues an instruction a plurality of times at a predetermined timing to cause the user to move the body. As a result, it is possible to obtain a motion (voluntary movement) that is based on the user's intention and has reproducibility, and more reliably control the interface by issuing an instruction multiple times. It is possible to determine the gestures that can be used.

  The gesture determination unit 290 determines a part and a gesture associated with the interface control from one or a plurality of candidate parts of the user. The body parts that can be moved vary depending on the user, and the degree and method of movement vary. For this reason, in the gesture determination mode, the gesture determination unit 280 reads the gesture instruction program 288 from the storage unit 280, instructs the user to perform the gesture at a predetermined timing, and is a candidate body part to be associated with the interface control. Is determined based on the amount of change in the part detected by the part change detection unit 250. In the case of a change in the instructed timing, it can be determined that the user has moved with intention, so that a reproducible movement, that is, a gesture that can be used for interface control can be reliably acquired. In addition, by issuing an instruction a plurality of times and moving the body part, it is possible to determine a body part that can be used for a gesture, and to determine a threshold that can be recognized as a gesture more reliably.

  For example, the gesture determination unit 290 determines a change amount recognized as a gesture from a change amount at a predetermined timing that can be acquired a plurality of times in the gesture determination mode. The parameter viewed as the amount of change differs depending on each body part. For example, a finger bending gesture is a change in the angle between a finger and a hand, and an arm swing is a change in the arm position. A change amount and a threshold value to be finally recognized as a gesture are determined from the changes acquired a plurality of times. At this time, an average value of a plurality of acquired values may be determined as a threshold value. For example, in the case of finger bending, the average value of the angle between the hand and the finger acquired at the timing instructed by the gesture instruction program is calculated and determined as the amount of change. In addition, the gesture determination unit 290 stores one or more values of the amount of change determined in the past, calculates the average of the average value calculated this time and the past value, and determines it as a new threshold value. May be. In this case, the stored past value may be configured such that the oldest value is deleted (FIFO) each time a new value is stored.

  The body part that can be used for the gesture determined by the gesture determination unit 290 and the amount of change related to the movement of the gesture are stored in the parameter database 284 as a threshold for gesture recognition in association with the user. In addition, the body part and the change amount that can be used for the gesture determined by the gesture determination unit 290 are stored in the corresponding gesture database 286 as a gesture in association with the interface control.

  In the gesture recognition mode, the gesture recognition unit 260 reads the change amount (part change detection parameter) in the predetermined body part determined by the gesture determination part 290, and the change amount of the predetermined part detected by the part change detection unit 250 is the part. If it is equal to or greater than the threshold value of the change detection parameter, it is recognized that a predetermined gesture has been performed. If the gesture recognition unit 260 determines that a gesture has been made, the gesture recognition unit 260 sends the determined gesture content to the interface control unit 270. The interface control unit 270 performs interface control associated with the gesture recognized by the gesture recognition unit 260.

  FIG. 15 is a flowchart illustrating an example of a flow of processing for determining a user's gesture performed in the gesture recognition device 20 in the gesture determination mode according to the second embodiment of the present invention. This gesture determination mode is a mode that is always performed when a user uses the gesture recognition device for the first time. In the gesture determination mode, the body part used for the gesture and the part change used as the gesture are determined, and the subsequent gesture recognition is performed. When the gesture is recognized in the mode, the determined body part and the part change amount are used. Note that the gesture determination mode may be performed every day. By carrying out every day, it is possible to change the amount of change for gesture recognition according to the physical symptoms of the user that change daily, and it is possible to realize a more accurate and user-friendly gesture recognition device.

  Before the gesture determination mode is started, the user is in a position where the imaging apparatus 10 can photograph the whole body or the front of the upper body or the lower body. First, when the gesture determination mode is started, the gesture instruction program 288 is read and started by the gesture determination unit 290 (step S1601). The gesture instruction program 288 is a program for instructing and instructing the user to move the body part at a predetermined timing. The instruction indicates the timing to perform the movement in accordance with the music, but it flows from left to right, and when it comes to the predetermined position on the left side, it makes a sound by making a gesture. After the first time, the program is capable of giving instructions to two or more candidate parts at different timings or giving instructions simultaneously.

  Part region extraction unit 220 detects a candidate part region (step S1602). The part region extraction unit 220 extracts a candidate part region of the body that can be used for the gesture from the distance image data from the imaging device 10 acquired by the image capturing unit 210. Examples of regions of the body part that are candidates include the head, hands, arms, knees, toes, fingers, and mouth. Since the range in which the predetermined part in the distance image data can exist is determined from the positional relationship between the imaging device 10 and the user, the three-dimensional coordinate data (x, x, which is the part region extraction parameter stored in the parameter database 284). y, z) is read out and the candidate region is detected.

  Next, the part detection unit 230 detects a predetermined part from the part region (step S1603). The part detection unit 230 is based on part detection parameters such as texture information, color information (for example, hue, saturation, brightness), and shape in the distance image data of the part region extracted by the part region extraction unit 220, for example. Detect the region of the part. The part detection unit 230 reads the part detection parameter from the parameter database 284 based on the corresponding body part and uses it for detection.

  The part change detection unit 250 detects a coordinate change of the part (step S1604). The part detection unit 230 detects a change in the coordinates of the candidate part between the distance image data based on the candidate part detected in each distance image data. For example, the part change detection unit 250 calculates the coordinate change between the candidate part region in the distance image data of the previous frame to be compared and the candidate part region in the distance image data of the current frame, which are temporarily stored in the cache unit 240. To do. For example, when the candidate site is the tongue, the change amount of the tongue region is calculated, and when it is the knee, the change amount of the knee position is calculated.

  Next, when the region change detection unit 250 detects a region change, the gesture determination unit 290 determines whether the coordinate has changed at a predetermined timing (step S1605). The predetermined timing is a timing when the gesture instruction program instructs the user to move the body part. If you have a physical disability, your body may move regardless of your will. However, if the coordinate changes at a predetermined timing, it is clear that the coordinate change is caused by the user's intentional movement, and this is a coordinate change that can be used as a gesture. Here, for example, if the coordinate change is within 0.5 seconds from the instructed timing, it is determined that the coordinate is changed at a predetermined timing.

  In the case of a coordinate change at a predetermined timing (yes), the gesture determination unit 290 stores the coordinate change amount (step S1606). Specifically, the starting point and end point coordinate values of the movement in the candidate part region calculated by the part change detection unit 250 and the time from the start point to the end point are stored. These data are sequentially stored until the gesture instruction program ends.

  If it is not a coordinate change at a predetermined timing (no), the part change detection unit 250 continues to detect a coordinate change of the candidate part (step S1604).

  Then, the gesture determination unit 290 determines whether the gesture instruction program has ended (step S1607). When the gesture instruction program ends (yes), the gesture determination unit 290 calculates an average value based on the stored change amount (step S1608). Specifically, a threshold value used for gesture recognition is calculated from a plurality of obtained change amounts. Here, the average value is calculated, but the present invention is not limited to this. For example, an intermediate value between the maximum value and the minimum value among the obtained change amounts may be determined as the threshold value. Moreover, you may determine the minimum value as a threshold value among what has a value more than fixed among the acquired variation | change_quantity. Since the parameter viewed as the amount of change differs for each body part, how the threshold value is determined from a plurality of amounts of change may differ for each body part.

  In calculating the threshold value, if the threshold value has been determined by executing the gesture instruction program for the same user in the past, the threshold value is stored in the past. An average of the threshold value at the time of execution and the value calculated from the amount of change at the time of execution this time may be further calculated as the threshold value. In this case, the stored past threshold value may be configured such that the oldest value is deleted (FIFO) each time a new value is stored. In addition, when there is a part having a large variation in the obtained values among the candidate parts, the gesture determining unit 290 determines that the candidate part is not used for the gesture without calculating the average value.

  If the gesture instruction program has not ended (no), the part change detection unit 250 continues to detect a coordinate change of the candidate part (step S1604).

  Next, the gesture determination unit 290 updates a value stored in the parameter database 284 to determine a gesture (step S1609). Specifically, the gesture determination unit 290 stores the calculated average value as a threshold value in association with the body part determined to be used for gesture recognition. The updated region change detection parameter is used for gesture recognition in the gesture recognition mode. In addition, when the gesture determination unit 290 determines a part to be used for the gesture, the gesture determination unit 290 updates the corresponding gesture database 286 in order to associate the body part with the interface control. Then, the gesture determination mode ends. As described above, by providing the gesture determination mode, it is possible to determine a threshold for gesture recognition that matches each person's symptom, so that gesture recognition can be performed with high accuracy.

  In the gesture determination mode, when the body part used for the gesture and the threshold value for gesture recognition are determined, in the gesture recognition mode, the threshold value determined in the gesture determination mode is read from the parameter database 284 to recognize the gesture. To control the interface. Since the gesture recognition process is the same as that of the first embodiment, the description thereof is omitted.

  FIG. 16 is an example of an instruction screen in the gesture instruction program 288 used in the present invention. In the gesture determination mode, when the gesture instruction program 288 is executed, an instruction is given to move the body on the screen. Here, four parts can be assigned, and the assigned body is moved at the timing when a circle arrives at the left end. This is merely an example, and any program may be used, such as instructing by voice, as long as the program instructs the user to move the body part at a predetermined timing.

  When the gesture determination mode ends, the gesture recognition mode is entered, and the gesture recognition unit basically follows the flow diagram of FIG. 2 based on the body part used for gesture recognition determined by the gesture determination unit 290 and its threshold value. Perform gesture recognition. Since the gesture recognition process in each body part is the same as that in the first embodiment, a description thereof will be omitted.

(Third embodiment)
The following third embodiment will be described. Also in the third embodiment, a gesture determination mode is provided to determine a threshold for gesture recognition tailored to each person, but unlike the second embodiment, the detection site is where on the body. Decide on a gesture without being aware of it. The gesture recognition system in the third embodiment is the same as that in the second embodiment. Description of the same contents as those of the second embodiment is omitted.

  FIG. 17 is a flowchart illustrating an example of a flow of processing for determining a user's gesture performed in the gesture recognition device 20 in the gesture determination mode according to the third embodiment of the present invention.

  Before the gesture determination mode is started, the user makes a position where the body part used for the gesture is closest to the imaging apparatus 10. Examples of body parts used for gestures include hands, arms, knees, toes, fingers, mouths, and ears. Of these, the body parts with small movements are the finger, mouth, ears, and the like. For example, when the body part used for the gesture is a foot tip, the positions of the imaging device and the user are adjusted so that the foot tip is closest to the imaging device 10. If the user is a bedridden user, the sleeping floor plane is estimated, and the positions of the imaging device and the user are positioned so that the body part used for the gesture is closest to the imaging device 10 from above the floor plane. adjust. By setting the position so as to be closest, the movement can be easily detected. When the gesture determination mode is started, the gesture instruction program 288 is read by the gesture determination unit 290, and the program is started (step S1701). The gesture instruction program 288 signals and instructs the user to move the body part at a predetermined timing.

  Part region extraction unit 220 detects a candidate region (step S1702). The part region extraction unit 220 extracts candidate regions from the distance image data from the imaging device 10 acquired by the image capturing unit 210. A threshold value is provided for the parallax value so that a region having parallax data close to the imaging device 10 is extracted using the fact that the body part used for the gesture is close to the imaging device 10 in advance. Extract candidate areas by threshold.

  Next, the part detection unit 230 detects a rectangle including the extracted region as a part (step S1703). The part detection unit 230 sets, for example, a rectangle including the candidate area extracted by the part region extraction unit 220 for each frame image. That is, in a frame image for a predetermined number of frames, a rectangle that includes each motion region is determined. This rectangle is specified by three-dimensional coordinates (x, y, z) and screen coordinates (u, v).

  The part change detection unit 250 detects a change in the rectangle (step S1704). The part detection unit 230 sets a rectangle including the candidate area in each distance image data. Then, the region change detection unit 250 temporarily stores the rectangular area in the distance image data of the previous frame to be compared in the past frame images for a predetermined number of frames, which are temporarily stored in the cache unit 240, and the distance image of the current frame. The change in the rectangular area in the data is calculated. A change is a coordinate change of a feature point or a difference between frames of an edge image from a gray image or a gray image. In order to detect a change in a part with a small movement such as a mouth or an ear, it is desirable to calculate a difference between frames of a gray image or an edge image from the gray image. Therefore, when it is known that the mouth and ears can be moved only slightly depending on the user's symptoms, the change detection may be set in advance so as to be performed with the difference between frames. In the case of a user who can move the body to such an extent that it can be seen immediately from other people, the coordinate change of the feature point is detected.

  Next, when the region change detection unit 250 detects a change, the gesture determination unit 290 determines whether the movement is at a predetermined timing (step S1705). The predetermined timing is a timing when the gesture instruction program instructs the user to move the body part. For example, the gesture determination unit 290 determines that the movement is at a predetermined timing when the movement is within a 0.5 second width from the instructed timing. If you have a physical disability, your body may move regardless of your will. However, if the coordinate changes at a predetermined timing, it is clear that the coordinate change is caused by the user's intentional movement, and this is a coordinate change that can be used as a gesture.

  In the case of movement at a predetermined timing (yes), the gesture determination unit 290 determines a rectangle and stores the amount of change (step S1706). In the case of a change at a predetermined timing, the gesture determination unit 290 determines the largest rectangle from the rectangles set by the part detection unit 230 for each frame image of a predetermined number of frames from the beginning to the end of the change. Thus, by specifying an image area that changes when a gesture is performed, the area to be tracked is limited, and the processing can be performed at high speed. Then, the determined rectangle is stored in the parameter database 284 as a part detection parameter in three-dimensional coordinates (x, y, z) and screen coordinates (u, v).

  Then, the gesture determination unit 290 calculates a three-dimensional trajectory based on the coordinate change of the feature point calculated by the region change detection unit 250 from the changed first image to the end image. In the three-dimensional trajectory, the coordinates in the steady state as the starting point of the motion and the coordinates of the end point at the time of maximum movement are stored as the three-dimensional coordinates and the screen coordinates. Note that not only the coordinate value but also the distance between the start point and the end point may be stored together. When detecting the interframe difference as a change, the interframe difference based on the gray image or the edge image from the gray image is stored. These data are sequentially stored until the gesture instruction program ends.

  If it is not a movement at a predetermined timing (no), the gesture determination unit 290 ignores the change detected by the part change detection unit, and the part change detection unit 250 detects a movement at a predetermined position (step S1704).

  Then, the gesture determination unit 290 determines whether the gesture instruction program has ended (step S1707). When the gesture instruction program ends (yes), the gesture determination unit 290 calculates an average value based on the stored plurality of changes (step S1608). The calculated average value becomes a threshold value for gesture recognition. For example, when a three-dimensional trajectory due to a coordinate change is acquired, an average value is calculated for each of the coordinate values that are the starting point and the end point of the movement. Here, although the average value is used as a threshold value, it is not necessarily limited to the average value. For example, an intermediate point between the coordinates of the start point and the end point of the motion may be determined as a threshold value, or the moving locus position may be histogrammed and determined by a binary threshold value selection method. In addition, when the value of the difference between frames is used as the amount of change, the maximum value of the amount of change is accumulated, the minimum value among the maximum values is calculated, and the minimum value is determined as a threshold for recognizing the gesture. May be. In this case, lower maximum values that deviate from the normal distribution of the maximum values by a certain amount may be excluded.

  In calculating the threshold value, if the threshold value is calculated by executing a gesture instruction program for the same user in the past, one or more threshold values calculated in the past are stored. The average of the threshold value when executed in the past and the threshold value calculated from the amount of change when executed this time may be further calculated as the threshold value. In this case, the stored past threshold value may be configured such that the oldest value is deleted (FIFO) each time a new value is stored. In addition, when there exists a site | part with a large dispersion | variation in the value which can be acquired among the candidate site | parts, the gesture determination part 290 determines not to use an average value and not to use for a gesture.

  If the gesture instruction program has not ended (no), the region change detection unit 250 continues to detect movement at a predetermined position (step S1704).

  Next, the gesture determination unit 290 updates the value stored in the parameter database 284 to determine a gesture (step S1709). Specifically, the gesture determination unit 290 uses a rectangular coordinate value for extracting a gesture from an image and a threshold value for detecting a change for gesture recognition as a part for performing the gesture recognition. Remember me. The threshold for gesture recognition is a part change detection parameter, and is either the coordinates of the start point and end point or the inter-frame difference value. The updated region change detection parameter is used for gesture recognition in the gesture recognition mode. In addition, when the gesture determination unit 290 determines a gesture, the gesture determination unit 290 updates the corresponding gesture database 286 in order to associate the gesture with the interface control. Then, the gesture determination mode ends. As described above, according to the third embodiment, it is possible to perform gesture recognition in accordance with the symptom of each individual without being aware of where the moving part is in the body.

  FIG. 18 is an example showing data storage contents of the parameter database 284 and the corresponding gesture database 286 according to the third embodiment of the present invention. The parameter database 284 stores a parameter for specifying a region and a rectangular region used for gesture recognition and a threshold value for detecting a change for gesture recognition in association with the user. However, the part information is stored as, for example, “part 1” and “part 2” as information for distinguishing when there are a plurality of areas used for gestures, not information specifying which part of the body is used as the part. Yes. As described above, in the third embodiment, the rectangular coordinates for detecting the motion region and the gesture recognition are recognized without being aware of where the actual body part is to be recognized. By storing only the coordinate value or the inter-frame difference value, which is a threshold value, gesture recognition is possible. Here, the coordinates of the four points of the three-dimensional coordinates and the screen coordinates are stored as the information of the rectangular area. However, the present invention is not limited to this, and other methods may be used as long as the information specifies the rectangular area. In addition, when a candidate area is detected based on a threshold value based on a parallax value, when a different threshold value is set for each individual, the parallax threshold value may be stored together as a part area extraction parameter.

  The corresponding gesture database 286 stores a change amount recognized as a gesture and interface control content in association with each other. In the third embodiment, it is determined that a gesture is recognized when a certain place moves more than a predetermined value, not what kind of movement by which body part is detected. The contents of interface control are stored for each user.

  FIG. 19 is a flowchart illustrating an example of a flow of interface control processing corresponding to a gesture recognized as recognition of a user's gesture motion performed in the gesture recognition device 20 according to the third exemplary embodiment of the present invention.

  In the gesture recognition mode, the part region extraction unit 220 extracts a region close to the imaging device 10 (step S1901). Specifically, the part region extraction unit 220 extracts a region that satisfies the threshold value based on the threshold value based on the parallax value from the distance image data captured by the image capturing unit 210.

  The part detection unit 230 detects a rectangle as a predetermined part (step S1902). The coordinate value of the rectangular area memorize | stored in the parameter database 284 is read, and the rectangle which includes one of the area | regions which the part area | region extraction part 220 extracted is set. When a plurality of parts are used for gesture recognition, a plurality of rectangles are set.

  Next, the site | part change detection part 250 detects the change in the set rectangle (step S1903). When the coordinate value is stored as the part change detection parameter, the change between the coordinate values and the inter-frame difference value are calculated when the inter-frame difference value is stored.

  The gesture recognition unit 260 determines whether the change in the rectangle detected by the region change detection unit 250 is greater than or equal to a predetermined value (step S1904). This is determined based on the part change detection parameter stored in the parameter database 284. If the change is greater than or equal to a predetermined value (yes), it is recognized as a gesture (step S1905). When the change in the rectangle is not greater than or equal to the predetermined value (no), the part change detection unit 250 continues to detect the change in the rectangle (step S1903). When the gesture recognition unit 260 recognizes that the gesture has been made, the gesture recognition unit 260 outputs the fact that the gesture has been made to the interface control unit 270.

  The interface control unit 270 refers to the corresponding gesture database 286 according to the output from the gesture recognition unit 260, and performs interface control on the interface 30 (step S1906). In this way, gesture recognition can be performed without having a model for gesture recognition for each body part.

  FIG. 20 is an example of a hardware configuration diagram of the gesture recognition device. The gesture recognition device 20 may be configured as a personal computer owned by a system user and a program executed on the personal computer. The personal computer includes a CPU (Central Processing Unit) 1001, a RAM (Random Access Memory) 1003 connected to the CPU 1001 via a bus, a ROM (Read Only Memory) 1005, an external storage device 1007 such as a hard disk drive, an I / O An O interface 1009, a communication interface 1011 for connecting to a communication network line, and the like. A camera 1013, an alarm 1015, a button 1017, and a switch 1019 are connected to the interface 1009. In this case, for example, the functions of the image capturing unit 210, the part region extracting unit 220, the part detecting unit 230, the part change detecting unit 250, the gesture recognizing unit 260, the interface control unit 270, and the gesture determining unit 290 of the gesture recognition device 20 are provided. The functions of the storage unit 280 are realized by the external storage device 1007, and the functions of the imaging device 10 and the interface 30 are realized by a camera, an alarm, a button, and a switch, respectively. Programs that realize various functions are stored in the external storage device 1007, read into the RAM 1003, and then executed by the CPU 1001.

DESCRIPTION OF SYMBOLS 10 Imaging apparatus 20 Gesture recognition apparatus 30 Interface 210 Image taking-in part 220 Part area extraction part 230 Part detection part 240 Cache part 250 Part change detection part 260 Gesture recognition part 270 Interface control part 280 Storage part

Claims (8)

A gesture recognition device that recognizes a user's gesture based on distance image data captured by an imaging device and performs interface control associated with the recognized gesture on the interface device,
An image capturing unit that captures distance image data output from the imaging device;
A region extraction unit for extracting from the distance image data a region where one or a plurality of candidate regions of the user exists each time the distance image data is captured;
A site detector that detects one or more candidate sites from the extracted site region;
A part change detection unit that detects a change amount of the detected candidate part based on the detected predetermined part in each distance image data;
A gesture recognition unit for recognizing that a gesture has been performed when a change in the detected predetermined part is equal to or greater than a predetermined value;
When it is recognized that a gesture has been performed, an interface control unit that performs interface control associated with the gesture;
A gesture determining unit that determines a portion associated with interface control and a gesture in the portion from one or a plurality of candidate portions of the user ;
The gesture determination unit instructs to perform a gesture at a predetermined timing, and determines a part associated with interface control and a threshold to recognize the gesture at the part based on a change amount of the candidate part at the predetermined timing. And
The gesture recognizing apparatus recognizes that a gesture has been performed when a change amount or more determined in the region determined by the gesture determining unit is detected .
A gesture recognition device that recognizes a user's gesture based on distance image data captured by an imaging device and performs interface control associated with the recognized gesture on the interface device,
  An image capturing unit that captures distance image data output from the imaging device;
  A region extraction unit for extracting from the distance image data a region where one or more predetermined candidate portions of the user exist each time the distance image data is captured;
  A part detection unit for detecting a rectangle including one or more candidate predetermined parts from the extracted part region;
  A part change detecting unit for detecting a change in the rectangle;
A gesture determining unit that determines a rectangle that includes a region with a change based on a change detected by the part change detection unit, and that determines a threshold value to be recognized as a rectangle and a gesture associated with interface control;
A gesture recognition unit that recognizes that a gesture has been performed when a change in the detected rectangle is equal to or greater than a predetermined value;
  An interface control unit that performs interface control associated with the gesture when the gesture is recognized, and
  The gesture determination unit instructs to perform a gesture at a predetermined timing, determines a threshold based on a change amount in a rectangle at the predetermined timing,
The gesture recognition device recognizes that a gesture has been performed when a threshold value determined in the rectangle determined by the gesture determination unit is detected.
The gesture recognition device according to claim 1 or 2 ,
In addition, it has a parameter database that stores thresholds for detecting changes,
The gesture determination unit stores a change amount determined to be recognized as a gesture in a parameter database in association with a user as a threshold value,
The gesture recognition unit reads a threshold value stored in the parameter database, and recognizes a gesture when the change is equal to or greater than the threshold value.
The gesture recognition apparatus according to claim 2 or 3, wherein the part region extraction unit sets a threshold value for the parallax value and extracts a region where a candidate part exists based on the threshold value.
The gesture recognition apparatus according to claim 2, wherein the part change detection unit detects a coordinate change of a feature point or an inter-frame difference of an edge image as a change.
A gesture recognition apparatus according to claim 5, wherein,
The gesture determination section, for the same user, and thresholds were determined as the gesture in the past, the amount of change at a predetermined timing newly acquired, a new threshold that recognized as the gesture based on the A gesture recognition device characterized in that
A gesture recognition program for recognizing a user's gesture based on distance image data and performing interface control associated with the recognized gesture,
A gesture recognition program that causes a computer to function as the gesture recognition device according to claim 1 .
An imaging device for imaging distance image data;
A gesture recognition system comprising: a gesture recognition device that recognizes a user's gesture based on distance image data captured by the imaging device and performs interface control associated with the recognized gesture on the interface device. ,
The imaging device images a user's body,
The gesture recognition device according to claim 1 , wherein the gesture recognition device is a gesture recognition device according to claim 1 .