JP5381569B2 - Gesture recognition device, gesture recognition method, and gesture recognition program - Google Patents

Description

  The present invention relates to an apparatus, a method, and a program for recognizing a gesture using an image obtained by a camera.

  As one of the interfaces for selecting an object displayed on the display device, the image obtained by the camera is used to move the user's hand while maintaining a predetermined shape such as the shape of a janken par. Gesture recognition for detecting hand movement has been proposed and put into practical use. An interface based on gesture recognition is realized by preparing a camera for photographing a user without wearing a special device. For this reason, an interface based on gesture recognition is expected to be applied to an environment in which an input device such as a keyboard or a mouse is not suitable (for example, an interaction using a large display provided in a public space).

  Gesture recognition is realized, for example, by detecting the movement of the user's hand from moving image data obtained by photographing the user. Hereinafter, an example of a method for detecting the movement of a user's hand using moving image data will be described with reference to FIG.

  In FIG. 1, image frames 1 to 3 are a part of moving image data obtained by a camera. The image frame 2 is a picture after the image frame 1, and the image frame 3 is a picture after the image frame 2. The image frames 1 to 3 include the image of the user's hand 101.

  In the method shown in FIG. 1, gesture recognition is realized by tracking feature points in an area corresponding to the hand 101. For example, the feature point a1 of the image frame 1 is detected as the feature point a2 in the image frame 2, and is detected as the feature point a3 in the image frame 3. In this case, it is determined that the hand 101 has moved along a path from the feature point a1 to the feature point a3 via the feature point a2. Thereby, it is recognized how the user moved the hand. A method for detecting a specific motion pattern by extracting and tracking a region corresponding to a user's hand from moving image data is described in Non-Patent Document 1, for example.

IPSJ Journal, Vol.44, No.SIG5 (CVIM 6), Apr.2003 "Multiple fingertip tracking and its application for extended desk interface"

  In the method shown in FIG. 1, it is difficult to stably recognize a gesture. In particular, when detecting the movement of the user's hand, the shape of the hand may change significantly on the image, and the hand and background (eg, the user's clothes or another person moving behind the user) Can be difficult to identify, and it is difficult to track feature points.

  For example, in the example shown in FIG. 1, the shape of the hand 101 is changed on the image, and the feature point corresponding to the feature point b1 of the image frame 1 is not detected in the image frame 2. That is, the tracking of the feature point b1 has failed. In this case, the movement of the hand 101 cannot be detected. In the image frame 2, noise 102 is generated. The noise 102 is caused by, for example, background movement. At this time, if the feature point c1 of the image frame 1 is detected as the feature point c2 in the image frame 2 due to mistracking, a gesture different from the actual hand movement is recognized. As described above, in the method of tracking feature points, erroneous detection or non-detection of feature points may occur, and a gesture cannot be recognized stably.

  An object of the present invention is to stably recognize a gesture using an image obtained by a camera.

  A gesture recognition device according to one aspect of the present invention is a device that recognizes a gesture based on a movement trajectory of an object, and in each of a plurality of image frames obtained at different times, each object region corresponding to the object A target area extracting unit for extracting a target point, a target point extracting unit for extracting a target point from the target area, and a moving direction of the target point in the plurality of image frames, respectively. Based on transition data obtained by arranging the determination results of the movement direction determination unit for determining the movement direction of the target region based on the movement direction determination unit for the plurality of image frames in time series A movement trajectory detection unit for detecting a movement trajectory of the object;

  According to the configuration or method disclosed in the present application, it is possible to stably recognize a gesture using an image obtained by a camera.

It is a figure explaining an example of the method of detecting a user's hand movement. It is a figure which shows an example of a gesture recognition system. It is a figure which shows the structure of the gesture recognition apparatus of 1st Embodiment. It is a figure explaining operation | movement of a gesture recognition apparatus. It is a figure explaining an example of the method of calculating the motion vector of an attention point. It is a figure explaining a direction field. It is a figure explaining operation | movement of a transition data collation part. It is a flowchart which shows the gesture recognition method of 1st Embodiment. It is a figure explaining the gesture recognition method of 2nd Embodiment. It is an Example of an icon image data storage part. It is a figure explaining the gesture recognition method of 3rd Embodiment. It is a figure which shows the image frame from which the several object area | region was extracted. It is a figure which shows the hardware constitutions of a gesture recognition apparatus.

  FIG. 2 is a diagram illustrating an example of a gesture recognition system that provides the gesture recognition method according to the embodiment. In this embodiment, the gesture recognition system includes a gesture recognition device 1, a camera 2, and a display device 3. In this embodiment, the gesture recognition apparatus 1 is realized by executing a gesture recognition program using a computer, and recognizes a gesture of a person (hereinafter referred to as a user) photographed by the camera 2.

  The camera 2 is installed in the vicinity of the display device 3 so as to photograph the front direction of the display device 3 (direction substantially perpendicular to the display screen). The camera 2 may be attached to the display device 3 or may be built in the display device 3. The camera 2 is a digital video camera, for example, and moving image data obtained by the camera 2 is transmitted to the gesture recognition device 1 by wire or wireless communication. The display device 3 is connected to the gesture recognition device 1 by wired or wireless communication, and can display a moving image captured by the camera 2 in real time. Therefore, when a user is positioned in front of the display device 3, image data including the user is transmitted to the gesture recognition device 1 and an image including the user is displayed on the display device 3.

  The gesture recognition device 1 recognizes a gesture of a user (that is, a user photographed by the camera 2) located in front of the display device 3 using moving image data obtained by the camera 2. In this embodiment, the gesture recognition device 1 detects in which trajectory pattern the user's hand is moved. In FIG. 2, the gesture recognition device 1 recognizes that the user has drawn a “ring” by hand.

  In this embodiment, the user can perform a desired gesture while looking at his / her appearance displayed on the display device 3 as if he / she was looking at his / her appearance in the mirror. Therefore, although not particularly limited, the display device 3 may display a horizontally reversed image of the image obtained by the camera 2 as shown in FIG. However, the display device 3 is not an essential component for realizing the gesture recognition method of the embodiment.

<First Embodiment>
FIG. 3 is a diagram illustrating a configuration of the gesture recognition device 1 according to the first embodiment. In this embodiment, the gesture recognition device 1 includes a target area extraction unit 11, a point of interest extraction unit 12, a movement direction determination unit 13, and a movement locus detection unit 14. Then, the moving image data obtained by the camera 2 is input to the gesture recognition device 1. The moving image data is not particularly limited, but is, for example, 30 frames / second. The basic configuration of the gesture recognition device is the same in the second and third embodiments described later.

  The target area extraction unit 11 extracts a target area corresponding to each target object in a plurality of image frames obtained at different times. At this time, the gesture recognition device 1 extracts, for example, each target region in each image frame constituting the input moving image data. However, the target area extracting unit 11 may extract the target area from each image frame of the moving image data that has been thinned out at a predetermined rate.

  In this embodiment, the object is the user's “hand” taken by the camera 2. Therefore, the target area extraction unit 11 extracts an image area corresponding to the user's hand in each image frame. The image area corresponding to the user's hand is extracted based on the color of the pixel, for example. In addition, the gesture recognition device 1 detects and recognizes a user's gesture (that is, “movement”). Therefore, the target area extraction unit 11 may extract an area moving in the image frame as an image area corresponding to the user's hand. Further, the target area extraction unit 11 may extract an image area corresponding to the user's hand by combining the two extraction conditions.

  FIG. 4 is a diagram for explaining the operation of the gesture recognition device 1. In FIG. 4A, image frames 1 to 3 are a part of moving image data obtained by the camera 2. The image frame 2 is a picture after the image frame 1, and the image frame 3 is a picture after the image frame 2. Here, the image frames 1 to 3 may be, for example, three consecutive image frames.

  In this embodiment, the target area 21 is an image area corresponding to the user's hand photographed by the camera 2 as described above. Here, in the example shown in FIG. 4A, the position of the user's hand moves and the shape thereof changes with the passage of time. For this reason, in the image frames 1 to 3, the position of the target area 21 has moved, and the shape of the target area 21 has also changed. Note that only the target area 21 is detected in the image frames 1 and 3, but the target area 21 and the target area 22 are detected in the image frame 2. The target area 22 is an area detected according to the above-described extraction condition, but is not an image area corresponding to the user's hand. That is, the target area 22 is a noise image area. Note that the noise image region is generated due to, for example, the movement of the background of the user.

  The point-of-interest extraction unit 12 extracts one or more points of interest from the target regions (21, 22) extracted by the target region extraction unit 11 in each image frame. In this embodiment, a plurality of points of interest are extracted in each image frame. In the image frame 1, the points of interest a <b> 1 to g <b> 1 are extracted from the target region 21. In the image frame 2, the points of interest a <b> 2 to g <b> 2 are extracted from the target region 21, and the point of interest h <b> 2 is extracted from the target region 22. Furthermore, in the image frame 3, the points of interest a <b> 3 to g <b> 3 are extracted from the target region 21.

  The point of interest is not particularly limited, but is a feature point on the image. A feature point is a point with a large change in pixel value (or a pixel at a position with a large change in pixel value), and can be extracted using a known technique (for example, a digital filter). Note that the feature points exist at the corners of the edges on the image, for example. The pixel value is, for example, luminance information or color information of each pixel. Here, if a feature point is used as a point of interest, the accuracy of motion vector calculation described later is improved. However, the point of interest does not need to be a feature point, and may be, for example, the center of gravity of the target region. Note that the point-of-interest extraction unit 12 can extract one or more points of interest independently for each image frame. That is, the point-of-interest extraction unit 12 can extract the points of interest a2 to g2 and a3 to g3 regardless of the points of interest a1 to g1 extracted in the image frame 1, and the points of interest extracted in the image frame 2 Regardless of the points a2 to g2, the points of interest a3 to g3 can be extracted. As described above, in the gesture recognition method according to the embodiment, it is not necessary to track a point of interest (or a feature point) between image frames, and it is not necessary to associate a point of interest between image frames.

  The movement direction determination unit 13 includes a motion vector calculation unit 13a and a histogram creation unit 13b. Then, the movement direction determination unit 13 determines the movement direction of the target area extracted by the target area extraction unit 11 in each image frame. Here, the moving direction of the target area is determined based on the moving direction of each point of interest extracted from the target area.

  The motion vector calculation unit 13a calculates a motion vector of each target point. Here, a method for calculating a motion vector of an arbitrary image region in moving image data is not particularly limited, and a known technique can be used.

  FIG. 5 is a diagram for explaining an example of a method for calculating a motion vector of a point of interest. Here, FIG. 5A shows an image frame (hereinafter referred to as the frame) in which the motion vector of the point of interest is to be calculated. In the following description, it is assumed that the pixel located at the coordinates (4, 4) is the point of interest in FIG. FIG. 5B shows an image frame immediately before the frame. In FIGS. 5A and 5B, the numerical value corresponding to each pixel corresponds to, for example, a pixel value such as luminance information or a calculation result obtained by performing a predetermined filter operation on the entire image frame. To do.

  First, the motion vector calculation unit 13a extracts a 3 × 3 pixel region (hereinafter referred to as a region 41) including a target pixel and eight pixels adjacent to the target pixel in the frame illustrated in FIG. . Subsequently, the motion vector calculation unit 13a searches for a 3 × 3 pixel region having the highest correlation with the region 41 in the image frame illustrated in FIG. The correlation between regions is calculated by, for example, the sum of absolute values of pixel value differences between corresponding pixels. In this case, for example, a 3 × 3 pixel region having the minimum sum is extracted. In the example shown in FIG. 5, a region 42 surrounded by a broken line frame is detected. Then, based on the coordinate difference between the regions 41 and 42, the motion vector of the point of interest is calculated. In this example, the coordinate of the point of interest in the frame is (4, 4), and the center coordinate of the area 42 of the immediately preceding frame is (5, 5). Therefore, “−1, −1” is obtained as the motion vector of this point of interest. In the coordinate system described above, the right direction is the X-axis positive direction and the lower direction is the Y-axis positive direction on the paper surface shown in FIG.

  Hereinafter, similarly, the motion vector calculation unit 13a calculates the motion vector of each point of interest in each image frame. Note that the motion vector of the point of interest is calculated using, for example, an image frame including the point of interest and an image frame immediately before the frame, but is not limited thereto. That is, a motion vector may be calculated using an image frame including a point of interest and an image frame two or more before the image frame. Alternatively, the motion vector may be calculated using an image frame including the point of interest and an image frame after the image frame.

  The histogram creation unit 13b determines which of the direction areas 1 to 9 illustrated in FIG. 6 the moving direction of each point of interest belongs to in each image frame. For example, in the example illustrated in FIG. 5, the movement vector of the point of interest is “−1, −1”. In this case, the moving direction of this point of interest is 135 °. Then, this point of interest is determined to belong to the direction area 4. Furthermore, the direction area to which the moving direction belongs is similarly determined for other points of interest. Then, the histogram creation unit 13b creates a moving direction histogram by counting the number of points of interest belonging to each direction area.

  FIG. 4B is an example of a moving direction histogram created for the image frames 1 to 3 shown in FIG. In the image frame 1, many points of interest (for example, a1, d1, e1, f1) are moving rightward or substantially rightward. Therefore, in the histogram for the image frame 1, the number of points of interest belonging to the direction region 1 (0 to 40 °) is the largest. In the image frame 2, many points of interest (for example, b2, d2, f2, and g2) are moved in the lower right direction or substantially in the lower right direction. Therefore, in the histogram for the image frame 2, the number of points of interest belonging to the direction area 8 (280 to 320 °) is the largest. Furthermore, in the histogram for the image frame 3, the number of points of interest belonging to the direction area 7 (240 to 280 °) is the largest.

  The histogram creation unit 13b determines the movement direction of the target region 21 for each image frame based on the created movement direction histogram. In this example, the direction region with the largest number of points of interest (the direction region with the highest frequency of interest of points of interest) is determined as the likely moving direction of the target region 21. Therefore, in the example illustrated in FIG. 4, “1”, “8”, and “7” are output as the determination results of the moving direction of the target area 21 for the image frames 1, 2, and 3, respectively.

  The movement trajectory detection unit 14 includes a transition data creation unit 14a, a transition data verification unit 14b, and a gesture determination unit 14c. Then, the movement trajectory detection unit 14 detects the movement trajectory of the object based on data generated by arranging the determination results of the movement direction determination unit 13 for each image frame in time series. Here, this time-series data represents a temporal change in the moving direction of the reference region 21 in each image frame. Therefore, hereinafter, this time-series data may be referred to as direction transition data.

  The transition data creation unit 14a generates direction transition data by arranging the determination results of the moving direction of the target region in each image frame in time series. For example, when the moving direction histogram shown in FIG. 4B is created for the image frames 1 to 3 shown in FIG. 4A, the determination results “1”, “8”, and “7” are obtained. In this case, the direction transition data shown in FIG. 4C is generated.

The transition data collating unit 14b compares the direction transition data generated by the transition data creating unit 14a with reference pattern data representing a predetermined movement locus in the same format as the direction transition data. Here, it is assumed that the predetermined movement trajectory is “circle”. When the movement locus is a circle, the moving direction of the object changes continuously with time. That is, in this case, the reference pattern data is expressed as follows, for example. In the following example, the reference pattern data is represented by 18 digits.
Reference pattern data (circle) = 112234345567678899
In this example, it is assumed that the following direction transition data is obtained.
Direction transition data = 13223335555656788889

  As shown in FIG. 7, the transition data matching unit 14 b calculates the absolute value of the difference value between the direction transition data and the reference pattern data for each digit. Here, the value of each digit of the direction transition data represents the direction area shown in FIG. 6 (that is, the moving direction of the target area 21) as described above.

  The transition data matching unit 14b calculates the sum of the absolute difference values for each digit. In the example shown in FIG. 7, “6” is obtained. Further, the transition data collating unit 14b compares this sum with a preset collation threshold value. The comparison result is given to the gesture determination unit 14c.

  The gesture determination unit 14c determines that the movement trajectory represented by the direction transition data is similar to the movement trajectory represented by the reference pattern data if the sum is smaller than the collation threshold. In the above example, it is determined that the movement trajectory of the object (that is, the user's hand photographed by the camera 2) is a circle. In other words, the gesture determination unit 14c determines that the user has made a gesture for drawing a circle by hand.

  FIG. 8 is a flowchart illustrating the gesture recognition method according to the first embodiment. In step S <b> 1, each image frame of moving image data obtained by the camera 2 is input to the gesture recognition device 1. In step S2, the target area extraction unit 11 extracts the target area in each image frame. In the above example, the target area is an image area corresponding to the hand of the user photographed by the camera 2. In step S3, the point-of-interest extraction unit 12 extracts a plurality of points of interest from the extracted target area in each image frame. The point of interest is, for example, a feature point of the image.

  In step S4, the motion vector calculation unit 13a calculates the motion vector of each point of interest in each image frame. Then, the motion vector calculation unit 13a determines the moving direction of each point of interest (the direction area shown in FIG. 6 in the above example) based on the calculated motion vector. In step S <b> 5, the histogram creation unit 13 b creates a movement direction histogram representing the appearance frequency of each focus point in the movement direction for each image frame. The moving direction histogram created for each image frame is stored in a memory area provided in the gesture recognition device 1. At this time, the gesture recognition device 1 may hold only the movement direction histogram for the image frames for the latest several seconds.

  In step S6, the transition data creation unit 14a refers to the movement direction histogram for each image frame and determines the movement direction with the highest appearance frequency. Then, the transition data creating unit 14a generates direction transition data by arranging the determination results for each image frame (that is, data representing the likely moving direction of the target region) in time series. In step S7, the transition data matching unit 14b calculates a difference between the created direction transition data and reference pattern data. The reference pattern data represents a specific gesture determined in advance.

  In step S8, the gesture determination unit 14c compares the difference between the direction transition data and the reference pattern data with a matching threshold value. And if the said difference is smaller than a collation threshold value, the gesture determination part 14c will determine with the user having performed the said specific gesture. On the other hand, if the difference is equal to or greater than the collation threshold, the gesture determination unit 14c determines that the user is not performing the specific gesture.

  As described above, the gesture recognition device 1 according to the first embodiment extracts a target area (an image area corresponding to a user's hand) for each image frame, and extracts a point of interest from the target area. Then, the movement direction from the previous image frame is specified for the extracted point of interest, and the movement locus of the object (user's hand) is detected based on the time change of the movement direction. That is, the user's gesture is recognized. Thereby, even if the time change of the position and shape of the object is large on the image, the movement trajectory can be detected.

  In addition, the gesture recognition device 1 extracts a plurality of points of interest for each image frame, and calculates a motion vector (that is, a moving direction) of each point of interest. The moving direction of each point of interest is managed by a histogram that counts the number of appearances for each predetermined angle width. Using this histogram, the moving direction with the highest appearance frequency is specified, and the specified moving direction is determined as the moving direction of the object. For this reason, for example, even when the moving direction of each point of interest is not constant due to a change in the shape of the object (user's hand), the moving direction of the entire object can be detected with high accuracy. Further, as shown in FIG. 4A, even if noise (target region 22) occurs in the image frame 2, the influence of the noise is appropriately removed or suppressed. Therefore, the gesture recognition device 1 according to the first embodiment can accurately detect the movement trajectory of the object and can accurately recognize the user's gesture.

  Next, the method shown in FIG. 1 is compared with the method of the first embodiment. In the method shown in FIG. 1, first, feature points are extracted in a certain image frame, and the feature points are tracked in subsequent image frames. That is, feature points are correlated in time series over a plurality of image frames. For this reason, if the position and shape of the user's hand changes greatly, it is difficult (or impossible) to detect feature points to be associated between image frames.

  On the other hand, according to the method of the first embodiment, a plurality of points of interest are extracted from regions corresponding to the user's hand in each image frame, and the moving direction of each point of interest is specified. Then, the dominant moving direction is determined based on the moving direction with the highest appearance frequency. At this time, the process of determining the moving direction can be performed independently for each image frame, and it is not necessary to track and associate the point of interest over a plurality of image frames. For this reason, according to the method of the first embodiment, there is no problem of misdetection or non-detection of feature points, and recognition accuracy does not decrease.

Note that when the user performs a gesture in front of the camera 2, the moving speed of the hand varies from user to user. For this reason, the gesture recognition device 1 may hold a plurality of reference pattern data having different data lengths and collate the direction transition data with each reference pattern data. In this case, the gesture recognition device 1 holds the following reference pattern data, for example.
Reference pattern (high speed) = 1234456789
Reference pattern (medium speed) = 1122334455567787899
Reference pattern (low speed) = 11212223334445555666778889999
Alternatively, the gesture recognition device 1 may expand or compress the data length of the generated direction transition data so as to match the data length of the reference pattern data as necessary.

Further, when the user makes a gesture of drawing a circle by hand, the start position of the movement locus differs for each user. For example, one user may start a trajectory from the lowest point of the circle, and another user may start a trajectory from the highest point of the circle. Therefore, the gesture recognition apparatus 1 may hold a plurality of reference pattern data having different start positions, and collate the direction transition data with each reference pattern data. In this case, the gesture recognition device 1 holds the following reference pattern data, for example.
Reference pattern = 112234345456678789
Reference pattern = 222334455566778789911
Reference pattern = 3344555667788991122
Reference pattern = 4455667778899112233
Reference pattern = 5566777889911223344
Reference pattern = 6677788991122334455
Reference pattern = 778889911223454566
Reference pattern = 889911223344565677
Reference pattern = 99112233345456667788

Furthermore, when the user makes a gesture of drawing a circle by hand, the circle may be drawn clockwise or the circle may be drawn counterclockwise. Therefore, the gesture recognition device 1 may hold, for example, the following two reference pattern data, and collate the direction transition data with each reference pattern data.
Reference pattern = 112234345456678789
Reference pattern = 11998887766555443322

  In the above-described embodiment, one movement direction is specified in the movement direction histogram for each image frame, but the appearance frequencies of two or more movement directions may be substantially the same. For example, in the image frame 1, it is assumed that the appearance frequencies of the direction area 1 and the direction area 2 are approximately the same. In this case, the movement locus detection unit 14 includes first direction transition data in which the direction area data of the image frame 1 is “1”, and second direction transition data in which the direction area data of the image frame 1 is “2”. Is generated. Then, the movement trajectory detection unit 14 collates the first and second direction transition data with the reference pattern data. At this time, if the sum of the differences between one of the first and second direction transition data and the reference pattern data is smaller than the collation threshold, the movement trajectory detection unit 14 causes the user to perform a gesture corresponding to the reference pattern data. It can be determined that it has been performed.

  Furthermore, in the above-described embodiment, a plurality of points of interest are extracted in each image frame, and a gesture is recognized based on the moving direction of each point of interest. However, the present invention is not limited to this. That is, one focus point may be extracted from the target area in each image frame, and the gesture may be recognized based on the moving direction of the focus point. However, if the number of points of interest to be extracted in each image frame is increased, the gesture recognition accuracy increases.

  Furthermore, in the above-described embodiment, the gesture recognition device 1 recognizes a gesture in which a user draws a “circle” by hand, but the present invention is not limited to this. That is, the gesture recognition device 1 according to the first embodiment may recognize other gestures. For example, the gesture recognition device 1 can recognize figures, characters, numbers, and the like that can be drawn with a single stroke. In this case, the gesture recognition apparatus 1 holds reference pattern data corresponding to each figure, character, and number, and matches the direction transition data created from each image frame with each reference pattern data, thereby making it possible to recognize the user's gesture. recognize.

<Second Embodiment>
As shown in FIG. 9, the gesture recognition apparatus according to the second embodiment can superimpose a desired image on a moving image obtained by the camera 2. The image to be superimposed is not particularly limited, but is, for example, an image representing an object selected by the user (hereinafter, an icon image). In FIG. 9, icons 23 to 25 are displayed on the display device 3. In this case, when the user is positioned in front of the display device 3, the user image captured by the camera 2 and the icons 23 to 25 are superimposed on the display device 3. At this time, the user can see his / her appearance displayed on the display device 3 (and the icons 23 to 25 displayed in a superimposed manner).

  When the user selects a desired icon displayed on the display device 3, the image corresponding to his / her hand draws a circle on the screen of the display device 3 in the vicinity of the icon to be selected. Move your hand. In the example shown in FIG. 9, the icon 23 is selected. At this time, if the gesture recognition device 1 detects that the movement locus of the target region corresponding to the user's hand is a circle in the region near the icon 23, the gesture recognition device 1 determines that the user has selected the icon 23.

  The gesture recognition device 1 according to the second embodiment includes an icon image data storage unit 31 and an image superimposing unit 32 indicated by dotted lines in addition to the configuration shown in FIG. As shown in FIG. 10, the icon image data storage unit 31 stores image data, display position data, and determination area data for each icon displayed superimposed on the display device 3. The image data is, for example, bitmap data of icons to be displayed. The display position data represents the center coordinates of the icon display area on the display screen of the display device 3. The determination area data specifies the range of the determination area for detecting the movement trajectory of the object on the display screen of the display device 3. In this example, the determination area is a rectangle, and coordinates of the upper left corner and the lower right corner are stored.

  The image superimposing unit 32 creates display data by superimposing an icon image on each image frame of moving image data input from the camera 2. The display position of each icon follows the display position data stored in the icon image data storage unit 31. Then, the display device 3 superimposes and displays the camera image and the icon image according to the display data.

  The target area extraction unit 11 extracts a target area (here, an image area corresponding to the user's hand) in each image frame. However, in the second embodiment, the target area extraction unit 11 may extract the target area within the determination area specified by the determination area data, instead of extracting the target area from the entire image frame. . In the example shown in FIG. 9, determination areas 26 to 28 are set for the icons 23 to 25, respectively. Although the determination areas 26 to 28 are drawn with broken lines in FIG. 9, they are not displayed on the screen of the display device 3. However, the determination area may be displayed on the display device 3.

  The operations of the point-of-interest extraction unit 12, the movement direction determination unit 13, and the movement locus detection unit 14 are basically the same as those described with reference to FIGS. That is, a plurality of points of interest are extracted from the target region, the moving direction of each point of interest is calculated, and the moving direction of the object in each image frame is determined based on the moving direction with a high appearance frequency. Then, the movement trajectory of the object is detected based on the temporal change in the movement direction of the object.

  However, the gesture recognition apparatus according to the second embodiment has a function of specifying in which determination region 26 to 28 the movement trajectory of the object by the user's gesture is detected. For example, when a predetermined movement trajectory (in this example, a trajectory for drawing a circle) determined in advance in the determination area 26 is detected, the gesture recognition device determines that the icon 23 has been selected by the user. In this case, the gesture recognition device (or another computer connected to the gesture recognition device) executes processing corresponding to the selected icon.

  As described above, the gesture recognition apparatus according to the second embodiment can detect an object selected by the user from among the selection objects displayed on the display device 3 by recognizing the user's gesture. Note that in the second embodiment, if image processing (region extraction, focus point extraction, motion vector calculation, etc.) is performed only for the determination region, the processing amount of the gesture recognition device is reduced.

<Third Embodiment>
Similar to the first or second embodiment, the gesture recognition device of the third embodiment includes a target region extraction unit 11, a point of interest extraction unit 12, a movement direction determination unit 13, and a movement trajectory detection unit 14, A movement trajectory is detected based on a temporal change in the moving direction of the object. And the gesture recognition apparatus of 3rd Embodiment complements the detection process of the said movement locus | trajectory using the data showing the time change of the positional information on the target object on an image frame. The third embodiment will be described below with reference to FIG.

  Also in the third embodiment, the target area extraction unit 11 extracts a target area (an image area corresponding to the user's hand) in each image frame. Subsequently, the gesture recognition device calculates the barycentric position of the extracted target region in each image frame. The center-of-gravity position is obtained, for example, by calculating the center-of-gravity coordinates of a plurality of points of interest extracted by the point of interest extraction unit 12. The barycentric position is represented by position detection blocks 1 to 9 shown in FIG. The position detection block is set by dividing an image frame or a partial area thereof into a plurality of areas.

The gesture recognition device according to the third embodiment compares position transition data obtained by arranging centroid positions of image frames in time series and reference pattern data. This reference pattern has the same data format as the position transition data, and represents the movement trajectory of the object corresponding to a predetermined gesture. Here, assuming that the movement trajectory of the object is a circle, the object moves from the position detection block 1 to the position detection blocks 2, 3, 6, 9 on the image frame as shown in FIG. , 8, 7 and 4 in order, and then move along the route returning to the position target block 1. In this case, the reference pattern data is expressed as follows, for example. In the following example, the reference pattern data is represented by 19 digits.
Reference pattern data (circle) = 1123363699987744112
In this example, it is assumed that the following position transition data is obtained by arranging the centroid positions of the target regions of the image frames in time series.
Transition data = 112223336698888774411

  Subsequently, the gesture recognition device calculates the degree of difference between the position transition data and the reference pattern data. The degree of difference is obtained by calculating the absolute value of the difference between the values of the position transition data and the reference pattern data for each digit and calculating the sum of the absolute values of the differences. If the correlation is smaller than a predetermined threshold value, it is determined that the movement locus of the object is a circle.

  Note that the gesture recognition device may calculate the degree of correlation between the position transition data and the reference pattern data. The degree of correlation is obtained by comparing the values of position transition data and reference pattern data with each other and counting the number of matching digits for each digit. In this case, if the degree of correlation is higher than a predetermined threshold, it is determined that the movement locus of the object is a circle.

  Furthermore, the gesture recognition device compares the detection result based on the temporal change in the moving direction of the target object with the detection result based on the temporal change in the position information of the target object. If these detection results match each other, the detection results are output. For example, when the movement trajectory obtained based on the time-series data in the movement direction is a circle and the movement trajectory obtained based on the time-series data in the position information is also a circle, the gesture recognition device causes the user to Recognize that you have made a drawing gesture. On the other hand, even if the movement trajectory obtained based on the time-series data in the moving direction is a circle, if a similar detection result is not obtained based on the time-series data of the position information, the gesture recognition device allows the user to change the circle. It is possible not to recognize that a gesture has been made. Therefore, according to the third embodiment, the recognition accuracy of the user's gesture is further improved.

  As described above, in the third embodiment, the movement trajectory of the object is detected based on the movement direction of the object on the image, and further, the detection is performed using the position information of the object on the image. Processing is complemented. Therefore, for example, as shown in FIG. 12, when the target area extraction unit 11 detects a plurality of target areas 21 and 22 and the moving directions of the points of interest on the target areas are different from each other, The detection accuracy of the movement trajectory is improved. Alternatively, the possibility of erroneously recognizing the user's gesture is reduced.

  The configurations and operations of the first to third embodiments may be arbitrarily combined as long as no contradiction arises.

<Hardware configuration of gesture recognition device>
FIG. 13 is a diagram illustrating a hardware configuration of the gesture recognition device. In FIG. 13, the CPU 51 provides a gesture recognition method according to the embodiment by executing a gesture recognition program using the memory 53. The storage device 52 is, for example, a hard disk, and stores a gesture recognition program. The storage device 52 may be an external recording device. The memory 53 is a semiconductor memory, for example, and includes a RAM area and a ROM area. The reference pattern data is stored in the storage device 52 or the memory 53, for example.

  The reading device 54 accesses the portable recording medium 55 according to an instruction from the CPU 51. The portable recording medium 55 includes, for example, a semiconductor device, a medium in which information is input / output by a magnetic action, and a medium in which information is input / output by an optical action. The communication interface 56 transmits / receives data via a network in accordance with instructions from the CPU 51. The input / output device 57 corresponds to, for example, a device that receives an instruction from a user.

The gesture recognition program according to the embodiment is provided in the following form, for example.
(1) Installed in advance in the storage device 52.
(2) Provided by the portable recording medium 55.
(3) Download from the program server 60.

  And the gesture recognition apparatus concerning embodiment is implement | achieved by running a gesture recognition program with the computer of the said structure. That is, by executing the gesture recognition program on the computer having the above-described configuration, a part or all of the target region extraction unit 11, the point of interest extraction unit 12, the movement direction determination unit 13, and the movement locus detection unit 14 is realized.

The following supplementary notes are further disclosed with respect to the embodiments including the above examples.
(Appendix 1)
A gesture recognition device for recognizing a gesture based on a movement trajectory of an object,
In a plurality of image frames obtained at different times, respectively, a target area extraction unit that extracts a target area corresponding to the target object;
In each of the plurality of image frames, a point-of-interest extraction unit that extracts a point of interest from the target region,
In each of the plurality of image frames, a moving direction determination unit that determines a moving direction of the target area based on a moving direction of the point of interest;
A movement trajectory detection unit for detecting a movement trajectory of the object based on transition data obtained by arranging the determination results of the movement direction determination unit for the plurality of image frames in time series;
Gesture recognition device.
(Appendix 2)
The gesture recognition device according to attachment 1, wherein
The point-of-interest extraction unit extracts a plurality of points of interest from the target area in each of the plurality of image frames,
The gesture recognition device, wherein the movement direction determination unit determines a likely movement direction of the target region based on a movement direction of each target point in each of the plurality of image frames.
(Appendix 3)
The gesture recognition device according to attachment 2, wherein
The moving direction determination unit determines in each of the plurality of image frames whether a moving direction of each target point belongs to a plurality of direction areas divided by a predetermined angle, and the moving direction of the target point is The gesture recognizing device, wherein the likely moving direction is determined based on a direction region having a high frequency of belonging.
(Appendix 4)
The gesture recognition device according to attachment 3, wherein
When the movement direction of the point of interest belongs to the first and second direction areas, the movement trajectory detection unit performs a first transition including the movement direction determined based on the first direction area. A gesture recognizing device, wherein the movement trajectory of the object is detected using second transition data including a movement direction determined based on the data and the second direction area.
(Appendix 5)
The gesture recognition device according to any one of appendices 1 to 4,
An image superimposing unit that generates an image frame by superimposing a selection target image on a camera image capturing a person;
The object is the hand of the person;
The gesture recognition device, wherein the target region extraction unit extracts a target region corresponding to the hand of the person in a predetermined determination region including the selection target image in the image frame.
(Appendix 6)
The gesture recognition device according to any one of appendices 1 to 5,
The gesture recognizing apparatus, wherein the movement trajectory detection unit detects a movement trajectory of the object by comparing reference pattern data representing a predetermined movement trajectory with the transition data.
(Appendix 7)
The gesture recognition device according to any one of appendices 1 to 6,
The movement trajectory detection unit generates position information representing a temporal change in the position of the object using the plurality of image frames, and detects the movement trajectory of the object based on the transition data and the position information. The gesture recognition apparatus characterized by the above-mentioned.
(Appendix 8)
In order to recognize a gesture based on the movement trajectory of the object,
In a plurality of image frames obtained at different times, a target area extracting unit that extracts a target area corresponding to the target object,
In the plurality of image frames, a point-of-interest extraction unit that extracts a point of interest from the target region,
In each of the plurality of image frames, a movement direction determination unit that determines a movement direction of the target area based on a movement direction of the point of interest,
A movement trajectory detection unit for detecting a movement trajectory of the object based on transition data obtained by arranging the determination results of the movement direction determination unit for the plurality of image frames in time series;
Gesture recognition program to function as.
(Appendix 9)
A gesture recognition method for recognizing a gesture based on a movement trajectory of an object,
In a plurality of image frames obtained at different times, respectively, extract a target area corresponding to the target object,
In each of the plurality of image frames, a point of interest is extracted from the target area,
In each of the plurality of image frames, the moving direction of the target area is determined based on the moving direction of the point of interest,
Detecting a movement locus of the object based on transition data obtained by arranging the determination results for the plurality of image frames in time series;
A gesture recognition method characterized by the above.

DESCRIPTION OF SYMBOLS 1 Gesture recognition apparatus 2 Camera 3 Display apparatus 11 Target area extraction part 12 Point of interest extraction part 13 Movement direction determination part 13a Motion vector calculation part 13b Histogram creation part 14 Movement locus detection part 14a Transition data creation part 14b Transition data collation part 14c Gesture Determination unit 31 Icon image data storage unit 32 Image superimposition unit

Claims (6)

A gesture recognition device for recognizing a gesture based on a movement trajectory of an object,
In a plurality of image frames obtained at different times, respectively, a target area extraction unit that extracts a target area corresponding to the target object;
In each of the plurality of image frames, a point-of-interest extraction unit that extracts a point of interest from the target region,
In each of the plurality of image frames, a moving direction determination unit that determines a moving direction of the target area based on a moving direction of the point of interest;
A first detection unit that detects a movement trajectory pattern of the object based on transition data obtained by arranging the determination results of the movement direction determination unit for the plurality of image frames in time series ;
A second detection unit that generates position transition data representing a temporal change in the position of the object using the plurality of image frames, and detects a movement trajectory pattern of the object based on the position transition data;
A gesture recognition unit that identifies a gesture corresponding to the detected movement trajectory pattern when the movement trajectory patterns detected by the first detection unit and the second detection unit coincide with each other;
Gesture recognition device. The gesture recognition device according to claim 1,
The point-of-interest extraction unit extracts a plurality of points of interest from the target area in each of the plurality of image frames,
The gesture recognition device, wherein the movement direction determination unit determines a likely movement direction of the target region based on a movement direction of each target point in each of the plurality of image frames. The gesture recognition device according to claim 2,
The moving direction determination unit determines in each of the plurality of image frames whether a moving direction of each target point belongs to a plurality of direction areas divided by a predetermined angle, and the moving direction of the target point is The gesture recognizing device, wherein the likely moving direction is determined based on a direction region having a high frequency of belonging. The gesture recognition device according to any one of claims 1 to 3,
An image superimposing unit that generates an image frame by superimposing a selection target image on a camera image capturing a person;
The object is the hand of the person;
The gesture recognition device, wherein the target region extraction unit extracts a target region corresponding to the hand of the person in a predetermined determination region including the selection target image in the image frame. In order to recognize a gesture based on the movement trajectory of the object,
In a plurality of image frames obtained at different times, a target area extracting unit that extracts a target area corresponding to the target object,
In the plurality of image frames, a point-of-interest extraction unit that extracts a point of interest from the target region,
In each of the plurality of image frames, a movement direction determination unit that determines a movement direction of the target area based on a movement direction of the point of interest,
A first detection unit that detects a movement trajectory pattern of the object based on transition data obtained by arranging the determination results of the movement direction determination unit for the plurality of image frames in time series;
A second detection unit that generates position transition data representing a temporal change in the position of the object using the plurality of image frames, and detects a movement trajectory pattern of the object based on the position transition data;
A gesture recognition unit that identifies a gesture corresponding to the detected movement trajectory pattern when the movement trajectory patterns detected by the first detection unit and the second detection unit coincide with each other;
Gesture recognition program to function as. A gesture recognition method for recognizing a gesture based on a movement trajectory of an object,
In a plurality of image frames obtained at different times, respectively, extract a target area corresponding to the target object,
In each of the plurality of image frames, a point of interest is extracted from the target area,
In each of the plurality of image frames, the moving direction of the target area is determined based on the moving direction of the point of interest,
Based on the direction transition data obtained by arranging the determination results for the plurality of image frames in time series, the movement trajectory pattern of the object is detected ,
Using the plurality of image frames to generate position transition data representing temporal changes in the position of the object, detecting a movement trajectory pattern of the object based on the position transition data;
When the movement trajectory pattern obtained based on the direction transition data and the movement trajectory pattern obtained based on the position transition data match each other, a gesture corresponding to the detected movement trajectory pattern is specified.
A gesture recognition method characterized by the above.