JP4992618B2 - Gesture recognition device and gesture recognition method - Google Patents

Description

  The present invention relates to a gesture recognition device and a gesture recognition method.

2. Description of the Related Art A gesture recognition device that makes a computer recognize the meaning content of a gesture is known. For example, in Patent Document 1, a frame image obtained by capturing an image of a person who performs a gesture is divided into a plurality of image blocks, a motion amount indicating a change between frames is calculated for each image block, and each block is calculated. An apparatus for discriminating a gesture based on a movement amount is disclosed.
JP 2006-235771 A

  However, in the remote control device described in Patent Document 1, for example, the influence of noise due to shooting conditions is not considered, and the gesture is not recognized from the entire divided block. There was a problem.

  More specifically, in the above-described remote control device, even if the amount of movement increases due to the influence of noise, if the amount of movement exceeds a predetermined threshold, it is determined that there is movement in the block. End up. For this reason, even if there is no gesture, there is a possibility that a gesture is erroneously recognized. For example, if the shooting range includes something that is constantly moving, such as a television, it recognizes that there is a gesture as a result of recognizing the movement on the screen such as a television that should not be recognized as the movement due to the gesture as the movement due to the gesture. there's a possibility that.

  The present invention has been made in view of the above problems, and an object thereof is to provide a gesture recognition device and a gesture recognition method with high recognition accuracy.

In order to achieve the above object, a gesture recognition apparatus according to the first aspect of the present invention includes:
In a gesture recognition device for recognizing a gesture based on an action of an imaged object,
Recognition condition setting means for setting gesture recognition conditions;
A motion amount detection means for detecting a motion amount based on the amount of change in luminance of each pixel in the divided region for each divided region obtained by dividing the captured image into a plurality of regions;
A motion region detection unit that detects a segmented region in which a motion amount detected by the motion amount detection unit is equal to or greater than a predetermined threshold as a motion region;
Specific area determination means for determining whether or not the movement area detected by the movement area detection means is a specific area to be adopted for gesture recognition;
Gesture recognition means for recognizing a gesture based on at least a motion area determined to be a specific area by the specific area determination means and a gesture recognition condition set by the recognition condition setting means;
It is characterized by comprising.

The recognition condition setting means includes threshold setting means for setting the predetermined threshold,
The motion region detection unit may detect a divided region in which the motion amount detected by the motion amount detection unit is equal to or more than a threshold set by the threshold setting unit as a motion region.

  The threshold value setting means may set the threshold value based on the amount of change in luminance of each pixel of sequentially captured images.

  The motion amount detection means may detect a sum of values obtained by squaring the amount of change in luminance of each pixel in the divided region as a motion amount.

  The specific area determination unit may not determine that the movement area detected by the movement area detection unit for a predetermined time or longer is the specific area.

  The specific area determination unit is included in a predetermined area including the plurality of movement areas among the movement areas detected by the movement area detection unit when a plurality of movement areas are detected by the movement area detection unit. The motion area to be moved may be determined as the specific area.

In the case where there are a plurality of motion areas determined to be specific areas by the specific area determination means, the movement of the gesture in the image is determined from the positions of the plurality of motion areas determined to be the specific areas in the image. A center position detecting means for detecting a center position;
The gesture recognition means may recognize a gesture based on the center position detected by the center position detection means.

Movement direction determination means for determining the direction of movement of a gesture from a plurality of captured images;
In a case where there are a plurality of motion areas determined to be the specific area by the specific area determination unit, the motion detected by the center position detection unit among the plurality of motion areas determined to be the specific area. Tip position detecting means for detecting a movement region existing at the most distant position in the direction of movement of the gesture determined by the movement direction determining means with reference to the center position; and
The gesture recognizing unit may recognize a gesture based on a position in the image of a motion region present at the most distant position detected by the tip position detecting unit.

Imaging time acquisition means for acquiring the imaging time of images taken sequentially, further comprising:
The gesture recognizing unit obtains the time when the specific region determining unit continuously determines that the motion region is a specific region based on the image capturing time of the image acquired by the imaging time acquisition unit, It may be recognized as a gesture according to the given time.

In order to achieve the above object, a gesture recognition method according to the second aspect of the present invention includes:
In a gesture recognition method for recognizing a gesture based on an action of an imaged object,
A recognition condition setting step for setting a recognition condition for the gesture;
A motion amount detection step for detecting a motion amount based on the amount of change in luminance of each pixel in the divided region for each divided region obtained by dividing the captured image into a plurality of regions;
A motion region detection step of detecting, as a motion region, a divided region in which the motion amount detected in the motion amount detection step is equal to or greater than a predetermined threshold;
A specific region determination step for determining whether or not the motion region detected in the motion region detection step is a specific region to be adopted for gesture recognition;
A gesture recognition step for recognizing a gesture based on at least a motion region determined to be a specific region in the specific region determination step and a gesture recognition condition set in the recognition condition setting step;
It is characterized by comprising.

  According to the gesture recognition apparatus and the gesture recognition method according to the present invention, it is possible to improve gesture recognition accuracy.

  Hereinafter, the configuration and operation of a gesture recognition device according to an embodiment of the present invention will be described with reference to the drawings.

  First, the physical configuration of the gesture recognition apparatus 100 according to the embodiment of the present invention will be described with reference to FIG.

  As shown in FIG. 1, the gesture recognition device 100 is realized by the CPU 20 executing a program stored in the ROM 30, and the screen 400 supplied from the camera 200 and the person to be imaged 500 are included in the imaging range. Recognize gestures based on the included image data. In addition, the gesture recognition apparatus 100 displays image corresponding to the image data on the screen 400 by supplying the projector 300 with image data corresponding to the command corresponding to the recognized gesture.

  The gesture recognition apparatus 100 includes a bus 10, a CPU (Central Processing Unit) 20, a ROM (Read Only Memory) 30, a RAM (Random Access Memory) 40, an I / O (Input Output) unit 50, and a hard disk 60. An input buffer 70, an output buffer 80, and a timer counter 90.

  The bus 10 connects each device in the gesture recognition apparatus 100 to each other. Each device transmits and receives data to and from each other via the bus 10.

  The CPU 20 executes a program stored in the ROM 30, thereby analyzing the image data supplied from the camera 200 to recognize a gesture, and executes a command corresponding to the recognized gesture. Further, the CPU 20 sets gesture recognition conditions based on image data supplied to the input buffer 70 and data supplied from an external device (not shown) via the I / O unit 50.

  The ROM 30 stores a program for controlling the operation of the CPU 20.

  The RAM 40 functions as a work area for the CPU 20. Further, the RAM 40 stores various variables used in a threshold setting process (step S50) and a continuous noise removal process (step S60) which will be described later. Further, the RAM 40 stores the image data of the previous frame supplied from the input buffer 70 and the image data of the current frame. The RAM 40 stores image data acquisition time, which will be described later.

  The I / O unit 50 is connected to an external device (not shown) and inputs / outputs data to / from the external device. Specifically, the I / O unit 50 inputs image data to be displayed on the projector 300, data indicating a gesture recognition condition used in gesture recognition processing, and the like from an external device. Further, the I / O unit 50 may output data corresponding to the command corresponding to the recognized gesture to the external device.

  The hard disk 60 stores image data supplied from an external device via the I / O unit 50, data indicating gesture recognition conditions, and the like. Specifically, the hard disk 60 stores image data for a plurality of pages to be displayed on the projector 300, data indicating a recognition condition of a gesture such as a block size and pattern data indicating a gesture pattern to be described later.

  Here, a block is a collection of pixels when pixels arranged two-dimensionally in the vertical and horizontal directions constituting an image for one frame are divided every predetermined number of pixels in the vertical and horizontal directions. . For example, if the vertical direction is divided every 8 pixels and the horizontal direction is divided every 8 pixels, one block is composed of 8 × 8 = 64 pixels. When recognizing a gesture, it is possible to reduce the amount of calculation and the capacity of the RAM to be used by capturing the movement of the image in units of blocks.

  The input buffer 70 is connected to the camera 200 and stores image data supplied from the camera 200. When the input buffer 70 detects that one frame of image data has been supplied from the camera 200, the input buffer 70 transmits an interrupt signal to the CPU 20 via the bus 10.

  The output buffer 80 is connected to the projector 300 and supplies the stored image data to the projector 300. Image data stored in the output buffer 80 is supplied from the hard disk 60 under the control of the CPU 20.

  The timer counter 90 is composed of a circuit having a crystal oscillator, for example. The timer counter 90 includes a register for storing values of the post-detection timer, the standard timer, and the post-stop timer, and counts up the value of each timer. The post-detection timer is a timer for each block indicating an elapsed time from the time when each block is detected as a motion block (that is, a time during which a motion amount equal to or greater than the threshold of each block is continuously detected). The standard timer is a timer indicating an elapsed time from the time when the gesture recognition process is started. The post-stop timer is a timer indicating an elapsed time from the time when no motion block is detected.

  The camera 200 captures an area including the screen 400 and the hand of the person 500 to be captured, and supplies the captured image data to the input buffer 70.

  The projector 300 projects an image corresponding to the image data supplied from the output buffer 80 on the screen 400.

  An image corresponding to the image data supplied from the projector 300 is projected on the screen 400.

  The person 500 to be imaged stands on the edge of the screen 400 and performs a gesture to operate the image projected on the screen 400.

  Next, a functional configuration realized by the above configuration will be described.

  FIG. 2 is a functional block diagram of gesture recognition apparatus 100 according to the embodiment of the present invention.

  The gesture recognition apparatus 100 includes an image input unit 110, an imaging time acquisition unit 120, a recognition condition setting unit 130, a motion amount detection unit 140, a motion block detection unit 150, a specific block detection unit 160, a gesture recognition unit 170, and a command processing unit 180. , And functional blocks of the image output unit 180.

  The image input unit 110 inputs image data supplied from the imaging unit 210.

  The time acquisition unit 120 acquires time. The time acquisition unit 120 supplies data corresponding to the acquired time to the noise removal unit 161 and the gesture recognition unit 170. The time acquisition unit 120 includes an imaging time acquisition unit 121. The imaging time acquisition unit 121 acquires an imaging time that is the time when the image data is supplied to the image input unit 110, and supplies data corresponding to the imaging time to the pattern matching unit 174.

  The recognition condition setting unit 130 sets gesture recognition conditions. The recognition condition setting unit 130 includes a threshold setting unit 131. The threshold setting unit 131 sets a threshold for detecting a motion block based on the image data supplied from the image input unit 110.

  The motion amount detection unit 140 detects the motion amount for each block based on the image data supplied from the image input unit 110.

  The motion block detection unit 150 detects a motion block based on the threshold set by the threshold setting unit 131 and the motion amount for each block detected by the motion amount detection unit 140.

  The specific block detection unit 160 detects a specific block described later from the motion blocks detected by the motion block detection unit 150. The specific block detection unit 160 includes a continuous noise removal unit 161 and a minute noise removal unit 162.

  The continuous noise removal unit 161 excludes motion blocks that have been continuously detected as motion blocks from the motion blocks detected by the motion block detection unit 150 from the specific block candidates. This is because the blocks that are continuously detected as motion blocks are only blocks that are erroneously detected as motion blocks due to noise, and are not attributed to gestures.

  The minute noise removal unit 162 excludes motion blocks outside a predetermined range including a plurality of motion blocks from the motion block detected by the motion block detection unit 150 from the specific block candidates. A block detected as a motion block due to a gesture is detected at a high rate within a predetermined range, and a motion block detected outside the predetermined range is erroneously detected as a motion block due to noise. This is because it is considered to be only a block.

  The gesture recognition unit 170 recognizes a gesture based on the specific block detected by the specific block detection unit 160. The gesture recognition unit 170 includes a center position detection unit 171, a tip position detection unit 171, a locus detection unit 173, and a pattern matching unit 170.

  The center position detection unit 171 detects the center position of the specific block from the specific block detected by the specific block detection unit 160.

  The tip position detection unit 172 detects the tip position of the specific block from the specific block detected by the specific block detection unit 160.

  The locus detection unit 173 detects the locus of the center position detected by the center position detection unit 171 and the locus of the tip position detected by the tip position detection unit 172.

  The pattern matching unit 174 performs matching between the locus of the center position detected by the locus detection unit 173 and the locus of the center position corresponding to each preset gesture pattern. Even when the pattern matching unit 174 matches the same pattern, the pattern matching unit 174 sends different commands depending on the gesture time obtained from the imaging time acquired by the imaging time acquisition unit 120 and the tip position locus detected by the locus detection unit 173. May be issued.

  The command processing unit 180 executes a command corresponding to the gesture pattern recognized by the gesture recognition unit 170.

  The image output unit 190 supplies image data corresponding to the command to the projection unit 310 under the control of the command processing unit 180.

  Next, the operation of the gesture recognition device 100 will be described. Each unit shown in FIG. 1 executes the operation described below, thereby realizing each functional block shown in FIG. In order to facilitate understanding, the function blocks are not referred to one by one.

  When the gesture recognition processing start signal is received from the external device via the I / O unit 50 with the camera 200 and the projector 300 turned on, the CPU 20 executes the gesture recognition processing shown in the flowchart of FIG. To do.

  First, the CPU 20 performs initial setting (step S10). In the initial setting, the CPU 20 initializes various variables stored in the RAM 40. Specifically, the CPU 20 stops and clears all post-detection timers. Further, the CPU 20 clears the standard timer and starts it. In addition, the CPU 20 inputs image data, data indicating gesture recognition conditions, and the like from an external device via the I / O unit 50, and stores the input data in the hard disk 60.

  When the initial setting is completed, the CPU 20 starts image display (step S20). The CPU 20 supplies image data corresponding to the initial image to be displayed on the screen 400 among the image data stored in the hard disk 60 to the output buffer 80. The image data supplied to the output buffer 80 is immediately supplied to the projector 300, and the projector 300 projects an image corresponding to the supplied image data on the screen 400. By these operations, the image output unit 190 and the projection unit 310 are realized.

  Next, the CPU 20 acquires image data for one frame (step S30). The input buffer 70 transmits an interrupt signal to the CPU 20 when image data for one frame is supplied from the camera 200. When receiving the interrupt signal, the CPU 20 transfers the image data for one frame supplied to the input buffer 70 to the RAM 40. Further, the CPU 20 stores the value indicated by the standard timer in the RAM 40 together with data for identifying the image data of the current frame. With these operations, the image input unit 110 is realized.

  When the acquisition of the image data for one frame is completed, the CPU 20 executes a threshold setting process (step S40). In the threshold value setting process, the CPU 20 sets a threshold value for determining whether or not the image of each block has changed from the previous frame, based on the image data of the current frame and the image data of the previous frame. By these operations, the threshold setting unit 131 is realized. Details will be described later.

  When the threshold setting process is completed, the CPU 20 executes a continuous noise removal process (step S50). The CPU 20 detects the amount of motion and removes continuous noise after detecting the motion block. In the continuous noise removal process, the CPU 20 excludes blocks whose images have been continuously changed for a certain time or more from specific block candidates. By these operations, the motion amount detection unit 140, the motion block detection unit 140, and the continuous noise removal unit 161 are realized. Details will be described later.

  When the continuous noise removal process is completed, the CPU 20 executes a minute noise removal process (step S60). In the fine noise removal process, the CPU 20 regards the image that is the target of the gesture as having a certain size or more, and identifies the block in which the image has changed away from other blocks in which the image has changed as a specific block. Exclude from candidates. By these operations, the minute noise removing unit 162 is realized. Details will be described later.

  When the fine noise removal process is completed, the CPU 20 executes a pattern detection process (step S70). In the pattern detection process, the CPU 20 compares the detected image change with each pattern of a preset gesture, and executes a command corresponding to the matched pattern if they match. With these operations, the gesture recognition unit 170 and the command processing unit 180 are realized. Details will be described later.

  The gesture recognition apparatus 100 executes a threshold setting process (step S40), a continuous noise removal process (step S50), a minute noise removal process (step S60), or a pattern detection process (step S70), which will be described in detail later. It becomes possible to improve the recognition accuracy of gestures.

  Next, details of the threshold setting process (step S40) will be described using the flowchart shown in FIG.

  First, the CPU 20 detects a luminance difference for each pixel (step S41). Specifically, the CPU 20 corresponds to the pixel constituting the image corresponding to the image data of the previous frame from the luminance of each pixel constituting the image corresponding to the image data of the current frame stored in the RAM 40. Reduce pixel brightness. When each pixel is composed of three primary colors of RGB, a difference in luminance is obtained for each primary color. The CPU 20 stores the obtained luminance difference in the RAM 40.

  Next, the CPU 20 determines whether or not the detection of the luminance difference has been completed for all the pixels (step S42). If the CPU 20 determines that the detection of the luminance difference has not been completed for all the pixels (step S42: NO), the CPU 20 performs the luminance difference detection for the pixels for which the detection of the luminance difference has not been completed (step S42). S41). On the other hand, if the CPU 20 determines that the detection of the luminance difference has been completed for all the pixels (step S42: YES), the CPU 20 calculates an average value of the squared values of the luminance differences (step S43).

Specifically, in the average value calculation of the square of the luminance difference (step S43), the CPU 20 obtains the luminance difference (ΔB ₁ , ΔB ₂ ,..., ΔB _N ) for all the pixels, and the luminance of each pixel. The value (Σ (ΔB _n ) ² ) obtained by adding the values ((ΔB ₁ ) ² , (ΔB ₂ ) ² ,..., (ΔB _N ) ² ) for all the pixels is divided by the total number of pixels. Thus, an average value (Σ (ΔB _n ) ² / N) of values obtained by squaring the luminance difference is calculated. Note that the luminance of the i-th pixel is B _i , the luminance difference of the i-th pixel is ΔB _i , and the total number of pixels is N. When each pixel is composed of three primary colors of RGB, an average value of values obtained by squaring the luminance difference for each primary color is calculated.

Next, the CPU 20 updates the threshold setting (step S44). Specifically, for example, the CPU 20 sets one block to the average value (Σ (ΔB _n ) ² / N) of the square of the luminance difference calculated in the average square calculation of the luminance difference (step S43). A value (Σ (ΔB _n ) ² / N × M) obtained by multiplying the number of per pixel M is stored in the RAM 40 as a threshold value. CPU20 will complete | finish a threshold value setting process (step S40), if threshold value setting update (step S44) is completed.

  As described above, when the threshold value is set based on the average value of the squares of the luminance differences, the threshold value is set high when the movement of the entire screen is intense, and the threshold value is set low when the movement is small. When such a threshold value is used, it is possible to automatically set an optimum threshold value according to the intensity of movement of the entire screen, thereby reducing false detection.

  Next, the continuous noise removal process (step S50) will be described in detail using the flowchart shown in FIG. The continuous noise removal process is a process in which a block whose image has been changed continuously for a certain time or more is regarded as a block whose image has been changed due to noise, and is excluded from specific block candidates.

First, the CPU 20 detects the movement amount X for each block (step S51). Specifically, for each block, the CPU 20 obtains the total amount (Σ (ΔB _m ) ² ) obtained by squaring the luminance difference of each pixel in the block as the motion amount X.

Next, the CPU 20 determines whether or not the motion amount X is equal to or greater than a threshold value (step S52). Specifically, the CPU 20 uses the threshold (Σ (ΔB _n ) ² / N × M) in which the motion amount X of each block detected in the motion amount detection (step S51) is updated in the threshold setting update (step S44). ) It is determined whether or not it is above.

  If the CPU 20 determines that the motion amount X is not greater than or equal to the threshold (step S52: NO), the CPU 20 regards the target block as a non-motion block and sets it as a non-motion block (step S53). Specifically, the CPU 20 stores the attribute information of the target block in the RAM 40 as “non-motion block”. When step S53 is completed, the CPU 20 stops and clears the post-detection timer (step S54).

  On the other hand, if the CPU 20 determines that the motion amount X is equal to or greater than the threshold (step S52: YES), the CPU 20 regards the target block as a motion block and sets it as a motion block (step S53). Specifically, the CPU 20 stores the attribute information of the target block in the RAM 40 as “motion block”.

  The CPU 20 determines whether or not the post-detection timer is 2 seconds or more (step S56), and if it is determined that the post-detection timer is 2 seconds or more (step S56: YES), the target block is not employed for gesture recognition. A block is set (step S57). That is, even if the motion amount X is equal to or greater than the threshold value, a block in which a motion amount equal to or greater than the threshold value is continuously detected for 2 seconds or more is treated as a block that is erroneously detected as a motion block due to noise. Specifically, the CPU 20 rewrites the attribute information of the target block stored in the RAM 40 to “a block not adopted for gesture recognition”.

  When the CPU 20 determines that the timer after detection is not 2 seconds or more (step S56: NO), or after the setting to the non-motion block (step S57) is completed, the CPU 20 determines whether or not the timer is stopped. A determination is made (step S58). When determining that the post-detection timer is stopped (step S58: YES), the CPU 20 starts the post-detection timer (step S59).

  When the CPU 20 determines that the post-detection timer is not stopped (step S58: NO), the post-detection timer is started (step S59), or the post-detection timer is stopped and cleared (step S54). After that, it is determined whether or not the processing for all the blocks has been completed (step S510).

  If the CPU 20 determines that the settings for all blocks have been completed (step S510: YES), the CPU 20 ends the continuous noise removal process. On the other hand, if the CPU 20 determines that the setting for all blocks has not been completed (step S510: NO), the CPU 20 detects the motion amount of the block for which setting has not been completed (step S51).

  As described above, in the continuous noise removal process, it is set which block corresponds to each block among a motion block, a non-motion block, or a block that is not used for gesture recognition.

  Usually, it is rare to try to recognize the movement itself projected on a television or the like as a gesture. In a conventional gesture recognition device, when a gesture is to be recognized from image data obtained by imaging an area including a television or the like and a person performing the gesture, the movement projected on the television or the like is erroneously recognized as a gesture. There is a possibility that.

  However, in gesture recognition apparatus 100 according to the embodiment of the present invention, blocks detected as motion blocks continuously for a certain time or longer are excluded from blocks used for gesture recognition. For this reason, erroneous detection of gestures can be reduced.

In gesture recognition apparatus 100 according to the embodiment of the present invention, motion amount X of each block is the sum of values obtained by squaring the luminance difference of each pixel (Σ (ΔB _m ) ² ). Therefore, as in the apparatus described in Patent Document 1, the motion amount X of each block is more agile than the case where the sum of the luminance differences of each pixel (Σ | ΔB _m |) is used. It is possible to improve gesture recognition accuracy. The reason will be described below.

  When an image changes due to a gesture, a small number of pixels with a very large difference in luminance are expected to occur. On the other hand, when the image changes due to a change in the amount of light of the lighting device or a motion such as blurring of the imaging device, it is expected that a large number of pixels with a relatively small difference in luminance will occur.

  Here, when the sum of values obtained by squaring the luminance difference of each pixel in the block is used as the amount of motion, the amount of motion becomes a relatively large value if there is any pixel with a large luminance difference, and the luminance difference is Even if there are many small pixels, the amount of motion is a relatively small value.

  For this reason, it is possible to detect the change in the image caused by the gesture as the main component of the motion amount by using the sum of the squares of the luminance differences of the pixels in the block as the motion amount. Improvement in recognition accuracy can be expected.

  Next, the minute noise removal process (step S60) will be described in detail with reference to FIGS. FIG. 6 is a flowchart showing a minute noise removal process, and FIG. 7 is a diagram for explaining the minute noise removal process. In the fine noise removal process, the image subject to gesture is considered to have a certain size or more, and the image changes due to noise in the block where the image changes away from other blocks where the image changes. This is a process of considering a block being excluded and excluding it from a specific block candidate.

  FIG. 7 shows an image of 200 blocks (20 (X-axis direction) × 10 (Y-axis direction)) with the horizontal axis as the X-axis and the vertical axis as the Y-axis. The image shown in FIG. 7 is an image obtained by imaging the area including the image of Mt. Fuji, clouds, and an airplane projected from the projector 300 onto the screen 400 and the hand of the person 500 to be imaged from the right side of the screen 400. is there.

  Also, the shaded blocks shown in FIG. 7 are motion blocks set as motion blocks. When there is almost no change in the image projected from the projector 300, most of the blocks detected as motion blocks are blocks corresponding to the hand portion of the person 500 to be imaged.

  First, the CPU 20 counts the number of motion blocks for each column from the block attribute information stored in the RAM 40, and obtains the number of motion blocks in each column (step S61).

  Similarly, the CPU 20 counts the number of motion blocks for each row, and obtains the number of motion blocks for each row (step S62).

  Next, the CPU 20 sets a range of columns having a large number of motion blocks based on the number of motion blocks of each column obtained in step S61 (step S63).

  Various methods are conceivable for setting the range of a column having a large number of motion blocks. For example, a range of columns in which two or more columns each having one or more motion blocks are consecutive is set. In FIG. 7, since the number of motion blocks is continuously 1 or more in the range from the 12th column to the 20th column, the CPU 20 sets the range from the 12th column to the 20th column as the range of columns having a large number of motion blocks. To do.

  Subsequently, the CPU 20 sets a range of rows having a large number of motion blocks based on the number of motion blocks of each row obtained in step S62 (step S64). The CPU 20 sets the range of the sixth row to the eighth row as a range of rows with a large number of motion blocks based on the same criteria as in step S63.

  Next, the CPU 20 sets a rectangular range with a large number of motion blocks (step S65). The CPU 20 sets the range belonging to both the column range having a large number of motion blocks set in step S63 and the row range having a large number of motion blocks set in step S64 as a rectangular range having a large number of motion blocks. To do. In FIG. 7, a range surrounded by a thick line is a rectangular range having a large number of motion blocks.

  When completing the setting of the rectangular range having a large number of motion blocks (step S65), the CPU 20 determines whether or not each block is within the set rectangular range (step S66). When determining that each block is within the set rectangular range (step S66: YES), the CPU 20 determines whether or not the determination for all the blocks has been completed (step S68).

  On the other hand, if the CPU 20 determines that each block is not within the set rectangular range (step S66: NO), the CPU 20 sets the target block as a block that is not adopted for gesture recognition (step S67). Specifically, the CPU 20 rewrites the attribute information of the target block stored in the RAM 40 to “a block not adopted for gesture recognition”. When completing step S67, the CPU 20 determines whether or not the determination for all blocks is completed (step S68).

  If the CPU 20 determines that the determination for all blocks has been completed (step S68: YES), it ends the minute noise removal process. On the other hand, if the CPU 20 determines that the determination for all blocks has not been completed (step S68: NO), the CPU 20 determines whether or not the block for which determination has not been completed is within the rectangular range (step S66).

  In gesture recognition apparatus 100 according to the embodiment of the present invention, blocks that are not within a rectangular range with a large number of motion blocks are excluded from the blocks used for gesture recognition. For this reason, it is possible to exclude a region that is unrelated to the gesture and has a large change in luminance due to noise from the target of gesture recognition, thereby reducing erroneous detection of the gesture.

  Next, the pattern detection process (step S70) will be described in detail using the flowchart shown in FIG.

  First, the CPU 20 determines whether or not there is a motion block (step S71). When determining that any of the blocks is a motion block (step S71: YES), the CPU 20 performs center position detection (step S72). On the other hand, when determining that all the blocks are not motion blocks (step S71: NO), the CPU 20 determines whether the post-stop timer is 1 second or longer (step S75).

  In the center position detection (step S72), the CPU 20 detects the center position of the motion block. The method for detecting the center position of the motion block will be described in detail with reference to FIG. 9A.

  FIG. 9A shows an image of 80 blocks (10 (X axis direction) × 8 (Y axis direction)) with the horizontal axis as the X axis and the vertical axis as the Y axis. The shaded block shown in FIG. 9A is a motion block.

  In the image shown in FIG. 9A, there are nine motion blocks. The CPU 20 obtains the X coordinate of the center position by dividing the sum of the X coordinates of the nine blocks by 9, and obtains the Y coordinate of the center position by dividing the sum of the Y coordinates of the nine blocks by 9.

  However, in order to facilitate understanding, a case where the coordinates of the block closest to the coordinates of the center position are approximated as the coordinates of the center position will be described below as an example. In FIG. 9A, the coordinates of the block (X coordinate = 9, Y coordinate = 7) marked with a bold line “◯” are the coordinates of the center position. The CPU 20 stores the obtained coordinates of the center position in the RAM 40.

  CPU20 will perform front-end | tip position detection (step S73), if center position detection (step S72) is completed. In the tip position detection, the CPU 20 obtains the direction of the gesture movement from the center position obtained in the center position detection (step S72), and further, the motion block existing at the most distant position in the direction of the gesture movement based on the center position. Is detected as the tip position.

  A method for detecting the tip position of the motion block will be described in detail with reference to FIG. 9B. Note that the image shown in FIG. 9B is an image of the next frame of the image shown in FIG. 9A.

  In the image shown in FIG. 9B, there are 19 motion blocks. When the coordinates of the center position are obtained by the method described above, the coordinates of the block (X coordinate = 7, Y coordinate = 6) marked with a bold line “◯” become the coordinates of the center position. In FIG. 9B, the coordinates (X coordinate = 9, Y coordinate = 7) of the block marked with a broken line “◯” are the coordinates of the center position one frame before.

  Here, the direction indicated by the difference vector formed when connecting the coordinates of the center position of the previous frame to the coordinates of the center position of the main frame is considered as the direction in which the object of the gesture proceeds. Then, the coordinates of the motion block on the extension line of the difference vector and located farthest from the center position are set as the coordinates of the tip position.

  In FIG. 9B, the coordinates of the block (X coordinate = 3, Y coordinate = 4) marked with a thick line “Δ” are the coordinates of the tip position. The CPU 20 stores the obtained coordinates of the tip position in the RAM 40.

  When the tip position detection (step S73) is completed, the CPU 20 restarts the timer after stopping (step S74). Specifically, the CPU 20 clears the timer value after stopping and starts the timer after stopping.

  CPU 20 completes timer restart after stop (step S74) or is there a motion block? If it is determined as NO in (Step S71), it is determined whether or not the value of the post-stop timer is 1 second or more (Step S75). When the CPU 20 determines that the value of the timer after the stop is 1 second or more (step S75: YES), the CPU 20 performs locus detection (step S76). On the other hand, when the CPU 20 determines that the value of the timer after stop is not 1 second or more (step S75: NO), the pattern detection process (step S70) is terminated. Step S75 is a process for preventing the command from being issued before 1 second has elapsed since the motion block is no longer detected.

  In the locus detection (step S76), the CPU 20 detects the locus of the center position based on the coordinates of the center position stored in the RAM 40 in the center position detection (step S72). Further, the CPU 20 detects the locus of the tip position based on the coordinates of the tip position stored in the RAM 40 in the tip position detection (step S73).

  When the locus detection (step S76) is completed, the CPU 20 determines whether or not the gesture pattern matches a predetermined pattern (step S77). The CPU 20 performs pattern matching based on the locus of the center position detected in the locus detection (step S76), the locus of the tip position, and the data indicating the gesture pattern stored in the hard disk 60 in advance.

  FIG. 10 shows an example of a gesture pattern. “Pattern” indicates a serial number of a gesture pattern. “The locus of the center position” indicates the locus of the center position obtained in the locus detection (step S76). “Movement distance” indicates the movement distance of the center position. “Movement time” is the time from when YES is determined to when NO is determined in the presence of a motion block (step S71), that is, the motion as a gesture candidate after the motion as a gesture candidate is recognized. Indicates the time until disappears.

  “Command display” indicates the contents of command processing executed when a pattern is matched, that is, when all of the above-mentioned conditions of “center position locus”, “movement distance”, and “movement time” are satisfied. .

  If the CPU 20 determines that the detected gesture pattern matches any of the patterns indicating the patterns stored in the hard disk 60 (step S77: YES), the CPU 20 performs command processing (step S78). On the other hand, if the CPU 20 determines that the detected gesture pattern does not match any pattern (step S77: NO), the CPU 20 performs pattern detection initialization (step S79).

  If the CPU 20 determines that the command processing (step S78) is completed or that the detected gesture pattern does not match any pattern (step S77: NO), the CPU 20 performs pattern detection initialization (step S79). In pattern detection initialization (step S79), the CPU 20 initializes various variables used in the pattern detection process (step S70). Specifically, the history of the center position and the tip position stored in the RAM 40 is deleted.

  After completing the pattern detection initialization (step S79), the CPU 20 stops and clears the timer after the stop (step S710). When the CPU 20 completes stopping and clearing the timer after stopping (step S710), the CPU 20 ends the pattern detection process (stop S70), and returns the process to frame image acquisition (step S30).

  Here, an example of pattern matching (step S77) and command processing (step S78) will be described in detail with reference to FIGS. 11A to 11F and FIGS. 12A to 12D.

  First, pattern matching and command processing when the “pattern” of the gesture is “1” will be described.

  FIG. 11A is a diagram illustrating an image immediately after the center position of the motion block is detected in the lower right specific region. In FIGS. 11A to 11F, the 3 × 3 block area surrounded by the bold lines shown at the four corners is a specific area. Also, the coordinates of the block with the bold line “◯” are the coordinates of the center position of the motion block. As shown in FIG. 11A, when the hand of the person to be imaged 500 enters the lower right region within the angle of view from the outside of the angle of view, the center position of the motion block is detected in the lower right specific region.

  FIG. 11B is a diagram illustrating an image when the center position of the motion block is detected in the lower right specific region. As shown in FIG. 11B, while the hand of the person 500 to be imaged is moving in the lower right area within the angle of view, the center position of the motion block is continuously detected in the lower right specific area.

  FIG. 11C is a diagram illustrating an image immediately after the center position of the motion block is no longer detected from within the lower right specific region. In FIG. 11C, the coordinates of the block with the broken line “◯” are the coordinates of the center position detected last. As illustrated in FIG. 11C, when the hand of the person to be imaged 500 moves out of the angle of view from the lower right region within the angle of view, the center position of the motion block is not detected from within the lower right specific region.

  When one second elapses after the center position of the motion block is no longer detected and the post-stop timer reaches 1 second or longer, the CPU 20 performs pattern matching after locus detection (step S76) (step S77). As described above, the trajectory of the center position indicates a trajectory that deviates from the angle of view after entering the specific area at the lower right of the angle of view from the outside of the angle of view. Fulfill.

  However, the content of the command executed by the CPU 20 differs depending on the “movement time”, here, the time at which the center position of the motion block stays in the specific area at the lower right of the angle of view. Note that the CPU 20 obtains the movement time based on the acquisition time of the image data stored in the RAM 40 in the frame image acquisition (step S30).

  When the “movement time” is 2 seconds or more, the CPU 20 does not switch the image. Specifically, as illustrated in FIG. 11D, the CPU 20 continues to project the image of the page that has been displayed so far onto the screen 400.

  When the “movement time” is 1 second or more and less than 2 seconds, the CPU 20 displays the image of the next page. Specifically, the CPU 20 transfers the image data of the image of the next page stored in the hard disk 60 to the output buffer 80, whereby the image projected on the screen 400 by the projector 300 is as shown in FIG. 11E. Switch to the next page image.

  When the “movement time” is less than 1 second, the CPU 20 displays the image of the page one after another. Specifically, the CPU 20 transfers the image data of the image of the next page different from the image of the next page from the hard disk 60 to the output buffer 80, whereby the image projected on the screen 400 by the projector 300 is shown in FIG. 11F. Switch to the next page image as shown.

  As described above, in gesture recognition apparatus 100 according to the embodiment of the present invention, CPU 20 stores the time at which image data is acquired in RAM 40 using the standard timer that timer counter 90 counts up. For this reason, the speed of the gesture can be accurately measured. Therefore, even when the locus of the center position and the movement distance are the same, different commands can be prepared according to the gesture speed.

  Further, by detecting the center position of the gesture, it is possible to easily grasp the gesture operation.

  Next, pattern matching and command processing when the gesture pattern is “pattern 5” will be described.

  FIG. 12A is a diagram illustrating an image immediately after the center position of the motion block is detected. In FIG. 12A and FIG. 12B, the coordinates of the block marked with a bold line “◯” are the coordinates of the center position of the motion block. As shown in FIG. 12A, when the object of the gesture enters the angle of view from outside the angle of view, the center position of the motion block is detected.

  FIG. 12B is a diagram illustrating an image when the tip position of the motion block is on the leftmost side. In FIG. 12B, the coordinates of the block marked with a thick line “Δ” are the coordinates of the tip position of the motion block. As shown in FIG. 12B, when the hand of the person to be imaged 500 moves to the leftmost position within the angle of view, the tip position of the motion block is on the leftmost side.

  FIG. 12C is a diagram illustrating an image immediately after the center position of the motion block is no longer detected. In FIG. 12C, the coordinates of the block with the broken line “◯” are the coordinates of the center position that was detected last, and the coordinates of the block with the broken line “Δ” are the leftmost position. The coordinates of the tip position (hereinafter referred to as the peak coordinates of the tip position). Note that, as shown in FIG. 12C, when the hand of the person being imaged 500 remains within the angle of view, when the hand is stopped, the motion block is the same as when the hand is out of the angle of view. No center position is detected.

  When one second elapses after the center position of the motion block is no longer detected and the post-stop timer reaches 1 second or longer, the CPU 20 performs pattern matching after locus detection (step S76) (step S77). As described above, “the locus of the center position” is right → left → right, and “the movement distance of the center position” is 1/20 or more of the angle of view, so the condition of the pattern 5 is satisfied.

  Here, when the “movement time” is less than 3 seconds, the CPU 20 enlarges and displays the currently displayed image. Specifically, the CPU 20 creates image data corresponding to an image enlarged around the peak coordinates of the tip position based on the image data corresponding to the currently displayed image stored in the hard disk 60. To do. Then, the CPU 20 transfers the created image data to the output buffer 80, thereby switching the image projected on the screen 400 by the projector 300 to an enlarged image as shown in FIG. 12D.

  On the other hand, when the “movement time” is 3 seconds or more, the CPU 20 does not switch the image.

  As described above, the gesture recognition apparatus 100 according to the embodiment of the present invention can expand the degree of freedom of the command pattern by detecting the tip position of the gesture.

  In addition, this invention is not limited to the said Example, A various deformation | transformation and application are possible.

  In the above embodiment, the number M of pixels per block is 8 pixels in the vertical direction × 8 pixels in the horizontal direction = 64, but the size of the block is arbitrary. For example, if it is desired to detect a smaller motion, the block size is reduced (for example, 4 pixels in the vertical direction × 4 pixels in the vertical direction), and if it is desired to detect a faster motion, the block size is increased (for example, the vertical size). 16 pixels in the direction × 16 pixels in the vertical direction).

In the above-described embodiment, the threshold value of the motion amount X is set to the average value (Σ (ΔB _n ) ² / N of the square value of the luminance difference calculated in the average value calculation of the square of the luminance difference (step S43). ) Multiplied by the number of pixels M per block (Σ (ΔB _n ) ² / N × M). However, the threshold value of the motion amount X is a value obtained by further multiplying the above threshold value by a constant K (Σ (ΔB _n ) ² / N × M × K), or an average value of absolute values of luminance differences. Other values obtained in accordance with the amount of change in luminance of the entire screen, such as a value obtained by multiplying the number M (Σ | ΔB _n | / N × M), may be used.

  In the above embodiment, the threshold value of the motion amount X is obtained by calculation. However, an appropriate threshold value may be selected according to the amount of change in luminance of the entire screen. In this case, a plurality of threshold values may be stored in advance in the hard disk 60 or the like as candidate values.

  In the above-described embodiment, the threshold value of the motion amount X is obtained according to the amount of change in the brightness of the entire screen. However, when the intensity of motion of the entire screen can be predicted, the threshold value may be a fixed value. . In this case, the threshold value may be stored in advance in the hard disk 60 or the like.

  Moreover, in the said embodiment, although only one rectangular range with many motion blocks was set, you may set two or more rectangular ranges. In addition, the shape set as the range having a large number of motion blocks is not limited to a rectangle, and may be an arbitrary shape such as a circle, an ellipse, or a polygon other than a rectangle according to the shape of the gesture target.

  In the above-described embodiment, an example in which all frame images supplied from the camera 200 are processed has been described. However, when the frame rate at the time of imaging is higher than the processing speed of the gesture recognition device, the thinned frame image may be processed.

  In the present invention, the above-described gesture patterns and commands are not limited to the above examples, and can be arbitrarily set.

  Furthermore, it goes without saying that the present invention is not limited to the procedures shown in the configuration examples and flowcharts described above.

1 is a configuration diagram of a system to which a gesture recognition device according to an embodiment of the present invention is applied. It is a functional block diagram of the gesture recognition apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the gesture recognition process of the gesture recognition apparatus shown in FIG. It is a flowchart which shows an example of the threshold value setting process shown in the flowchart of FIG. It is a flowchart which shows an example of the continuous noise removal process shown in the flowchart of FIG. It is a flowchart which shows an example of the minute noise removal process shown in the flowchart of FIG. It is a figure for demonstrating the removal process of a minute noise. It is a flowchart which shows an example of the pattern detection process shown in the flowchart of FIG. It is a figure for demonstrating the detection method of the center position of a motion block. It is a figure for demonstrating the detection method of the front-end | tip position of a motion block. It is a figure which shows an example of the pattern of gesture. It is a figure for demonstrating a matching process with the pattern 1. FIG. It is a figure for demonstrating a matching process with the pattern 1. FIG. It is a figure for demonstrating a matching process with the pattern 1. FIG. FIG. 10 is a diagram for explaining command processing of pattern 1; FIG. 10 is a diagram for explaining command processing of pattern 1; FIG. 10 is a diagram for explaining command processing of pattern 1; 10 is a diagram for explaining a matching process with a pattern 5. FIG. 10 is a diagram for explaining a matching process with a pattern 5. FIG. 10 is a diagram for explaining a matching process with a pattern 5. FIG. 10 is a diagram for explaining command processing of pattern 5. FIG.

Explanation of symbols

10 Bus 20 CPU (Central Processing Unit)
30 ROM (Read Only Memory)
40 RAM (Random Access Memory)
50 I / O (Input Output) unit 60 Hard disk 70 Input buffer 80 Output buffer 90 Timer counter 100 Gesture recognition device 110 Image input unit 120 Time acquisition unit 121 Imaging time acquisition unit 130 Recognition condition setting unit 131 Threshold setting unit 140 Motion amount detection Unit 150 motion block detection unit 160 specific block detection unit 161 continuous noise removal unit 162 minute noise removal unit 170 gesture recognition unit 171 center position detection unit 172 tip position detection unit 173 locus detection unit 174 pattern matching unit 180 command processing unit 190 image output Section 200 Camera 210 Imaging section 300 Projector 310 Projection section 400 Screen 500 Person to be imaged

Claims (10)

In a gesture recognition device for recognizing a gesture based on an action of an imaged object,
Recognition condition setting means for setting gesture recognition conditions;
A motion amount detection means for detecting a motion amount based on the amount of change in luminance of each pixel in the divided region for each divided region obtained by dividing the captured image into a plurality of regions;
A motion region detection unit that detects a segmented region in which a motion amount detected by the motion amount detection unit is equal to or greater than a predetermined threshold as a motion region;
Specific area determination means for determining whether or not the movement area detected by the movement area detection means is a specific area to be adopted for gesture recognition;
Gesture recognition means for recognizing a gesture based on at least a motion area determined to be a specific area by the specific area determination means and a gesture recognition condition set by the recognition condition setting means;
A gesture recognition device characterized by comprising: The recognition condition setting means includes threshold setting means for setting the predetermined threshold,
The motion region detection means detects a segmented region in which the motion amount detected by the motion amount detection means is equal to or greater than a threshold set by the threshold setting means as a motion region;
The gesture recognition apparatus according to claim 1. The threshold setting means sets the threshold based on the amount of change in luminance of each pixel of the sequentially captured images;
The gesture recognition apparatus according to claim 2. The motion amount detection means detects a sum of values obtained by squaring the amount of change in luminance of each pixel in the divided region as a motion amount.
The gesture recognition apparatus according to claim 1. The specific area determination means does not determine that the movement area detected by the movement area detection means continuously for a predetermined time or more is the specific area;
The gesture recognition apparatus according to claim 1. The specific area determination unit is included in a predetermined area including the plurality of movement areas among the movement areas detected by the movement area detection unit when a plurality of movement areas are detected by the movement area detection unit. A movement area to be determined is the specific area,
The gesture recognition apparatus according to claim 1. In the case where there are a plurality of motion areas determined to be specific areas by the specific area determination means, the movement of the gesture in the image is determined from the positions of the plurality of motion areas determined to be the specific areas in the image. A center position detecting means for detecting a center position;
The gesture recognition means recognizes a gesture based on a center position detected by the center position detection means;
The gesture recognition apparatus according to claim 1. Movement direction determination means for determining the direction of movement of a gesture from a plurality of captured images;
In a case where there are a plurality of motion areas determined to be the specific area by the specific area determination unit, the motion detected by the center position detection unit among the plurality of motion areas determined to be the specific area. Tip position detecting means for detecting a movement region existing at the most distant position in the direction of movement of the gesture determined by the movement direction determining means with reference to the center position; and
The gesture recognizing means recognizes a gesture based on the position in the image of the motion region existing at the most distant position detected by the tip position detecting means;
The gesture recognition apparatus according to claim 7. Imaging time acquisition means for acquiring the imaging time of images taken sequentially, further comprising:
The gesture recognizing unit obtains the time when the specific region determining unit continuously determines that the motion region is a specific region based on the image capturing time of the image acquired by the imaging time acquisition unit, Recognized as a gesture according to the given time,
The gesture recognition apparatus according to claim 1. In a gesture recognition method for recognizing a gesture based on an action of an imaged object,
A recognition condition setting step for setting a recognition condition for the gesture;
A motion amount detection step for detecting a motion amount based on the amount of change in luminance of each pixel in the divided region for each divided region obtained by dividing the captured image into a plurality of regions;
A motion region detection step of detecting, as a motion region, a divided region in which the motion amount detected in the motion amount detection step is equal to or greater than a predetermined threshold;
A specific region determination step for determining whether or not the motion region detected in the motion region detection step is a specific region to be adopted for gesture recognition;
A gesture recognition step for recognizing a gesture based on at least a motion region determined to be a specific region in the specific region determination step and a gesture recognition condition set in the recognition condition setting step;
A gesture recognition method characterized by comprising:

Images