JP2001273503A - Motion recognition system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system which uses distance information and the feature quantity, which are operated from plural pictures obtained by picking up images by plural cameras, to recognize the motion of an object in a non-contact nanner independently of the background. SOLUTION: The system is comprised of a range finder part 101, an image pickup part 102, an information volume calculation part 103, a state recognition part 104, a pattern detection part 105, an inference part 106, a meaning recognition part 107.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for recognizing the motion of an object, and more particularly to a system for recognizing the motion of an object using a principle of triangulation from a plurality of images obtained by imaging the motion of the object by a plurality of cameras. The present invention relates to a motion recognition system provided with distance measuring means for calculating distance information.

[0002]

2. Description of the Related Art A hand gesture recognition system will be described as an example of a conventional motion recognition system. IEICE Transactions, D-II, Vol.J73-
D-II, No. 12, PP-1985-1992 (199
0), a hand gesture recognition system using a hand shape input device (using a commercially available Data Glove ™) is described. Data Grab TM
Is a product of VPL, USA, which measures the bending information of a finger based on the amount of light in an optical fiber, and calculates the second and third joints of the thumb and the first and second fingers of the other finger.
The bending angles of the two joints, which are corrected for individual differences, are measured. By combining a three-dimensional digitizer (3 SPACE TRACKER ™) manufactured by MacDonnell Douglas in the United States using a magnetic generator and a magnetic sensor, the direction of the hand and the three-dimensional position of the hand can also be measured. However, such a conventional system requires an input device in contact with an object for motion recognition, and is a very complicated and large-scale system that restricts the movement of the object making a motion. .

[0003] Alternatively, a person who enjoys a game actually moves his / her body, recognizes the motion of this person's body or a human body part, and controls a computer connected to a television or the like according to the recognition result, and as a whole, It is contemplated to configure the device. For example, two players face each other at a specified position, punch or kick, recognize the motion of these players, give the recognition result to the computer, and display the computer-generated game character on the monitor Then, on the monitor, the game character moves in the same manner as the players, and the scores of the two players are totaled and displayed.
In this type of computer game device, it is necessary to recognize the motion of a human body. 2. Description of the Related Art Conventionally, a whole or a part of a human body is imaged by a CCD (solid-state imaging device) camera, and the motion of the human body or a part of the human body is detected based on an image signal (electric signal) output by the CCD camera. There has been proposed a motion detection device configured to perform the motion detection. For example, "Artificial Retina C
hips as Image Input Interfaces for Multimedia Syst
ems "(First Optoelectronics and Communications Conf
ernce (OECC'96) Technical Digest, July 1996, Makuhari
Messe p516-p517) discloses a technique for extracting image outlines (edges) of an object in an image by processing image data captured by a CCD. The movement of the human body can be determined from the edge. Then, by inputting the detected movement of the human body to the computer game instead of the input by the controller, for example, a fighting game can be configured. However,
In such a conventional motion recognition device, when the image behind the human body is not uniform, a part of the background is recognized as a part of the human body, or when the brightness (luminance) of the human body is close to the background, Has the disadvantage that the human body may not be distinguishable from the background.

[0004]

SUMMARY OF THE INVENTION The present invention provides distance information calculated from a plurality of images captured by a plurality of cameras, and a non-contact object that performs motion using various features. An object of the present invention is to provide a system capable of recognizing motion regardless of background.

[0005]

In order to solve the above-mentioned problems, a motion recognition system according to the present invention comprises: 1) an image pickup means for picking up an image of an object; and 2) an image pickup of the object by a plurality of cameras. Distance measuring means for calculating distance information of an imaging area from a plurality of images obtained by using the principle of triangulation; and 3) the state of the object from the amount of information calculated from the image of the object and the distance information. 4) a pattern detecting means for detecting a pattern relating to movement from the recognized state, and 5) a semantic recognizing means for recognizing predetermined implications from the state and the pattern. It was adopted.

Alternatively, as in the second aspect of the present invention, 1) imaging means for imaging an image of an object; and 2) triangulation from a plurality of images obtained by imaging the object with a plurality of cameras. Distance measuring means for calculating distance information of the imaging area using the principle of 3); 3) state recognizing means for recognizing a state of the object from an image of the object and an information amount calculated from the distance information; 4) pattern detection means for detecting a pattern relating to movement from the recognized state; 5) a table storing a predetermined state, a basic state composed of the predetermined pattern, and an identifier corresponding to each of the basic states; 6) a dictionary storing the implication corresponding to the identifier; and 7) a meaning recognizing means for recognizing the implication by comparing the recognized state and the detected pattern with the dictionary. A system to be formed may be adopted.

[0007] If the semantic recognition means refers to information (estimated value) on the state of the object estimated using past information as necessary and recognizes the implication, the object can be recognized by other objects. Even if the current information is missing due to crossing with the object, it can be recognized.

The amount of information calculated from the distance information is:
At least one selected from a spatial position, a protrusion, a volume, a three-dimensional form, and a feature value is effective in recognizing a state.

Here, the state is desirably at least one selected from a part specified in advance, an object specified in advance, and an evaluation in comparison with a reference value.

Patterns relating to movement include the speed of a specific part or target, the acceleration of a specific part or target, the trajectory of a specific part or target, the stroke of a specific part or target, and the positional relationship between a plurality of specific parts or targets. And at least one selected from dynamic deformation of the feature points.

[0011] The implications are commands for instructing the operation of a machine or a system, movement of a robot or animation in a screen, instructions for screen display, generation of an object,
Control of object functions, instructions for editing documents, instructions for creating graphics, instructions for controlling voice or musical sounds,
It is useful that at least one is selected from interpretation of sign language and an evaluation value prepared in advance.

In the second aspect of the present invention, it is further useful to provide a system capable of changing a function of motion recognition by exchanging the basic state corresponding to an identifier or the implication corresponding to the identifier.

[0013] The recognized implication may be expressed by at least one selected from screen display, voice, and machine operation.

[0014] The object to be imaged is a person who is playing a sport, the movement pattern is a form of action characteristic of the sport, and the implication is an evaluation or explanation of the form. Good.

It is also effective to apply the present invention to control for controlling at least one selected from an animation and a robot in accordance with the recognized implication.

[0016] The system may further comprise sound generating means for generating a sound signal, and the sound signal is controlled based on the recognized implication.

The object to be imaged is a human, the movement pattern is a characteristic characteristic of a specific physiological condition, and the implication is an evaluation relating to a degree of the physiological condition. The system is also effective. here,
The specific physiological state is a dozing driving state of a person driving the machine, and the implication may be an evaluation regarding the degree of dozing.

[0018] The implication may include a start and an end, and the system may be characterized in that the illustration is generated based on a pattern related to a motion detected between the start motion and the end motion. Furthermore, it is more preferable to include the deformation and movement of the already generated illustration, and to deform or move the illustration according to the recognized implications.

[0019]

With the above-described system, it is possible to recognize a continuously changing movement of an object in a non-contact manner regardless of the background. Further, it is possible to activate a predetermined function according to the recognized implication.

[0020]

DETAILED DESCRIPTION OF THE INVENTION

Embodiment 1 FIG. 1 is a block diagram showing an embodiment of a motion recognition system according to the present invention. Distance measuring unit 101,
Imaging unit 102, information amount calculation unit 103, state recognition unit 10
4. The configuration includes a pattern detection unit 105, an estimation unit 106, and a meaning recognition unit 107.

A brief description of each component will be given. Distance measuring unit 1
In step 01, an object is imaged by a plurality of cameras, distance information for each pixel in the shooting area is calculated using the principle of triangulation, and sent to the information amount calculation unit 103. The imaging unit 102
An image of the object is acquired by an imaging camera and sent to the information amount calculation unit 103. The ranging camera has a configuration in which nine cameras 201 to 209 are laid out on a base plate 210 as shown in FIG. Here, the central camera 205 also functions as an imaging camera.

The information amount calculation unit 103 calculates the information amount from the image of the object accompanied by the distance information, that is, the distance image according to a predetermined calculation program. The amount of information to be calculated can be obtained using the captured image and distance information such as the direction of the object, the spatial position of the object, the protrusion, the volume, the three-dimensional shape, and the extraction of a three-dimensional structure with a specific pattern. Various information is listed. For example, if the amount of information is a protrusion,
The distance information of the distance images is compared, and a portion having a short distance is cut out. The calculated information amount is sent to the state recognition unit.

The state recognizing unit 104 recognizes a state based on the characteristic amount calculated by the information amount calculating unit. For example, the part “hand” is recognized as the state of the screen. The recognized state is the pattern detection unit 105 and the estimation unit 1
06 and the meaning recognition unit 107.

The pattern detecting section 105 tracks the movement or change of an area indicating a specific state and detects a pattern related to the movement. For example, the movement of the "hand" can be known by tracking the spatial movement of the state of the "hand" described above. Further, when the movement of the hand is repeated at a constant cycle, the repetition is understood and detected as, for example, a stroke or a trajectory. The detected pattern is sent to the meaning recognition unit 107.

The estimating section 106 stores the states from the state recognizing section 104 in a time-series manner, and estimates the current state. For example, if the state of "hand" is stored in the time series in the past, it is estimated that the state of "hand" exists in that area even if the state of "hand" cannot be recognized because it is mixed with other things. I do. The estimated value (that is, the estimated current state) is sent to the meaning recognition unit 107.

In the meaning recognition unit 107, the state, the pattern,
A predetermined implication is recognized with reference to the estimated value. For example, in a fighting game that moves a character on the screen as a result of motion recognition, the implication of a character's “straight punch” is recognized from a “track” protruding “forward” of a “hand”.

[0027]

Embodiment 2 FIG. 3 is a schematic configuration diagram of a game machine using a motion recognition system. Cameras 301 to 309 for distance measurement and a game screen 310 are arranged on the game machine main body 300. The camera 303 also functions as an imaging camera. When the game user stands in front of the game machine and takes a specific action, the character 311 in the game screen is displayed.
Makes a movement corresponding to the user's action.

FIG. 4 is a flowchart relating to the character control of the game. In the fighting game, an example will be described in which the character makes a punch based on the movement of the user's hand. In the information amount calculation step 401, the position coordinates, the volume, and the three-dimensional shape of the protrusion are calculated from the distance image of the game user acquired by the camera group. In the state recognition step 402, the table 40 prepared in advance is used.
The position of the hand and the state of the hand are recognized with reference to the volume of the state “hand” stored in 4 and the basic values related to the three-dimensional shape. In the pattern detection step 403, the table 4
With reference to the reference values relating to the speed, acceleration, and trajectory of “punch” stored in 04, a movement pattern relating to “punch” is detected.

In the action recognition step 405, referring to the dictionary 406, the implication of the motion detected in the previous steps is recognized. Here, the implication is the action of the character, such as punching and kicking, and sends an instruction related to the action of the character. In the character control step 407, the character on the screen is moved according to the command related to the action.

Here, the recognized state and pattern only control the movement of the character on the screen, but generate a sound effect such as a sound of cutting off the wind in accordance with the speed or acceleration of the hand movement. May be. For example, if a sharp wind noise is made to correspond to a fast movement, the sense of reality of the game is further enhanced.

Although a fighting game has been described as an example of application to the game, the present invention is not limited to this.
A game that recognizes a certain stroke from the movement of the hand and simulates a command or musical instrument that controls musical sounds and the like may be used. Alternatively, a game in which the movement of a real robot is controlled and moved instead of the character on the screen may be used.

[0032]

Embodiment 3 FIG. 4 is a schematic configuration diagram of a presentation system using the motion recognition system of the present invention. A presentation person 501, a motion recognition module 502 for recognizing a motion of the presentation person, a presentation device 503,
The screen 504 includes a dictionary 505 that stores commands for controlling the presentation device 503.

The motion recognition module 502 has basically the same configuration as the recognition system of the first embodiment.
A control provided with a plurality of CCD cameras for recognizing the state of a person 501 who makes a presentation, detecting a motion pattern, and recognizing the recognized state and the detected motion pattern as a result of referring to the dictionary 505 A command is input to the presentation module 503. The presentation module 503 generates a predetermined screen according to the control command and projects the screen on the screen 504. The presentation module is a projection device connected to a conventional personal computer, and projects a screen produced for presentation on a screen.

The motion recognition module 502 is, for example, “a quick protruding hand in a par-shaped hand”, “a left-right stroke in a par-shaped hand”, “a vertical stroke in a goo-shaped hand” of a person 501 giving a presentation. "Left and right stroke of choki hand""squatting""jump"
Recognize the six types of motion. , Identifiers (PFF, PLR, GUD, CLR, SD, JM) corresponding to each motion of the person 501 giving the presentation
Is formed, and a control command corresponding to the identifier is supplied to the presentation module 503 with reference to the identifier and the dictionary 505 corresponding to each motion. The person 501 giving the presentation can edit the dictionary 505 and replace the command corresponding to the identifier according to the presentation plan.

For example, a person 5 giving a presentation
When 01 is recognized as a motion that makes “a quick protrusion of the hand in the shape of a par”, the identifier PFF is activated for the presentation module 503 and “left and right of the hand in the shape of a par” is activated. When it is recognized that the motion is a “stroke”, the presentation module 503 is notified of the identifier PL.
Activate R. Similarly, if it is recognized that the current motion of the person 501 giving the presentation is “a vertical stroke in the shape of a goo hand”, the presentation module 503 is notified of the identifier GU.
D is activated, and when it is recognized as “the left and right strokes of the hand of the choki”, the identifier CLR is activated for the presentation module 503. Further, when the current motion of the person 501 giving the presentation is recognized as “squatting”, the identifier SD is activated for the presentation module 503, and when it is recognized as “jump”, Presentation module 503
Activate the identifier JM.

The presentation module 503 includes:
These identifiers (PFF, PLR, GUD, CLR, S
D, JM) to give a presentation 50
The motions of 1 are "Par-shaped hand protruding fast forward", "Par hand-shaped left and right stroke", "Goo hand-shaped up and down stroke", "Choki hand-shaped left and right stroke", "Squatting" The identifier (PF) is recognized as one of the six motions of “jump”.
F, PLR, GUD, CLR, SD, JM)
Advance the presentation.

For example, in the dictionary 505, the identifier PF
F is associated with a “start screen start command”, the identifier PLR is associated with a “command to switch to the next screen”, the identifier GUD is associated with an “enlarge screen command”, and the identifier CLR is associated with “identifier CLR”. "Scroll command of screen" is corresponded, and the identifier SD is "UNDO command".
And dictionary editing can be performed so that the identifier JM corresponds to the “end command”. Changing the correspondence from the necessity of presentation is easy only by editing the dictionary 505.

Since it is often effective to draw and explain a simple illustration in a presentation, it is also possible to track the trajectory of a hand and display the trajectory as an illustration on a screen of the presentation. For example, the motion of “quickly protruding the hand with the forefinger” as shown in FIG. 6 is made to correspond to the “drawing start command”, and the motion of “the hand with the forefinger is accelerated and exits the shooting area” is changed to “drawing end”. By tracking the trajectory of the index finger between the start and end of drawing and displaying the trajectory with a line of a predetermined color and thickness,
In other words, a gesture drawing system can be created. In addition, the `` Spread your hands '' action corresponds to the `` deformation command of extending the illustration in the direction of both hands '', and the `` Move your hand in the form of a hand holding an object '' When the “movement command” is associated, more advanced gesture drawing or gesture editing can be performed.

By adopting such a system configuration, in a presentation using a projector or the like,
In order to control the presentation screen, there is no need to interrupt the operation by touching the presentation device, the personal computer, or the like, and smooth progress can be achieved.

[0040]

[Embodiment 4] FIG. 7 is a flowchart in the case of evaluating a golf swing form by the system of the present invention. In an information amount calculation step 701, target position information, a three-dimensional shape, and a volume are calculated from a distance image when the target golfer swings the club. Here, it is assumed that shooting is performed at a video rate. In the state recognition step 702, “head”, “shoulder”, “elbow”, “hand” stored in the table 704
With reference to the basic state of the "club head", each part and its position are recognized for each frame. In the pattern detection step 703, the positional relationship of each frame of “head”, “shoulder”, “elbow”, “hand”, and “club head”, and the speed, acceleration and trajectory of “club head” are detected. In the estimating step 705, an estimated value relating to a state which is shaded by a twist of the body and is not acquired by the camera is calculated. In the form recognition step 706, the form is recognized by referring to the state, the pattern, the estimated value, and the dictionary 707 thus obtained. In this case, the result of form recognition is evaluation of the form. Display evaluation step 708
, The evaluation is displayed on a monitor (not shown) or the like.

In this case, the target is a golfer, but the form evaluation system of the present invention is not limited to this.
The present invention can also be used for general sports such as a baseball batter's swing and tennis serve evaluation. Also,
The acquired state, pattern, and estimated value may be applied to three-dimensional modeling for modeling the movement of a sporting person. Alternatively, it may be applied to sign language interpretation.

[0042]

Embodiment 5 FIG. 8 is a schematic diagram of a drowsiness prevention system using the motion recognition system of the present invention. The dozing state recognition module 801 refers to a dictionary 802 that stores basic values related to a “pendulum-like stroke of a head” specific to a dozing state based on distance images acquired by a plurality of cameras installed in the vehicle, and Evaluate the degree of
When the evaluation value indicates a specific value, a control signal is sent to the alarm device 803 to issue an alarm.

Here, the dozing state of the car was evaluated. However, a camera was installed in the sickroom, and a reference value for an action peculiar to seizure or abnormal operation of the sick person, for example, “pendulum movement before and after a short cycle of the head” Is stored in a dictionary,
It is also effective as a care monitoring system. Further, if a camera is installed in an art museum or the like and reference values relating to “reciprocal motion peculiar to a suspicious individual” are stored in a dictionary, it is effective as a security system.

[0044]

According to the present invention, distance information calculated from a plurality of images obtained by imaging with a plurality of cameras, and an object performing motion using various feature amounts can be contacted without contact.
In addition, it is possible to provide a system that can recognize motion regardless of the background.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment.

FIG. 2 shows details of a distance measuring unit.

FIG. 3 is a schematic diagram of a game machine.

FIG. 4 is a flowchart of a game.

FIG. 5 is a schematic diagram of a presentation system.

FIG. 6: Hand protruding with the index finger raised

FIG. 7 is a flowchart of form recognition.

FIG. 8 is a schematic diagram of a dozing driving prevention system.

[Explanation of symbols]

 Reference Signs List 101 distance measurement unit 102 imaging unit 103 information amount calculation unit 104 state recognition unit 105 pattern detection unit 107 meaning recognition unit

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (reference) G10K 15/04 302 G10K 15/04 302F 5D108 H04N 13/00 H04N 13/00 5L096 // H04N 7/18 7/18 K F-term (reference) 2F065 AA00 AA04 AA06 AA37 AA51 AA59 CC16 FF01 FF04 JJ03 JJ05 JJ26 QQ23 QQ31 RR06 SS02 SS09 SS14 2F112 AC06 5B057 AA05 BA11 CH07 DA07 DA20 DC08 DC09 DC30 DC33 5C05 CG03 FC03 FC03 GW06 FC03 GD06 HA04 HA15 HA16 5C061 AA29 AB04 AB08 AB24 5D108 CA03 CA07 CA16 CA29 5L096 AA09 BA04 BA05 BA16 CA05 FA66 HA07

Claims (16)

[Claims] 1. A motion recognition system, comprising: 1) an image pickup means for picking up an image of an object; and 2) a principle of triangulation from a plurality of images obtained by imaging the object with a plurality of cameras. Distance measuring means for calculating distance information of an imaging area; 3) state recognizing means for recognizing a state of the object from an image of the object and an information amount calculated from the distance information; A system comprising: pattern detection means for detecting a pattern relating to movement from a state; and 5) meaning recognition means for recognizing predetermined implications from the state and the pattern. 2. A motion recognition system comprising: 1) an image pickup means for picking up an image of an object; and 2) a principle of triangulation from a plurality of images obtained by imaging the object with a plurality of cameras. Distance measuring means for calculating distance information of an imaging area; 3) state recognizing means for recognizing a state of the object from an image of the object and an information amount calculated from the distance information; Pattern detection means for detecting a pattern relating to movement from a state; 5) a table storing a predetermined state and a basic state composed of the predetermined pattern and an identifier corresponding to each of the basic states; And 7) a meaning recognizing means for recognizing the implication by comparing the recognized state and the detected pattern with the dictionary. System, wherein the door. 3. The information (estimated value) on the state of the target object estimated using past information in the meaning recognition means.
3. The system according to claim 1 or 2, wherein the implication is recognized by referring to as necessary. 4. The information amount according to claim 1, wherein the information amount is at least one selected from an orientation, a spatial position, a position and a shape, a volume, and a three-dimensional form of the object. Item 3. The system according to Item 2 or 2. 5. The state is at least one selected from the presence / absence and state of a part specified in advance, the presence / absence and state of an object specified in advance, a reference value, and a comparative evaluation result. The system according to claim 1 or 2, wherein: 6. The pattern relating to the movement is selected from a velocity of the target area, an acceleration of the target area, a trajectory of the target area, a stroke of the target area, a positional relationship and a change of a plurality of target areas, and a dynamic deformation of the target area. 3. The system according to claim 1, wherein the system is at least one of: 7. The implication includes a command for instructing operation of a machine or a system, a motion of a robot, a motion of a character in a screen, a screen display instruction, an object generation,
Control of object functions, instructions for editing documents, instructions for creating graphics, instructions for controlling voice or musical sounds,
3. The system according to claim 1, wherein the system is at least one selected from interpretation of sign language, intention information to be transmitted through a human interface, and an evaluation value prepared in advance. 8. The system according to claim 2, wherein the function of motion recognition can be changed by exchanging the basic state corresponding to the identifier or the implication corresponding to the identifier. 9. The system according to claim 1, wherein the recognized implication is represented by at least one selected from a screen display, a voice, and a machine operation. . 10. The system according to claim 1, wherein said object to be imaged is a person who is playing sports, and said movement pattern is characteristic of said sports. A form of action, wherein the implication is an evaluation or commentary on the form. 11. The system according to claim 1, wherein at least one selected from an animation and a robot is controlled in accordance with the recognized implication. 12. The system according to claim 1, further comprising sound generating means for generating a sound signal, wherein the sound signal is controlled based on the recognized implication. system. 13. The system according to claim 1, wherein said object to be imaged is a human, and said movement pattern is a characteristic characteristic of a specific physiological state. The system wherein the implication is an assessment of the degree of a physiological condition. 14. The system according to claim 13, wherein the specific physiological state is a dozing driving state, and the implication is an evaluation relating to a degree of dozing. 15. The system according to claim 1 or 2, wherein said implication includes a start and an end, and a state or pattern related to said start implication and a state or pattern related to said end. A system for generating an illustration based on a pattern relating to a motion detected in the above. 16. The system of claim 14, wherein the implication is a transformation of an already generated illustration.
A system that includes a movement and transforms or moves the illustration according to the recognized implications.