EP2092408A1

EP2092408A1 - Integrated circuit for recognizing movements of persons

Info

Publication number: EP2092408A1
Application number: EP07820218A
Authority: EP
Inventors: Jens Schick; Alexander Wuerz-Wessel
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2006-11-14
Filing date: 2007-09-14
Publication date: 2009-08-26
Also published as: WO2008058783A1; US20090322888A1; DE102006053837A1

Abstract

Disclosed is an integrated circuit which processes image data of a video camera by determining the optical flow and calculates output data therefrom that are a measure of the position and/or movement of individuals' body parts or represent and encode individuals' gestures. Also disclosed are an electronic data processing system, a method, and a computer program.

Description

description

title

INTEGRATED CIRCUIT FOR DETECTING PERSON'S MOVEMENTS

State of the art

The invention relates to an integrated circuit, an electronic data processing system, a method for calculating output data of an integrated circuit and a computer program.

Personal computers (PC) have a keyboard and a mouse for input. The orientation in the program or the feedback takes place via a graphical interface on a screen partly together with a speaker output. The disadvantage is that in narrow workplaces such as the seat in an airplane, the mouse can not be moved freely and the operation of the keyboard is difficult.

Increasingly, video games are finding more and more followers and are becoming increasingly popular. Video games are used on personal computers as well as game consoles. The input is preferably via keyboard, mouse or joysticks. The disadvantage is that the use of these instruments the player to the

Device binds.

German patent application DE 195 14 877 A1 discloses a game console based on a personal computer. For the operation interfaces for joystick or trackball are provided. Furthermore, an interface is provided for the output of a screen over which sound and image data are output.

Disclosure of the invention

Advantages of the invention The integrated circuit described below has the advantage that the determination of the optical flow of image data and the integration of this algorithm in an integrated circuit enables cost-effective, precise and rapid determination of output data, which is a measure of the position and / or movement of

Body parts of a person are and / or represent the gestures of the person.

It is particularly advantageous if the first unit which implements the algorithm for determining the optical flow, the stereo disparities and / or the symmetry clusters is hard-wired, since this results in an optimization of the integrated

Circuits is possible, so that it works very fast and efficient. The programmable second unit, in which the calculation of the output data is performed, has the advantage that thereby an application and function-specific adjustment is possible, so that the same integrated circuit can be used in many applications.

It is also advantageous that the output data encode the sign language, as this deaf and very hard of hearing people a simple communication platform with electronic data processing systems is provided.

The advantages of the integrated circuit described above also apply accordingly to the electronic data processing system, the method and the computer program.

Further advantages will become apparent from the following description of embodiments with reference to the figures and from the dependent claims.

drawing

The invention is explained below with reference to the embodiment shown in the drawing.

Show it: 1 shows an electronic data processing system, Figure 2 shows an integrated circuit, Figure 3 shows a person in side view, - Figure 4 shows a person in front view.

Description of exemplary embodiments

An integrated circuit will now be described wherein the integrated circuit processes image data from a video camera by means of optical flow determination and calculates therefrom output data which is either a measure of the position and / or movement of human body parts or represent and encode gestures of people. Furthermore, an electronic data processing system, a method and a computer program are described.

Hereinafter, a high-resolution measurement of the gestures of persons in a dense stereo image for controlling a personal computer or a game console will be described. The video camera is designed as a stereo camera and is located above the screen and observes the space in front of the personal computer. In the preferred embodiment, the position of fingers, hands, arms,

Determine the torso, legs, feet and / or head of the person with their rotations from the image data of the video camera and use for input as an alternative to the mouse, keyboard or joystick. For this purpose, algorithms for measuring the optical flow, stereo disparities and / or symmetry clusters are used. In the preferred embodiment, this input of information over an optical channel serves to give a player of a video game a greater opportunity to intervene and thus to convey a higher game value. These intervention options are for controlling a virtual player or other object of a video game, such as a car.

1 shows an electronic data processing system 1 of the preferred embodiment, consisting of a personal computer 10 (PC), an integrated circuit 12 and a stereo camera 16. As an alternative to the personal computer 10, a notebook or a game console is used in a variant. The personal computer 10 includes a processor 14 for processing data, a Memory 20 for storing data and the integrated circuit 12. The processor 14 is connected via interfaces optionally with the interposition of other electronic components, such as interface modules, with a mouse 22 and a keyboard 24 as input units. Furthermore, the processor is optionally via interfaces with the interposition of other electronic components, such as

Interface modules, connected to a speaker 26 and a screen 28 as output units. An input of the processor 14 is further connected to an output of the integrated circuit 12. The integrated circuit in turn is optionally connected to a stereo camera 16 with the interposition of further electronic components, such as interface modules. The

Stereo camera 16 consists of two video cameras 18 which record substantially the same scene. The video cameras 18 are arranged side by side and their optical axes are substantially parallel so that the video cameras 18 record substantially the same scene, albeit from a slightly different angle. In the preferred embodiment, the stereo camera 16 is above the

Screen 28 is located and observes the area in which the operator of the personal computer 10 is located. The stereo camera 16 generates image data by means of the two video cameras 18 and transmits them to the integrated circuit 12. The structure of the integrated circuit 12 will be explained in more detail below with reference to FIG. In the memory 20 of the electronic data processing system 1 on the one hand the

Operating system of the personal computer 10 stored. On the other hand, the memory 20 serves on the one hand for the storage of programs of office applications, such as word processing programs, and on the other hand for the storage of software programs of video games. In the preferred embodiment, the electronic data processing system 1 for both office applications and for

Used video games.

The integrated circuit 12 is for calculating movements and distances of objects located in the area captured by the stereo camera 16. If, for example, a person raises a hand in the detection range of the stereo camera 16, the hand is detected by the movement and measured by means of a stereo evaluation, wherein the resolution enables the individual measurement of fingers. The integrated circuit 12 simultaneously detects and interprets the movement of all body parts of the persons located in the field of view of the stereo camera 16, the integrated circuit 12 providing output data which is a measure of the position and / or movement of Are body parts of a person and / or represent the gestures of the person at the output of the processor 14 provides. The integrated circuit 12 is thus configured such that on the one hand it provides pure position and / or movement data and on the other hand interpreted data which encode a gesture of the person.

Figure 2 shows the integrated circuit 12 consisting of a first unit 30 and a second unit 32. The integrated circuit 12 comprises two inputs 34 and 36 for connecting two video cameras 18 of a stereo camera 16 and an output 38. The integrated circuit 12 acts it is in the preferred embodiment to an ASIC. The abbreviation ASIC stands for "Application Specific Integrated Circuit", ie for an application-specific integrated circuit. An ASIC is an electronic circuit realized as an integrated circuit. In a variant of the preferred embodiment, an FPGA is used instead of the ASIC. The abbreviation FPGA stands for "Field Programmable Gate Array" and denotes a freely programmable logic circuit. Common to both variants is that the integrated circuit 12 consists of two logic units 30, 32. The first unit 30 is hardwired and not programmable. This first unit 30 calculates pre-processed image data by means of a determination of the optical flow from the image data of the stereo camera. In the preferred embodiment, the first unit 30 additionally computes stereo disparities and / or symmetry clusters. Overall, the first unit 30 calculates preprocessed image data and thereby performs data reduction. The preprocessed image data is passed to the second unit 32. In contrast to the first unit 30, the second unit 32 is characterized in that the second unit 32 is programmable. In the second unit, application-specific determines which output data the second unit 32 from the preprocessed

Calculated image data. The baseline data is a measure of the position and / or movement of a person's body parts and / or represents the gestures of the person being photographed. This output data is provided at the output 38 of the integrated circuit 12.

FIG. 3 schematically shows a person 40 taken by the video cameras in left side view to explain the output data which is a measure of the position and / or movement of body parts of persons 40 and provided by the integrated circuit at the output. The person 40 includes a head 42, a trunk 44, a right arm 46 with a right hand 48, and a left arm 50 with a left hand 52. Further, FIG. 3 shows a coordinate system 54 having a y and a z axis. In FIG. 3, crosses indicate some points which are output data of the integrated circuit and indicate positions of body parts 42, 44, 48, 52 of person 40:

P _RV = (z) of the foremost space point of the fuselage 44 P _HR ⁼ (x, y, z) of the foremost space point of the right hand 48 P _HL = (x, y, z) of the foremost space point of the left hand 52 P _K v ⁼ (z) of the foremost space point of the head 42 Pκo = (y) of the top spatial point of the head

Figure 4 schematically shows a person 40 taken by the video cameras in front view to explain the output data which is a measure of the position and / or movement of body parts of persons 40 and provided by the integrated circuit at the output. The person 40 includes a head 42, a hull

44, a right arm 46 with a right hand 48 and a left arm 50 with a left hand 52. Further, FIG. 4 shows a coordinate system 54 having an x and a y axis. In FIG. 4, some points are marked by crosses, which are output data of the integrated circuit and identify positions of body parts 42, 44, 48, 52 of person 40:

B _HR = (Xb, yb) of the foremost space point of the right hand 48 B _HL ⁼ (Xb, yb) of the foremost space point of the left hand 52 Φ _K s ⁼ angle of the symmetry axis in the image B _K G ⁼ (Xb, yb) as pixels the face reference

Bκo = (Xb, yb) of the top spatial point of the head 42

B _K S ⁼ (Xb, yb) of the point on the symmetry axis closest to B _K o

Further, from the positions of the body parts 42, 44, 48, 52 of the person 40 shown in FIGS. 3 and 4, further output data are calculated in the preferred exemplary embodiment and made available at the output of the integrated circuit:

I _KR = distance of the rightmost point of the head 42 B _KR to the symmetry axis in the picture I _KL = distance of the leftmost point of the head 42 B _K ι_ to the symmetry axis in the picture lκo = distance of the point B _K s to B _KG MHL = (XHL, YHL, ZHL-ZRV) ^■ $ Measures for the relative position of the left hand 52

M _HR = (X _HR , Y _HR , Z _HR -Z _RV ) -> Measures for the relative position of the right hand 48

M _GV = (Z _KV - Z _RV ) -> measure of the forward speed

M _GS = (Φ _KS ) "^ Measure for the sideways speed

MGR = (0.5 - IKL / (IKL ⁺ IKR)) ^■ $ Measure for body rotation

M _GH = (yκo (k) - yκo (k-1)) -> measure of jumping

MBR = (I κo / (IKL + IKR)) ^■ $ Measure for the viewing direction

In the preferred embodiment, the output data are further calculated in the integrated circuit from the image data of the video cameras by means of the determination of the optical flow, which is a measure of the position and / or movement of

Represent body parts of the person:

foot position

Knee angle of the foot to the lower leg - knee position

Bending angle between lower and thigh

Solid angle of the thigh

Bending angle between thigh and torso

Solid angle of the trunk - angles and positions of fingers and toes

The calculation of output data representing the gestures of the

Represent the person and thus code the gestures:

Lifting a finger of a person's hand means a start condition, lowering the finger a stop. Thus, the gesture of movement of this finger is one

Alternative to the mouse. An input confirmation similar to the keyboard "Enter" or a click of the right mouse button is generated by the abrupt movement of the finger and calculated by the integrated circuit.With involvement and combination of movement and measurement of the remaining body parts, the input variety is almost unlimited.

Further, the integrated circuit calculates output data suitable for controlling virtual objects such as figures, games, cars, video games. there In the preferred embodiment, the programmable second unit of the integrated circuit applies the following rules for coding the detected gestures of the person:

Locomotion coding: standing of the virtual figure no movement of the person's trunk -> no movement of the virtual

Figure coding: walking the virtual figure alternately change the position and angle of the two thighs of the person -> the frequency determines the speed of the virtual figure

Coding: running the virtual figure

Sprung walking, but with simultaneous evaluation of the

Body vertical movement of the person -> the frequency determines the

Speed of the virtual figure Coding: jumping of the virtual figure strong vertical movement in parallel position of the thighs of the person ->

The strength of the vertical movement determines the strength of the jump of the virtual

Figure Rotation Coding: Rotation vertical axis of the virtual figure

Rotation of the head around the vertical axis of the person -> rotational position of the head corresponds to rotation speed of the virtual player

Turning speed of the head of the person -> corresponds to spin of the virtual figure face towards the person's video camera -> means stoppage of the person

Rotation of the virtual figure (measurement of the angle of rotation of the head by distance from the head center axis to the axis of facial symmetry, face is the sum of the features of the eye, nose and mouth, measurement of the rate of rotation of the head by horizontal optical flow in the face and the displacement speed of the head center axis)

Coding: nod of the virtual figure

Rotation of the head about its horizontal axis, rotational position of the head corresponds to the viewing direction of the virtual figure, (measurement of the center of gravity relative to the top of the head, calibration at the beginning of the game) Coding: shaking of the virtual figure

Rotation of the head around the axis forward of the person -> rotational position of the head corresponds to the fast sideways deflection of the virtual figure (measurement of the direction of the axis of symmetry of the face) - actions, communication

Coding: positioning of virtual devices

Position of the hands in the space of the person for positioning of virtual devices (weapons, shields, tools, gear sticks, ...) relative to the body of the virtual figure, even in combination of the two hands (for example, the virtual steering wheel)

Encoding: Alignment of virtual devices

Direction of the person's thumb to align the virtual devices. Position and solid angle of the person's feet for driving virtual vehicle pedals (clutch, brake, gas) Coding: Activation of the devices

Number and movement of the outstretched fingers of the person to activate these devices or to communicate with another player.

Furthermore, it is provided in the preferred embodiment that a scene change and / or switching of devices is performed by combining gestures as pantomime.

In summary, by capturing the person by the video cameras, the processing of the image data by the integrated circuit and thus the provision of output data representing and encoding the person's gestures, and the association of the person's recorded gestures with behavioral elements of the video game's virtual objects are enabled that these virtual objects in the areas of locomotion (standing, walking, running at speeds, jumping with its strength), rotation (turning at its rotational speed around the vertical axis, pitch axis and roll axis), actions and communication (actions with both arms independently , Activation of devices, communication with partners) by the person being captured, controlled and controlled by the video camera.

In a variant of the preferred embodiment, output data is provided by the integrated circuit, the gestures of the widely used Codify sign language. In this case, for the purpose of coding, gestures of hands in connection with facial expressions and mouth images of the person are detected by the video cameras and evaluated by the integrated circuit and provided as output data. This is preferably evaluated by the integrated circuit in the context of body posture.

In one variant, aids, for example a rod and / or a dumbbell, which are used by the person, are used to improve the calculation. This contributes in particular to an improved fine measurement of the hand movements since their position can be measured more accurately, because the shape and coloring of the

Aids are known to the integrated circuit.

Another variant provides that the integrated circuit and the video camera replace the function of the keyboard. This is done by simultaneously observing the ten fingers of the person from the video cameras. For this purpose, both hands of the person are held in front of the video cameras. The kinking of one finger or the combination of several fingers completely emulates the keyboard through the integrated circuit.

The described integrated circuit, the data processing system, the method and the computer program are not limited to the field of personal computers and video games, but can also be used in industrial control, as well as screen-free systems. In this case, preferably the feedback of the input via other media, such as speakers. Particularly advantageous is the use of the integrated circuit in the field of

Driver assistance systems for detecting pedestrians in the environment of a motor vehicle with a video camera. Furthermore, as an alternative or in addition to the stereo camera, a single video camera is used.

Claims

claims

An integrated circuit having at least one input for connecting a video camera and for receiving image data of the video camera, the integrated circuit having a first unit which calculates from the received image data by means of a determination of the optical flow pre-processed image data, the integrated circuit a second Unit which calculates output data from the preprocessed image data which are a measure of the position and / or movement of a person's body parts and / or represent the person's gestures, the integrated circuit having at least one output for providing the output data.

2. Integrated circuit according to claim 1, characterized in that the first unit is hardwired and / or the second unit is programmable.

3. Integrated circuit according to one of the preceding claims, characterized in that the integrated circuit is an ASIC or an FPGA.

4. Integrated circuit according to one of the preceding claims, characterized in that the output data encode the sign language.

5. An electronic data processing system with an integrated circuit according to one of the preceding claims and at least one video camera.

6. An electronic data processing system according to claim 5, characterized in that the video camera is a stereo camera.

7. An electronic data processing system according to any one of claims 5 or 6, characterized in that the data processing system comprises a keyboard and / or a mouse and / or a screen and / or a speaker.

8. A method for calculating output data of an integrated circuit, preferably the integrated circuit according to one of claims 1 to 4, wherein from image data of a video camera by means of a determination of the optical flow preprocessed image data are calculated, are calculated from the preprocessed image data output data, the gestures represent a person.

9. The method according to claim 8, characterized in that objects of a video game are controlled in dependence of the output data.

Computer program with program code means for carrying out all the steps of the method according to one or all of claims 8 or 9, when the program resides on a computer and / or the integrated circuit according to one of claims 1 to 4 and / or the electronic data processing system one of claims 5 to 7 is executed.