KR20100075356A - Apparatus and method for detecting upper body pose and hand shape - Google Patents

Abstract

PURPOSE: An apparatus and a method for detecting an upper body pose and a hand shape are provided to grasp peripheral environments in 3 dimension when using at least two cameras. CONSTITUTION: A stereo vision computing unit(300) calculates three dimensional distance information from the image provided form at least two cameras(101-103), and a color pattern classification unit(400) outputs the color information and the pattern information by classifying the color and the pattern from the images. Plural location detecting units(610-650) extract 3D location information for each target, and an upper body pose recognition unit(710) recognizes an upper body by using the 3D location information for each target. A hand shape recognition unit(720) recognizes the shape of a hand by using the hand location information.

Description

Apparatus and method for detecting upper body pose and hand shape}

The present invention relates to an upper body posture and a hand-shaped detection device and a method thereof, wherein the upper body posture and the shape of the fine hand can be calculated and detected separately, and can also be detected in real time and can reduce the calculation time required for detection. It relates to a posture and hand-shaped detection device and a method thereof.

The present invention is derived from a study conducted as part of the IT source technology development project of the Ministry of Knowledge Economy [Task management number: 2008-F-037-01, Task name: u-robot HRI solution and core device technology development].

In the past, a background screen or a reference background screen was required to detect the shape or motion of a person. In other words, by using the difference between the reference background (or the background screen) and the current screen, a mass of each part of the body was made, and the human body model was constructed using each mass. Or after extracting the feature points to detect the shape or motion of the human body model or human, three-dimensional position calculation for each feature point was performed.

Therefore, in the related art, a background screen or a reference background screen is required to detect a shape or a motion of a person, or a detailed motion such as the shape of a hand cannot be detected, and it takes a lot of calculation time.

In order to solve the problems of the prior art described above, the upper body posture and hand shape that can directly detect the three-dimensional position and shape of the hand, face, torso, shoulder, elbow, hand using stereo vision and pattern color classifier A detection apparatus and a method thereof are provided.

The object of the present invention is not limited to the above-mentioned object, and other objects that are not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided an upper body posture and a hand shape detecting apparatus, including: a stereo vision calculator configured to calculate 3D distance information from an image provided from at least two cameras, and a color and a pattern from the image. A color pattern classification unit for classifying the color information and pattern information, a plurality of position detection units for extracting 3D position information for each object by using the 3D distance information, the color information, and the pattern information; The upper body posture recognition unit recognizes the upper body posture by using the three-dimensional position information for each object, and a hand shape recognition unit that recognizes the shape of the hand using the hand position information among the three-dimensional position information for each object.

According to another aspect of the present invention, an upper body posture and hand shape detecting apparatus includes a stereo vision calculating unit calculating three-dimensional distance information from an image provided from at least two cameras, and a color detecting unit detecting an area having a skin color from the image. And a pattern detection unit for detecting a pattern for each body part from the image, classifying each body part from the image by using the 3D distance information, the skin color region, and the pattern for each body part. A position detection unit for detecting a position, an upper body posture recognition unit for recognizing the upper body posture using the position of each body part, and a shape of the hand using the hand position information among the three-dimensional position information for each object It includes a hand recognition unit.

According to another aspect of the present invention, there is provided a method for detecting upper body posture and hand shape, comprising: calculating three-dimensional distance information from an image provided from at least two cameras, detecting an area having a skin color from the image, and Detecting a pattern of each body part from an image, detecting a location of each body part using the 3D distance information, the region of the skin color, and the pattern of each body part, and using the location of each body part Recognizing the upper body posture and the shape of the hand using the hand position information of the three-dimensional position information for each object.

Specific details of other embodiments are included in the detailed description and the drawings.

According to the present invention, when two or more cameras are used, the surrounding environment can be grasped in three dimensions, and a stereo vision calculator and a color pattern classifier can be used to grasp the three-dimensional position of the user's face, torso, shoulder, elbow, and hand in real time. have.

In addition, it is possible to configure the interface between the user and the computer without using a device such as a mouse or a remote control to enable the user's convenience and various applications. That is, by recognizing the shape of the hand, the user can give various commands even in the same posture, so that various applications can be configured. For example, when mounted on a TV, the TV can be manipulated by hand gestures and applied to various games.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Meanwhile, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. As used herein, “comprises” and / or “comprising” refers to the presence of one or more other components, steps, operations and / or elements. Or does not exclude additions.

An upper body posture and hand shape detecting apparatus and a detecting method according to an exemplary embodiment of the present invention will be described with reference to FIGS. 1 to 4. 1 is a block diagram illustrating an upper body posture and hand shape detection apparatus according to an embodiment of the present invention, FIG. 2 is a block diagram illustrating a stereo vision calculation unit of FIG. 1, and FIG. 3 is a block diagram illustrating a color pattern classification unit of FIG. 1. Fig. 4 is an exemplary view showing upper body posture and hand-shaped detection results.

Referring to FIG. 1, the upper body posture and hand shape detecting apparatus 10 according to an exemplary embodiment of the present invention includes at least two cameras 101, 102, and 103, an image data preprocessor 200, and a stereo vision calculator. 300, the color pattern classifier 400, the body part classifier 500, the plurality of position detectors 610, 620, 630, 640, and 650, the user upper body posture recognizer 710, and the hand. The shape recognition unit 720 and the application interface unit 800 is included.

Two or more cameras 101, 102, and 103 photograph a person (hereinafter referred to as a user) and output a multi-view image. The two or more cameras 101, 102, 103 may be mounted on a wall or a home appliance such as a computer monitor or a TV, for example, and may be driven to photograph a user. For example, the cameras 101, 102, and 103 may move to capture the user by recognizing the user's movement or voice.

The image data preprocessor 200 performs preprocessing on image data provided from two or more cameras 101, 102, and 103. For example, the image data preprocessor 200 may perform image quality enhancement such as noise removal, color correction, brightness correction, and screen distortion correction for image data provided from each of the cameras 101, 102, and 103. The preprocessed image is transmitted to the stereo vision calculator 300 and the color pattern classifier 400.

First, the stereo vision calculator 300 calculates a stereo image of images input from the plurality of cameras 101, 102, and 103 and provides 3D distance information to each object in the screen.

Referring to FIG. 2, the stereo vision calculator 300 includes a stereo vision preprocessor 310, a stereo matcher 320, and a stereo vision postprocessor 320.

The stereo vision preprocessor 310 adjusts the brightness of the plurality of cameras 101, 102, and 103, removes noise, or corrects distortion for stereo matching. Alternatively, the stereo vision preprocessor 310 may perform calibration, scale down filtering, correction, brightness control, and the like.

The stereo matching unit 320 extracts distance information by searching for portions (Stereo Matching) corresponding to each other from the images of the plurality of pre-processed cameras 101, 102 and 103. In this case, the distance information output from the stereo matching unit 320 may be, for example, an image expressed in brightness according to a distance (a close object is a bright color, and a far object is a dark color) or a color is different according to a distance. It may be an expressed image. The distant background can be deleted here.

The stereo vision post processor 320 post-processes the distance information output from the stereo matcher 320. For example, the stereo vision post-processing unit 320 may remove noise from the distance information output from the stereo matching unit 320 or perform segmentation and labeling of different objects from the distance information.

Next, the color pattern classification unit 400 of FIG. 1 classifies and analyzes colors and patterns in the screen to obtain color information and pattern information for recognizing a user's body part, for example, a face, a torso, a shoulder, an elbow, a hand, and the like. Output

More specifically, referring to FIG. 3, the color pattern classifier 400 includes a color detector 410 and a pattern detector 450.

The color detector 410 includes a color corrector 420, a skin color detector 430, and a morphological filter 440.

The color corrector 420 corrects the color from the image preprocessed by the image data preprocessor 200. For example, if the image is entirely red, the red component may be lowered. Alternatively, the color corrector 420 may adjust a ratio of R (Red), G (Green), and B (Blue) components of the image.

The skin color detector 430 may detect the skin color from the corrected image. That is, an area having a skin color may be detected in the corrected image.

The shape filter 440 determines the shape of the region having the skin color through filtering and outputs color information corresponding thereto. For example, the shape filter 440 may determine the shape of an area having a skin color by determining an edge of an area having a skin color output from the skin color detector 430 through shape filtering.

Next, the pattern detector 450 detects the pattern information from the image preprocessed by the image data preprocessor 200. The pattern detector 450 may detect patterns of body parts from the preprocessed image. For example, the pattern detector 450 may detect an area having a pattern of a face, a torso, a shoulder, an elbow, and a hand from the preprocessed image. The pattern detection unit 450 may detect an area having a pattern of a face, a torso, a shoulder, an elbow, and a hand from the image by comparing the pattern sample of the face, the body, the shoulder, the elbow, and the hand with an image, for example. .

As described above, the stereo vision calculator 300 and the color pattern classifier 400 may reduce the calculation time by separately processing the images preprocessed by the image data preprocessor separately and in parallel. In addition, the color pattern classification unit 400 may also internally operate the color detector 410 and the pattern detector 450 separately and in parallel to reduce the calculation time.

Meanwhile, the body part classifier 500 of FIG. 1 uses the 3D distance information output from the stereo vision calculator 300 and the color information and pattern information output from the color pattern classifier 400. Each body part can be classified. That is, the user's face, torso, shoulders, elbows, and hands are classified from the image.

In addition, the plurality of position detectors 610, 620, 630, 640, and 650 detect the position of each body part. For example, the face position detector 610 detects the three-dimensional position of the user's face, the torso detector detects the three-dimensional position of the user's torso, and the shoulder position detector 630 detects the three-dimensional position of the user's shoulder. The elbow position detector 640 detects a three-dimensional position of the elbow of the user, and the hand position detector 650 detects a three-dimensional position of the user's hand.

As described above, the plurality of position detection units 610, 620, 630, 640, and 650 detect positions of each body part individually and in parallel, so that calculation time is fast and three-dimensional distance information and color information of each body part are fast. And three-dimensional position information generated from the pattern information, so that each body part can be detected in detail.

The upper body posture recognizer 710 recognizes the upper body posture of the user by using three-dimensional position information for each body part. For example, the upper body posture recognizer 710 may recognize the upper body posture of the user by reconstructing the upper body posture of the user using 3D position information of the face, torso, shoulder, elbow, and hand.

In addition, the hand recognition unit 720 may recognize the shape of the user's hand using the three-dimensional position information of the hand.

Through this process, the upper body posture information and the hand shape information of the user is derived as shown in Figure 4, the upper body posture information and the hand shape information of the user is transmitted to the external device (or network) through the application interface unit 800 Are utilized.

As described above, the three-dimensional distance information, the color information, and the pattern information are generated separately, and the shape, face, torso, shoulder, elbow, and hand shape are independently detected using such information, so that the calculation time is increased. In addition, the detailed hand shape may be detected in real time using the 3D distance information, the color information, and the pattern information. In addition, it can be connected to an external device to finely control the external device through the user's upper body posture and hand shape. For example, different controls can be performed according to the upper body posture and the shape of the hand. In other words, by recognizing the shape of the hand, the user can give various commands even in the same posture, so that various applications can be configured.

An upper body posture and hand shape detection apparatus according to another embodiment of the present invention will be described with reference to FIG. 5. 5 is a block diagram of a color pattern classification unit of the upper body posture and hand shape detection apparatus according to another embodiment.

Referring to FIG. 5, the color pattern classifier 405 includes a color detector 415, a pattern detector 455, and a standard human model storage 465.

The color detector 415 includes a color corrector 420, a difference image calculator 425, a skin color detector 435, and a shape filter 445.

The color corrector 420 corrects the color from the preprocessed image. For example, if the image is entirely red, the red component may be lowered. Alternatively, the color corrector 420 may adjust a ratio of R (Red), G (Green), and B (Blue) components of the image.

The difference image calculator 425 calculates a difference image that is a difference between the current image and the previous image. The difference image is used by the skin color detector 435 to detect the skin color from the corrected image.

In detail, the difference image calculator 425 calculates the difference image and the motion vector by subtracting the previous image from the current image. The difference image is an image obtained by subtracting the previous image from the current image. The difference image is an image of a changed region (or a moved region), and the motion vector includes information indicating the changed region (or a moved region), for example, a direction and a direction for change or movement. Indicates the size. Since the object moved in the current image and the previous image is likely to be mainly a human image, the difference image may be mainly an image of a person, and the difference image mainly includes a human skin color region. That is, the motion vector, which is information representing the difference image, has information on the color of the human skin. Accordingly, the skin color detector 435 may recognize the skin color by analyzing the color of the changed region (or the moved region) by using the difference image or the motion vector, and detect the skin color from the corrected image by using the same. In this case, the skin color is detected from the image by using the skin color of the person who is actually photographed, without detecting the skin color from the image by using the detection skin color having a predetermined brightness and saturation, thereby obtaining more accurate and reliable results. have.

The shape filter 445 determines the shape of the region having the skin color through filtering and outputs color information corresponding thereto. For example, the shape filter 445 may determine the shape of an area having a skin color by determining an edge of an area having a skin color output from the skin color detector 435 through shape filtering.

Next, the pattern detecting unit 455 detects pattern information, for example, information about the position and shape of each part of the body, from the preprocessed image using the standard human model stored in the standard human model storage unit 465. The ratio of the human body, the shape of each part, and the location of each part tend to be constant according to gender and age, and the standardized human body model is standardized. For example, the standard human model may be a sample of body proportions, shapes and positions of each part that are common to gender and age, and the standard human model is used to detect information about the location and shape of each part of the body from an image. More accurate detection.

Those skilled in the art will appreciate that the present invention can be embodied in other specific forms without changing the technical spirit or essential features of the present invention. For example, a program for realizing the control method of the present invention may be implemented in various forms such as a recording medium in which a program is recorded. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the claims and their equivalents should be construed as being included in the scope of the present invention.

1 is a block diagram showing an upper body posture and hand shape detection apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a stereo vision calculator of FIG. 1.

3 is a block diagram illustrating a color pattern classification unit of FIG. 1.

Figure 4 is an exemplary view showing the upper body posture and hand detection results.

5 is a block diagram illustrating an upper body posture and hand shape detection apparatus according to another embodiment of the present invention.

(Explanation of symbols for the main parts of the drawing)

10: upper body posture and hand detection device 101 ~ 103: camera

200: image data preprocessor 300: stereo vision calculation unit

400: color pattern classification unit 500: body part classification unit

610: face position detection unit 620: trunk position detection unit

630: shoulder position detection unit 640: elbow position detection unit

650: hand position detection unit 710: upper body posture recognition unit

720: hand recognition unit 800: application interface unit

Claims (18)

A stereo vision calculator configured to calculate 3D distance information from images provided from at least two cameras; A color pattern classification unit for classifying colors and patterns from the image and outputting color information and pattern information; A plurality of position detectors extracting three-dimensional position information for each object by using the three-dimensional distance information, the color information, and the pattern information; Upper body posture recognition unit for recognizing the upper body posture using the three-dimensional position information for each object; And Upper body posture and hand shape detection apparatus comprising a hand shape recognition unit for recognizing the shape of the hand using the hand position information of the three-dimensional position information for each object. The method of claim 1, And an image data preprocessor configured to perform at least one of noise reduction, color correction, brightness correction, and screen distortion correction on the image input from each of the at least two cameras. The method of claim 1, The subject includes a face, a torso, a shoulder, an elbow and a hand, The plurality of position detectors include a face position detector, a body position detector, a shoulder position detector, an elbow position detector, and a hand position detector. The method of claim 3, wherein The upper body posture recognition unit recognizes the upper body posture by reconstructing the upper body of the person using the position information of the face, the body, the shoulders, the elbow and the hand. The method of claim 1, wherein the color pattern classification unit A color detector for detecting color information from the image; Upper body posture and hand shape detection apparatus comprising a pattern detection unit for detecting pattern information from the image. The method of claim 5, wherein the color detection unit A color corrector for correcting the color of the image; A difference image calculating unit calculating a difference image that is a difference between the image and the previous image; A skin color detector which detects an area of the skin color by using the color corrected image and the difference image; And a morphological filter configured to determine a shape of the skin color region through filtering and output the color information corresponding thereto. The method of claim 5, The color pattern classification unit further includes a standard human model storage unit for storing a standard human model, The pattern detecting unit detects a pattern for each body part from the image using the standard human model and outputs the pattern information corresponding to the upper body posture and hand shape detection apparatus. The stereo vision calculation unit of claim 1, wherein the stereo vision calculation unit A stereo vision preprocessor for preprocessing the images provided from the at least two cameras; A stereo matching unit for extracting distance information from the preprocessed image; Upper body posture and hand detection device comprising a stereo vision post-processing unit for post-processing the distance information. A stereo vision calculator configured to calculate 3D distance information from images provided from at least two cameras; A color detector detecting an area having a skin color from the image; A pattern detector detecting a pattern of each body part from the image; A position detector configured to classify each body part from the image and detect the position of each body part by using the 3D distance information, the region of the skin color, and the pattern for each body part; Upper body posture recognition unit for recognizing the upper body posture using the position of each body part; Upper body posture and hand shape detection apparatus comprising a hand shape recognition unit for recognizing the shape of the hand using the hand position information of the three-dimensional position information for each object. The method of claim 9, wherein the color detector A color corrector for correcting the color of the image; A skin color detector which detects an area of the skin color from the color-corrected image; Upper body and hand detection device comprising a morphological filter for determining the edge (edge) of the region of the skin color through shape filtering. The method of claim 9, wherein the pattern detection unit Upper body posture and hand-shaped detection device for detecting the pattern by the body part by comparing the image with the predetermined pattern of each body part pattern. The method of claim 9, The body part comprises a face, a torso, a shoulder, an elbow and a hand, The upper body posture recognition unit recognizes the upper body posture by reconstructing the upper body of the person using the position information of the face, the body, the shoulders, the elbow and the hand. The method of claim 9, Upper body posture and hand shape detection device further comprising an interface for transmitting the information about the upper body posture and the shape of the hand to the outside. Calculating three-dimensional distance information from images provided from at least two cameras; Detecting an area having a skin color from the image; Detecting patterns of body parts from the image; Detecting the position of each body part using the three-dimensional distance information, the region of the skin color, and the pattern for each body part; Recognizing the upper body posture using the position of each body part; And Recognizing the shape of the hand using the hand position information of the three-dimensional position information for each object comprising the upper body posture and hand shape detection method. 15. The method of claim 14, And an image data preprocessing step of performing at least one of noise reduction, color correction, brightness correction, and screen distortion correction on the image input from each of the at least two cameras. The method of claim 14, wherein the detecting of the region having the skin color is performed. Correcting the color of the image; Calculating a difference image that is a difference between the image and the previous image; Detecting an area of skin color by using the color corrected image and the difference image; And determining an edge of the region of the skin color through morphological filtering. 15. The method of claim 14, The detecting of the position of each body part may include classifying the body parts from the image using the three-dimensional distance information, the region of the skin color, and the pattern for each body part. Detection method. 15. The method of claim 14, The body part comprises a face, a torso, a shoulder, an elbow and a hand, Recognizing the upper body posture is a step of recognizing the upper body posture by reconstructing the upper body of the person by using the position information of the face, the body, the shoulders, the elbow and the hand.

Images