KR101741671B1 - Play Presentation Device and Method - Google Patents

Abstract

The present invention relates to a 3D camera-based presentation device and method for performing a slide show operation through motion recognition, and a special camera and an up-sampling technique for performing presentation alone more efficiently. In order to achieve the above object, the present invention provides an image processing method, including image collection for photographing a hand gesture of a person in a presentation for one or two or more images for a predetermined period of time, image correction for correcting the photographed image through calibration, Extracting a target to be extracted, resolving the resolution of the extracted hand region image, generating a command behavior to combine the discriminated one or two hand region images, and converting the generated command action into a command signal Generating a command signal, and performing an instruction to execute the command in accordance with the command signal. As described above, according to the present invention, it is possible to provide more improved motion recognition. Such a superior motion recognition is not limited to the presentation device but can be operated as a convenient tool that can manipulate all other objects only by human actions without any tools. Which can be applied as a useful technique.

Description

[0001] The present invention relates to a 3D presentation-based presentation apparatus and method,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to a 3D motion-based presentation apparatus and method, and more particularly, to an apparatus and method for efficiently and independently presenting a slide show control through motion recognition of a person performing a presentation. In particular, it is a 3D motion-based presentation device and method that provides improved perception of motion of a person performing a presentation.

Typically, existing presentation tools manipulate slideshows with a computer mouse or keyboard, or use smart pointers. In this case, the accuracy of the keyboard is high, but there is a disadvantage that the assistant is needed. In the case of the smart pointer, the pointing position can be accurately displayed, but the command system that can be operated is limited. In addition, studies have been conducted to detect and determine a specific body part and motion of a person using a color camera and to use it in a presentation. However, there is a limitation in extracting objects of interest and providing various presentation controls in an accurate and real time through a color camera.

In recent years, research on computer control using pose tracking and gesture recognition using real - time depth camera has been conducted with the development of depth sensing technology. In particular, the Xtion series of Microsoft's Kinect and Asus launched at the end of 2010 has been widely used, and a lot of research on presentation control software using depth camera has been done.

 There are two types of depth cameras: Time of Flight (TOF) and stereo matching. Among these methods, the stereo matching method using the phase difference between the irradiation wave and the reflected wave is most used. In the Kinect and Xtion series, there is a separation between the transmitter that emits light and the receiver that senses the reflected wave. Accordingly, the infrared light having a specific pattern is irradiated, and the depth information is extracted by stereo matching with the wave reflected from the object. However, there is a problem that an error occurs on the left or right side of the object regardless of the wave.

Korean Patent Registration No. 1407249 (2014.06.09)

Accordingly, the present invention has been made to solve the above problems of the related art, and it is an object of the present invention to provide an apparatus and a method for efficiently performing announcement by a slide show manipulation through motion recognition alone have.

According to an aspect of the present invention, there is provided a 3D motion-based presentation apparatus including an image collection unit, an image correction unit, a target extraction unit, a resolution determination unit, a command behavior generation unit, a command signal generation unit, And an execution unit. The image capturing unit captures one or two or more images of a person's hand gesture for a predetermined period of time, and the image correcting unit corrects the photographed image through calibration, and the target extracting unit extracts, from the corrected image, The resolution discrimination unit discriminates the resolution of the extracted hand region image, and the command behavior generating unit combines the discriminated one or two hand region images to generate a command action, and the command signal generating unit generates the command action, And the instruction execution unit executes the instruction according to the instruction signal.

In the present invention, the image correcting unit converts the captured hand motion image into OpenCV data for correcting the captured hand motion image.

로부터 일정 범위의 depth 값을 가지는

만을 검출한다. The target extraction unit uses the binarization conditional expression defined in Equation (1) to extract the hand region image

Having a certain range of depth values

.

[Equation 1]

는 현재 탐색하는 이미지 픽셀의 depth값,

는 손 중심점 픽셀의 depth 값)(

The depth value of the currently scanned image pixel,

Is the depth value of the hand center pixel)

To detect the center point of the hand region, the L_max value is set by setting each pixel value to L using the distance conversion algorithm defined by Equation 2, and then the hand center point, wrist detection point, and wrist detection point are extracted by dividing L by L_max .

&Quot; (2) "

L / L_max

(L is the distance from the hand center point to the nearest outline, L_max is the largest value of L for all pixels)

The resolution discriminator performs detailed element recognition of the hand region when the hand region image has a high resolution of 1920x1080 or more capable of recognizing detailed elements. At this time, the detailed element recognition of the hand region image is performed by connecting a straight line between the outermost points of the hand region and a straight line between the two points from the Convex Hull and the Convex Hull, The Convexity Defect, which is the farthest from the outline, is used.

The resolution determining unit generates a high resolution hand region image of 1920x1080 or more when the hand region image is a low resolution of less than 1920x1080 in which detailed elements can not be recognized. In this case, when an error occurs in generation of a high resolution due to a low resolution of the hand region image, a YCrCb image transformation defined by Equation (3) is further added to set a color value that best represents the Asian skin color do.

&Quot; (3) "

&&

&&

는 휘도와 적색 성분,

는 휘도와 청색 성분, minCr은 130, maxCr은 160, minCb는 110, maxCb는 140)(

The luminance and the red component,

The luminance and blue components, minCr is 130, maxCr is 160, minCb is 110, and maxCb is 140)

In order to achieve the above object, a 3D motion-based presentation method according to the technical idea of the present invention includes an image collection step, an image correction step, a target extraction step, a resolution determination step, a command behavior generation step, The present invention is characterized in its technical construction. The image capturing step captures the hand gesture of the person presenting the presentation in one or two or more images for a predetermined period of time. In the image correction step, the photographed image is calibrated through calibration, Extracting a corresponding image, determining a resolution of the extracted hand area image, and generating a command behavior by combining one or two or more discriminated hand area images, and the command signal generating step Converts the generated command behavior into a command signal, and the command execution step executes the command according to the command signal.

In the present invention, the image correcting step converts the captured hand motion image into OpenCV data for correcting the captured hand motion image.

로부터 일정 범위의 depth 값을 가지는

만을 검출한다. The target extraction step uses the binarization conditional expression defined in Equation (1) to extract the hand region image

Having a certain range of depth values

.

[Equation 1]

는 현재 탐색하는 이미지 픽셀의 depth값,

는 손 중심점 픽셀의 depth 값)(

The depth value of the currently scanned image pixel,

Is the depth value of the hand center pixel)

To detect the center point of the hand region, the L_max value is set by setting each pixel value to L using the distance conversion algorithm defined by Equation 2, and then the hand center point, wrist detection point, and wrist detection point are extracted by dividing L by L_max .

&Quot; (2) "

L / L_max

(L is the distance from the hand center point to the nearest outline, L_max is the largest value of L for all pixels)

The resolution determination step performs detailed element recognition of the hand region when the hand region image has a high resolution of 1920x1080 or more, which allows detailed elements to be recognized. At this time, the detailed element recognition of the hand region image is performed by connecting a straight line between the outermost points of the hand region and a straight line between the two points from the Convex Hull and the Convex Hull, The Convexity Defect, which is the farthest from the outline, is used.

The resolution determination step generates a high resolution hand region image of 1920x1080 or more when the hand region image is a low resolution of less than 1920x1080 in which detailed elements can not be recognized. In this case, when an error occurs in generation of a high resolution due to a low resolution of the hand region image, a YCrCb image transformation defined by Equation (3) is further added to set a color value that best represents the Asian skin color do.

&Quot; (3) "

&&

&&

는 휘도와 적색 성분,

는 휘도와 청색 성분, minCr은 130, maxCr은 160, minCb는 110, maxCb는 140)(

The luminance and the red component,

The luminance and blue components, minCr is 130, maxCr is 160, minCb is 110, and maxCb is 140)

According to the 3D motion-based presentation apparatus and method according to the present invention, the following effects can be obtained.

First, a presentation tool using motion recognition can provide improved motion recognition.

Second, excellent motion recognition is not limited to presentation devices, but can be a useful technique applicable to IoT, which is currently being used as a convenient tool to manipulate all other objects only by human actions without any tools.

1 is a block diagram of a presentation device based on 3D operation according to an embodiment of the present invention;
2 is a flowchart of a 3D motion-based presentation method according to an embodiment of the present invention
Figure 3 is an illustration of a color image and a depth image before calibrating according to an embodiment of the present invention.
Figure 4 is an exemplary diagram of a binned preprocessed image according to one embodiment of the present invention.
5 is a diagram illustrating an example of a center-of-gravity point and a center-of-hand point detection according to an embodiment of the present invention
6 is an illustration of a wrist removal method according to an embodiment of the present invention.
Figure 7 is an illustration of hand region layer separation in accordance with one embodiment of the present invention.
8 is an illustration of an up-sampling result (left: up-sampling depth image, right: RGB image) according to an embodiment of the present invention;
FIG. 9 is an illustration of a Convex Hull and a Convexity Defect according to an embodiment of the present invention. FIG.
FIG. 10 is a diagram illustrating an example of skin detection result using a YCrCb channel according to an embodiment of the present invention. FIG.

A 3D motion-based presentation apparatus and method according to embodiments of the present invention will be described in detail with reference to the accompanying drawings. The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It is to be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing. In the accompanying drawings, the dimensions of the structures are enlarged to illustrate the present invention, and are actually shown in a smaller scale than the actual dimensions in order to understand the schematic structure.

Also, the terms first and second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. On the other hand, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

1 and 2, a 3D motion-based presentation apparatus according to an exemplary embodiment of the present invention includes an image collection unit 100, an image correction unit 200, a target extraction unit 300, a resolution determination unit 400 A command behavior generating unit 500, a command signal generating unit 600, and an instruction performing unit 700. [

The image capturing unit 100 captures a hand gesture of a person who performs a presentation in one or two or more images for a predetermined period of time.

The image correcting unit 200 corrects the photographed image of the hand gesture through calibration. At this time, the image data is converted into OpenCV data for correcting the photographed hand motion image of the image correcting unit 200.

로부터 일정 범위의 depth 값을 가지는

만을 검출한다.The target extracting unit 300 extracts an image corresponding to the hand region from the corrected image. At this time, the object extracting unit 300 extracts only the hand region image from the hand motion image using the binarization conditional expression defined by Equation (1)

Having a certain range of depth values

.

[Equation 1]

는 현재 탐색하는 이미지 픽셀의 depth값,

는 손 중심점 픽셀의 depth값)(

The depth value of the currently scanned image pixel,

Is the depth value of the hand center pixel)

 The center point of the hand region is determined by finding the L_max value after setting each pixel value to L using the distance transformation algorithm defined by Equation (2), and dividing L by L_max.

&Quot; (2) "

L / L_max

(L is the distance from the hand center point to the nearest outline, and L-max is the largest value among L of all pixels)

The resolution determination unit 400 determines the resolution of the extracted hand region image.

The resolution determination unit 400 performs detailed element recognition of the hand region when the hand region image has a high resolution of 1920x1080 or more that allows detailed elements to be recognized. At this time, the detailed element recognition of the hand region image is performed by connecting a straight line between the outermost points of the hand region and a straight line between the two points from the Convex Hull and the Convex Hull, The Convexity Defect, which is the farthest from the outline, is used.

The resolution determination unit 400 generates a 1920x1080 high resolution hand region image when the hand region image is a low resolution of less than 1920x1080 in which detailed elements can not be recognized. In this case, when an error occurs in generation of a high resolution due to a low resolution of the hand region image, a YCrCb image transformation defined by Equation (3) is further added to set a color value that best represents the Asian skin color do.

&Quot; (3) "

&&

&&

는 휘도와 적색 성분,

는 휘도와 청색 성분, minCr은 130, maxCr은 160, minCb는 110, maxCb는 140)(

The luminance and the red component,

The luminance and blue components, minCr is 130, maxCr is 160, minCb is 110, and maxCb is 140)

The command behavior generating unit 500 combines the discriminated one or two hand region images to generate command actions.

 For example, a five-finger pinch gesture is a command to open a presentation device screen to start a presentation, a finger pin manipulation is a command to flip through a presentation slide, a two finger pinch is a presentation The hand movements of the punch state in which the slide is moved backward and the fingers are not extended are commands to turn off the presentation device screen to end presentation of the presentation.

The command signal generation unit 600 converts the generated command action into a command signal.

The instruction execution unit 700 executes the instruction according to the instruction signal.

Referring to FIG. 3, Kinect, which is a type in which a color camera and a depth camera are integrated into one device, is used as an example according to an embodiment of the present invention. Since both cameras are physically located at different positions, the image 201 taken by both cameras is distorted. In order to correct this distortion, it is known that calibration is performed and the corrected image 202 appears.

Referring to FIG. 4, an exemplary embodiment according to the present invention is a preprocessed image binarized with a hand center point Depth value provided by a Kinect and a Range value set by a program.

Referring to FIG. 5, an exemplary embodiment according to the present invention is a hand center point, a wrist point detection point, and a wrist point detection point using a distance transformation algorithm defined by Equation (2) in a binarized hand region candidate image. In particular, the distance conversion algorithm finds the L_max value after setting each pixel value with L, and divides L by L_max.

&Quot; (2) "

L / L_max

(L is the distance from the hand center point to the nearest outline, and L-max is the largest value among L of all pixels)

In the case of the hand center point 301, the circle 302 centered on the center of the hand is the L_max pixel set in the previous example, and the distance from the center line of the hand center point to the nearest outline is a radius, and another circle 303, Has a radius obtained by multiplying the radius of the circle 302 centered on the hand center by a constant value of 1.5.

In the case of detecting the wrist point, the other circle (303) centered on the center of the hand is divided into 360 by 1 degree, and when non-zero values appear consecutively 8 times (8 degrees) in the distance conversion matrix, Is detected as the second wrist candidate (304). The wrist detects the second wrist candidate 304 closest to the straight line connecting the center of hand 301 and the elbow point 305 as the final wrist candidate.

6, the smallest value among the neighboring pixels of the secondary wrist candidate 304 detected previously is set as the starting point 1, and the smallest pixel among the four pixels on the opposite side of the point 1 is set as the next starting point As the process is repeated, the starting points that are searched from the point where the starting point becomes 0, that is, the point which is not the hand candidate region, are detected by the wrists and the arm region is removed.

Referring to FIG. 7, when the up-sampling is performed, when the background and the two depth layers of the finger are blended at the boundary of the depth, the result is a case where the boundary is blurred.

8, since the resolution of the Kinect color image is about four times larger than the resolution of the depth image, the information of the color image is obtained by up-sampling (up / (Up-sampling) can be performed.

Referring to FIG. 9, a finger detection algorithm is shown. In FIG. 9A, a Convex Hull which is an algorithm that all points include by connecting the outermost points of a hand region in a straight line is shown. In this case, the inside of the outline is excluded because it includes the outline, and the Convex Hull is performed only for the outline detected earlier. In FIG. 9B, a Convexity Defect is used, which is a point far from the outermost line in a straight line between two points from the Convex Hull. At this time, the red dot is Convexity Defect and the green dot is Convex Hull.

Referring to FIG. 10, an example of detecting a color image-based hand region is illustrated. In the up-sampling step, a lot of errors are generated due to a problem of low resolution. Therefore, And realizes the use of hand region detection. For the hand region extraction using the skin color, the YCrCb image transformation defined by Equation 3 is further added to set the color value that best represents the Asian skin color.

&Quot; (3) "

&&

&&

는 휘도와 적색 성분,

는 휘도와 청색 성분, minCr은 130, maxCr은 160, minCb는 110, maxCb는 140)(

The luminance and the red component,

The luminance and blue components, minCr is 130, maxCr is 160, minCb is 110, and maxCb is 140)

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It is clear that the present invention can be suitably modified and applied in the same manner. Therefore, the above description does not limit the scope of the present invention, which is defined by the limitations of the following claims.

100: image collecting unit 200: image correcting unit
201: Images from two cameras 202: Corrected images
300: target extraction unit 301: hand center point
302: Circle centering on the hand center point 303: Other circle centered on the hand center point 400: Resolution discrimination unit
500: command behavior generating unit 600: command signal generating unit
700: Command execution unit

Claims (14)

A 3D motion-based presentation device,
An image collecting unit for photographing a hand gesture of a person making a presentation in one or two or more images for a predetermined period of time;
An image correcting unit for correcting the photographed image through calibration;
A target extracting unit for extracting an image corresponding to a hand region in the corrected image;
A resolution discrimination unit for discriminating a resolution of the extracted hand region image;
A command behavior generating unit for generating command actions by combining the determined one or two or more hand region images;
A command signal generating unit for converting the generated command action into a command signal; And an instruction execution unit for executing the instruction according to the instruction signal,
The hand region image extraction of the object extracting unit may be performed using a binarization conditional expression defined by Equation (1)

Having a certain range of depth values

Wherein the 3D motion-based presentation device detects only the 3D motion-based presentation device.
[Equation 1]

(

The depth value of the currently scanned image pixel,

Is the depth value of the hand center pixel) The image processing apparatus according to claim 1,
And converts the data into OpenCV data for correcting the photographed hand motion image. delete 2. The method of claim 1,
Wherein the center point detection is performed by setting each pixel value to L using a distance transformation algorithm defined by Equation (2) to find L_max, and then dividing L by L_max.
&Quot; (2) "
L / L_max
(L is the distance from the hand center point to the nearest outline, L_max is the largest value of L for all pixels) The apparatus according to claim 1,
Performing detailed element recognition of the hand region when the hand region image has a high resolution of 1920x1080 or more capable of recognizing detailed elements,
And generates a high resolution hand region image of 1920x1080 or more when the hand region image is a low resolution of less than 1920x1080 in which detailed elements can not be recognized. 6. The method of claim 5,
The detailed element recognition of the hand region image includes a convex hull including all the points connecting the outermost points of the hand region in a straight line,
And a Convexity Defect which is a point farthest from the outline in a straight line between the two points from the Convex Hull is used. 6. The method of claim 5, wherein in the hand region image,
When an error occurs in generation of a high resolution due to a low resolution, a YCrCb image transformation defined by Equation (3) is further added to set a color value that best represents the skin color of an Asian person Based presentation device.
&Quot; (3) "

&&

(

The luminance and the red component,

The luminance and blue components, minCr is 130, maxCr is 160, minCb is 110, and maxCb is 140) A 3D motion-based presentation method,
An image collecting step of photographing a hand gesture of a person making a presentation in one or two or more images for a predetermined period of time;
An image correction step of correcting the photographed image through calibration;
Extracting an image corresponding to a hand region in the corrected image;
Determining a resolution of the extracted hand region image;
A command behavior generating step of generating a command action by combining the determined one or two or more hand region images;
A command signal generating step of converting the generated command action into a command signal; And an instruction execution step of executing the instruction according to the instruction signal,
The hand region image extraction in the subject extraction step may be performed using a binarization conditional expression defined by Equation (1)

Having a certain range of depth values

Wherein the 3D motion-based presentation method comprises the steps of:
[Equation 1]

(

The depth value of the currently scanned image pixel,

Is the depth value of the hand center pixel) 9. The method according to claim 8,
And converting the image data into OpenCV data for correcting the captured hand motion image. delete 9. The method of claim 8,
Wherein the center point detection is performed by setting each pixel value to L using a distance conversion algorithm defined by Equation (2) to find L_max, and then dividing L by L_max.
&Quot; (2) "
L / L_max
(L is the distance from the hand center point to the nearest outline, L_max is the largest value of L for all pixels) 9. The method of claim 8,
Performing detailed element recognition of the hand region when the hand region image has a high resolution of 1920x1080 or more capable of recognizing detailed elements,
Wherein a high resolution hand region image of 1920x1080 or more is generated when the hand region image is a low resolution of less than 1920x1080 in which detailed elements can not be recognized. 13. The method of claim 12,
The detailed element recognition of the hand region image includes a convex hull including all the points connecting the outermost points of the hand region in a straight line,
And a Convexity Defect which is a point farthest from the outline in a straight line between the two points from the Convex Hull is used. 13. The method of claim 12, wherein in the hand region image,
When an error occurs in generation of a high resolution due to a low resolution, a YCrCb image transformation defined by Equation (3) is further added to set a color value that best represents the skin color of an Asian person Based presentation method.
&Quot; (3) "

&&

(

The luminance and the red component,

The luminance and blue components, minCr is 130, maxCr is 160, minCb is 110, and maxCb is 140)

Images