KR101595293B1 - space touch control system based on a depth sensor - Google Patents

Abstract

The present invention relates to an apparatus and method for controlling a spatial touch using a depth sensor, the apparatus comprising: a hand candidate region detecting unit for detecting a hand candidate region in an image acquired by a depth sensor; The width of the two lines located within the hand candidate region is calculated at a predetermined interval below the predetermined index and the hand candidate region is calculated based on the width of the two lines, A hand region judging unit for judging whether the region is a region; A detection position determination unit for determining a position of the detection area selected by the hand area determination unit; A thumb state determining unit for determining whether the thumb is folded in the hand region determined by the hand region determining unit; And a touch control unit for controlling the pointer on the scroll according to the detection position determined by the detection position determination unit and the thumb folding unit determined by the thumb state determination unit.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a space-

More particularly, the present invention relates to a space touch control apparatus and method, and more particularly to a space touch control apparatus and method that recognize a user's hand using a depth sensor (depth camera) without a separate input device, And more particularly, to a sensor-based spatial-touch-control device and method capable of controlling a three-dimensional pointer on a screen of a display device.

Recently, when a person's hand or object touches a character or a specific position displayed on a screen (screen) without using a keyboard, the position can be grasped and a direct input can be received from the screen Touch screen is widely used.

The touch screen displays character or picture information in advance, so that it can easily grasp the function to be selected by the user. It can be applied variously to subway, department store, bank, etc., have.

A conventional touch screen recognizes user input by adding a touch panel to a screen of a general monitor and changing characteristics of the touch panel when a finger or other object touches a predetermined area.

In addition, the conventional touch screen divides the entire screen into a two-dimensional lattice pattern and analyzes the contact position. This is because the touch panel to which the capacitance, the ultrasonic wave, the infrared ray, the resistive film, Use changing principles.

That is, since the conventional touch screen is configured in a two-dimensional form in which the display screen and the touch panel are located on the same plane, detection of the touch of the three-dimensional menu icon projected in space is not supported.

Accordingly, a space touch recognition technology has been proposed. Conventional space touch recognition technologies include recognition by an infrared ray projection image, recognition by a marker, and recognition of other contact type sensors.

However, the recognition using the infrared projection image uses a simple binary image and is sensitive to illumination, so that gesture recognition is limited, and recognition using a marker, a data glove, and the like is troublesome to use.

It is therefore an object of the present invention to provide a screen control input on a screen at a remote location using a depth camera, And a depth sensor-based spatial touch control apparatus and method capable of controlling the apparatus.

According to an aspect of the present invention, there is provided an apparatus for controlling a spatial depth based on a depth sensor, including: a hand candidate region detecting unit for detecting a hand candidate region in an image acquired by a depth sensor; The width of the two lines located within the hand candidate region is calculated at a predetermined interval below the predetermined index and the hand candidate region is calculated based on the width of the two lines, A hand region judging unit for judging whether the region is a region; A detection position determination unit for determining a position of the detection area selected by the hand area determination unit; A thumb state determining unit for determining whether the thumb is folded in the hand region determined by the hand region determining unit; And a touch control unit for controlling the pointer on the screen according to the detection position determined by the detection position determination unit and the thumb folding unit determined by the thumb state determination unit.

At this time, the hand candidate region detecting unit finds a point closest to the depth sensor in the image obtained by the depth sensor, and binarizes a region within a predetermined distance based on the detected depth candidate, thereby detecting a hand candidate region.

The hand candidate region detecting unit detects the largest region among the regions obtained by performing labeling on the binarized region as a hand candidate region.

The hand region determination unit compares the width of each of the two lines located within the hand candidate region at a predetermined interval below the selected index with the width and the maximum width of another line measured in the hand candidate region, It is judged whether it is a hand area.

The hand region determination unit may determine whether the hand candidate region is a hand region or not based on whether or not the relative ratio of the width of each of the two lines to the width of the other line and the relative ratio of the width and the maximum width of each of the two lines satisfy a predetermined condition .

At this time, the hand region determining unit determines that the hand candidate region is a hand region when each of the widths of the two lines is smaller than 1.5 times the width of the other lines and smaller than 1/2 of the maximum width.

On the other hand, the thumb state determination unit sets the edges of the detection axis and the hand region, and determines whether or not the thumb is folded by using the edges of the detected axis and the hand region.

The thumb state determination unit sets the detection axis by connecting the centers of the widths of the two calculated lines to the hand region determination unit.

The thumb state determination unit scans the left or right side by line on the basis of the set sensing axis and determines that the thumb is open if the number of straight lines having two or more edges is more than a certain number and determines that the thumb is folded .

At this time, the thumb state determining unit scans to the right when the hand is the left hand and to the left when the right hand.

According to another aspect of the present invention, there is provided a method for controlling a spatial depth based on a depth sensor, comprising: obtaining a hand image using a depth sensor and detecting a hand candidate region based on the obtained image; ; When the hand candidate region is detected, the detection is selected in the hand candidate region, and the widths of two lines positioned within the hand candidate region are calculated at predetermined intervals below the selected detection region. Based on this, ; Judging the position of the index finger and judging whether the thumb is folded or not if it is determined that the hand candidate area is the hand area; And controlling the pointer on the screen according to the determined detection position and whether the thumb is folded or not.

In the detecting of the hand candidate region, the hand candidate region may be detected by finding a point closest to the depth sensor in the obtained image, and binarizing an area within a predetermined distance based on the detected position.

The hand candidate region may be set as the largest region among the regions obtained by performing labeling on the binarized region.

Meanwhile, in the step of determining whether the hand region is included in the hand candidate region, the width of each of the two lines located within the hand candidate region is compared with the width and the maximum width of the other lines measured in the hand candidate region, And judges whether the hand candidate region is a hand region.

At this time, in the step of determining whether the hand region is the hand region, it is preferable that the hand region includes at least one of a relative ratio of a width of each of the two lines to a width of another line, And the like.

In the step of determining whether the hand region is the hand region, if the widths of the two lines are smaller than 1.5 times the width of the other lines and smaller than 1/2 of the maximum width, it can be determined that the hand candidate region is the hand region.

The step of determining whether or not the thumb is folded may include the step of setting the edges of the sensing axis and the hand area and determining whether the thumb is folded using the edges of the sensing axis and the hand area.

At this time, in the step of determining whether the thumb is folded, the detection axis may be set by connecting the centers of the widths of the two lines calculated in the step of determining whether the finger is the hand area.

In the step of determining whether or not the thumb is folded, it is determined that the thumb is scanned when the number of straight lines having two or more edges is more than a predetermined number, , It can be determined that the thumb is folded.

At this time, in the step of determining whether the thumb is folded, it is possible to scan to the right when the hand is the left hand and scan to the left when the right hand.

According to the embodiment of the present invention, the image of the hand used as the pointing target can be acquired using the depth sensor, and the pointer on the screen can be controlled according to the state of the finger and the thumb, Can be controlled.

Further, since the hand is used as an instruction target for controlling the pointer, the non-contact type control can be performed without a separate control device such as a remote controller.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram illustrating a state of a touch control using a depth sensor-based spatial touch control apparatus according to an embodiment of the present invention. FIG.
BACKGROUND OF THE INVENTION Field of the Invention [0001] The present invention relates to a depth-sensor-
FIG. 3 is a parameter setting state diagram used when the hand region determination unit shown in FIG. 2 determines a hand candidate region as a hand region.
4 is a state diagram showing a method of setting the detection axis by the detection state determination section shown in Fig.
5 is a state diagram showing a method of determining the thumb state by the detection state determination unit shown in FIG.
6 is a flowchart illustrating an operation of a depth-sensor-based spatial touch control method according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. To fully disclose the scope of the invention to a person skilled in the art, and the invention is only defined by the scope of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components throughout the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

The present invention relates to a space touch control system and method capable of providing a screen control input on a screen at a long distance using a user's hand, in particular a thumb and index finger, so that a device can be controlled remotely without requiring a separate input device I suggest.

FIG. 1 is a diagram illustrating a touch control using a depth sensor-based spatial touch control apparatus according to an exemplary embodiment of the present invention. FIG. 2 is a block diagram showing a configuration of a space touch control based on a depth sensor according to an embodiment of the present invention. FIG.

FIG. 3 is a diagram showing a parameter setting state used when the hand region judging section shown in FIG. 2 judges a hand candidate region as a hand region, FIG. 4 is a state diagram showing a method of setting a detection axis by the detection state determining section shown in FIG. And FIG. 5 is a state diagram showing a method of determining the thumb state by the detection state determination unit shown in FIG.

The configuration and operation of the depth-sensor based spatial-tactile control device according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 5. FIG.

1, a depth sensor-based spatial touch control apparatus 100 according to an exemplary embodiment of the present invention includes a display device 100 including a screen S (e.g., a smart TV, a tablet PC, Product) to recognize the user's hand movements using a depth sensor (depth camera), and then control the position of the pointer P on the screen or control the click operation according to the recognized hand movements.

The specific configuration and operation of the depth-sensor-based spatial-touch controller 100 according to the present invention will be described in detail with reference to FIG.

2, the spatial touch control apparatus 100 includes a depth sensor 110, a hand candidate region detection unit 120, a hand region determination unit 130, a detection position determination unit 140, a thumb state determination unit 150 and a touch control unit 160. [

The depth sensor 110 is an apparatus for acquiring an image of an instruction means such as a hand, and may be, for example, a 3D camera, which is an image sensor capable of sensing the depth of an image.

The hand candidate region detecting unit 120 finds a point (the latest point) closest to the depth sensor 110 in the image obtained by the depth sensor 110 and detects an area within a predetermined distance based on the latest point Binarized to detect a hand candidate region.

In this case, the binarization used for detecting the hand candidate region is widely used in the image field, and the colors distributed in various RGB values are represented by values of 0 and 1, A value exceeding 255 is used as a threshold value, and a value lower than 255 is used as a threshold value, and a detailed description thereof will be omitted.

On the other hand, the specific distance is preset as a distance to be regarded as a hand candidate region. For example, when 5 cm is set, an area included within 5 cm from the latest point is detected as a hand candidate region.

At this time, the hand candidate region detecting unit 120 detects the largest region among the regions obtained by performing labeling on the binarized region as a hand candidate region, and the image labeling technique is well-known, so a detailed description thereof will be omitted.

The hand region determination unit 130 selects a point at the top of the hand candidate region detected by the hand candidate region detection unit 120 based on the y axis and determines whether the hand candidate region is a hand region.

3, the hand area determination unit 130 determines the width (x axis thickness) W1 in the hand candidate region on the line L1 positioned below the first pixel in the y axis on the basis of the detection, The width (x axis thickness) W2 in the hand candidate region on the line L2 positioned two pixels below is calculated and the width (x axis thickness) per line in the hand candidate region is measured to calculate the maximum width Wm.

In this case, the number of times of width measurement in the hand candidate region is determined depending on how many pixels are formed on the y axis. For example, when the hand candidate region is composed of 80 pixels on the y axis, 80 widths are measured, . Here, the width (x axis thickness) can be defined as the number of pixels included in the hand candidate region in one line.

The hand region determination unit 130 determines that the hand candidate region is a hand region when each of the two widths W1 and W2 satisfies a predetermined condition and determines that the hand candidate region is noise if the predetermined condition is not satisfied, Wait until the candidate area is input.

In this case, the predetermined condition may be variously set in order to determine the hand candidate region as a hand region. Therefore, the hand region determination unit 130 may be configured to determine the hand region by using two widths W1, W2 are smaller than 1.5 times as large as the other widths and smaller than 1/2 of the maximum width Wm, it can be determined that the hand candidate region is a hand region.

The detection position determination unit 140 determines the position of the detection area selected by the hand area determination unit 130. [

The thumb state determining unit 150 determines the thumb state, and particularly determines whether the thumb is folded or not.

Since the thumb is located on the left or right side of the index finger, the thumb state determination unit 140 determines the thumb state using the detection axis.

4, the thumb state determining unit 150 sets the sensing axes by connecting the centers of the two widths W1 and W2 calculated by the hand region determining unit 130. In this case, as shown in FIG.

Further, the thumb state determining unit 150 sets the edge of the hand region after setting the detection axis, and determines the thumb state using the edge of the detected axis and the hand region.

5, the thumb state determining unit 150 makes a straight line orthogonal to the detection axis with the rim of the detection axis and the hand region set, moves leftward and rightward with respect to the detection axis, Or scan the right side.

At this time, when the right hand is used as the pointing means, the left side is scanned since the thumb is located on the left side of the index finger, and when the left hand is used as the pointing means, the thumb is located on the right side of the index finger, .

On the other hand, whether the left hand or the right hand is used as the pointing means can be preset by the user, and the thumb state determining unit 150 determines whether the left hand or the right hand And may be set to scan.

At this time, after the scan, the thumb state determining unit 150 determines that the thumb is opened when the number of lines having two or more edges is more than a certain number, and determines that the thumb is folded if the number is less than the predetermined number.

The touch control unit 160 controls the position of the pointer P on the screen S based on the position of the detection and the thumb state and selects the selected icon or button on which the pointer P is located.

At this time, the touch controller 160 controls the pointer P on the screen S based on the detection position determined by the detection position determiner 140 and the thumb state determined by the thumb state determiner 150, .

That is, in the embodiment of the present invention, the detection is used to control the position of the pointer P on the screen S, and the thumb serves as a selection button.

At this time, when the thumb is folded in the unfolded state, the touch controller 160 selects an object on which the pointer P is moved according to the control of the detection.

FIG. 6 is a flowchart illustrating an operation of a spatial touch control method based on a depth sensor according to the present invention.

Referring to FIG. 6, in order to control a pointer on a screen using the space touch control apparatus proposed by the present invention, when a user places his / her hand within the image capture range of the space touch control apparatus, The depth sensor 110 acquires an image of a hand (S110).

Then, based on the obtained image, the hand candidate region detection unit 120 detects a hand candidate region (S120). At this time, the depth sensor 110 is an image sensor capable of sensing the depth of an image, and the image obtained by the depth sensor 110 is a depth image.

The hand candidate region is detected by finding a current point in the acquired image and binarizing the predetermined distance region based on the latest point.

At this time, the largest region among the regions obtained by performing labeling on the binarized region is detected as the hand candidate region.

When the hand candidate region is detected as described above (S120), the hand region determination unit 130 selects the detection region in the hand candidate region (S130).

The widths W1 and W2 of the two lines positioned at a predetermined distance below the index finger are compared with the width and the maximum width Wm of the other lines measured in the hand candidate region, (S140).

At this time, the hand candidate region can be analyzed in pixel units, and the two lines are divided into a first line located under m pixels in the y-axis and a second line located under n (n> m) Line.

Therefore, the x axis thickness can be defined as the number of pixels included in the line in the hand candidate region.

On the other hand, the determination of the hand candidate region is based on the relative ratio between the widths W1 and W2 of the two lines and the widths of other lines measured in the hand candidate region, and the widths W1 and W2 and the maximum width Wm ) Satisfies a predetermined condition or not.

At this time, when the relative ratio satisfies the predetermined condition, it is determined that the hand candidate region is the hand region. If the relative ratio does not satisfy the predetermined condition, it is determined that the noise is present and the process waits until the next hand candidate region is input.

In this case, although it is not intended to limit the relative ratio in the present invention, for example, the widths W1 and W2 of the two lines may be 1.5 times And is smaller than 1/2 of the maximum width Wm, it can be set to judge that the hand candidate region is the hand region.

As a matter of course, the relative ratio may be set so that the two lines are different from each other.

If it is determined that the hand candidate region is the hand region as described above (S140-Yes), the detection position determination unit 140 determines the position of the detection, and the thumb state detection unit 150 determines whether the thumb is folded (S150 ).

At this time, whether or not the thumb is folded is determined by connecting the center of the width of the two lines to set the detection axis and setting the edge of the hand area. If the number of straight lines having two or more edges is more than a certain number, it is determined that the thumb is open. If it is less than the predetermined number, it is determined that the thumb is folded.

On the other hand, if the hand used as the pointing target is the left hand, the right side is scanned with respect to the sensing axis. If the hand is the right hand, the left side is scanned with respect to the sensing axis.

If the position of the index finger and the presence or absence of folding of the thumb are determined as described above (S150), the touch controller 160 controls the position of the pointer on the screen according to the position of the finger, and performs the selecting function according to the state of the thumb .

According to the embodiment of the present invention, the image of the hand used as the pointing target can be acquired using the depth sensor, and the pointer on the screen can be controlled according to the state of the finger and the thumb, Can be controlled.

Further, since the hand is used as an instruction target for controlling the pointer, the non-contact type control can be performed without a separate control device such as a remote controller.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. And various alternatives, modifications, and changes may be made within the scope of the invention.

Therefore, the embodiments described in the present invention and the accompanying drawings are intended to illustrate rather than limit the technical spirit of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments and accompanying drawings . The scope of protection of the present invention should be construed according to the claims, and all technical ideas within the scope of equivalents should be interpreted as being included in the scope of the present invention.

100: Space touch control device
110: Depth sensor
120: hand candidate region detection unit
130: hand area judgment unit
140: Detection position determination section
150: Thumb state determination unit
160: Touch control unit
P: Pointer
S: Screen

Claims (20)

A hand candidate region detection unit for detecting a hand candidate region in an image obtained by the depth sensor;
The width of the two lines located within the hand candidate region is calculated at a predetermined interval below the predetermined index and the hand candidate region is calculated based on the width of the two lines, A hand region judging unit for judging whether the region is a region;
A detection position determination unit for determining a position of the detection area selected by the hand area determination unit;
A thumb state determining unit for determining whether the thumb is folded in the hand region determined by the hand region determining unit; And
And a touch control unit for controlling the pointer on the screen in accordance with the detection position determined by the detection position determination unit and the thumb folding unit determined by the thumb state determination unit,
The hand region determination unit compares the width of each of the two lines located within the hand candidate region with a predetermined interval below the selected index to the width and the maximum width of another line measured in the hand candidate region, In addition,
The hand region judging section judges whether the hand candidate region is a hand region or not based on whether the relative ratio of the width of each of the two lines and the width of the other line and the relative ratio of the width and the maximum width of each of the two lines satisfy a predetermined condition Wherein the depth-sensor-based spatial-touch-control device is a depth-sensor-
The method according to claim 1,
Wherein the hand candidate region detection unit detects a hand candidate region by finding a point closest to the depth sensor in an image obtained by the depth sensor and binarizing an area within a predetermined distance based thereon, A touch control device.
3. The method of claim 2,
Wherein the hand candidate region detecting unit detects the largest region among regions obtained by performing labeling on the binarized region as a hand candidate region.
delete delete The method according to claim 1,
Wherein the hand region determination unit determines that the hand candidate region is a hand region if each of the widths of the two lines is less than 1.5 times the width of the other lines and less than 1/2 of the maximum width, Control device.
The method according to claim 1,
Wherein the thumb state determining unit sets the edges of the sensing axis and the hand area and determines whether the thumb is folded using the edges of the sensing axis and the hand area.
8. The method of claim 7,
Wherein the thumb state determination unit sets the sensing axis by connecting the centers of the widths of the two calculated lines to the hand region determination unit.
8. The method of claim 7,
The thumb state determination unit scans the left or right side by line on the basis of the set sensing axis and determines that the thumb is open if the number of straight lines having two or more edges is more than a certain number and determines that the thumb is folded when the number is less than the predetermined number A depth sensor-based spatial touch control device.
10. The method of claim 9,
Wherein the thumb state determination unit scans to the right when the hand is the left hand and scans to the left when the right hand.
Acquiring an image of a hand using a depth sensor, and detecting a hand candidate region based on the obtained image;
When the hand candidate region is detected, the detection is selected in the hand candidate region, and the widths of two lines positioned within the hand candidate region are calculated at predetermined intervals below the selected detection region. Based on this, ;
Judging the position of the index finger and judging whether the thumb is folded or not if it is determined that the hand candidate area is the hand area; And
Controlling the pointer on the screen according to the determined detection position and whether or not the thumb is folded,
The step of determining whether the hand region is the hand region may include comparing the width of each of the two lines positioned within the hand candidate region with a predetermined interval below the selected index to the width and the maximum width of another line measured in the hand candidate region, Judges whether the area is a hand area,
In the step of determining whether or not the hand region is in the hand region, the hand region is divided into a plurality of regions in accordance with whether or not the relative ratio of the width of each of the two lines to the width of the other line and the relative ratio of the width and the maximum width of each of the two lines satisfy a predetermined condition Based on a depth sensor-based spatial touch control method.
12. The method of claim 11,
In the step of detecting the hand candidate region,
Wherein the hand candidate region is detected by finding a point closest to the depth sensor in the acquired image and binarizing an area within a predetermined distance based on the detected point.
13. The method of claim 12,
Wherein the hand candidate region is the largest region obtained by performing labeling on the binarized region.
delete delete 12. The method of claim 11,
In the step of judging whether or not the hand region is in the hand region, if the width of each of the two lines is smaller than 1.5 times the width of the other lines and smaller than 1/2 of the maximum width, Sensor based space touch control method.
12. The method of claim 11,
Wherein the step of determining whether or not the thumb is folded is to set the edges of the sensing axis and the hand area and to determine whether the thumb is folded using the set edges of the sensing axis and the hand area.
18. The method of claim 17,
In the step of determining whether or not the thumb is folded,
Wherein the sensing axis is set by connecting the centers of the widths of the two lines calculated in the step of determining whether the sensing area is the hand area.
18. The method of claim 17,
In the step of determining whether or not the thumb is folded,
The sensor is scanned to the left or right side by line on the basis of the set sensing axis and it is judged that the thumb is opened when the number of straight lines having two or more edges is more than a certain number and the thumb is judged to be folded when the number is less than the predetermined number. A method for controlling a space of a touch screen.
20. The method of claim 19,
Wherein the step of determining whether the thumb is folded is to scan rightward if the hand is a left hand and scan to the left hand when a right hand.

Images