KR101785650B1 - Click detecting apparatus and method for detecting click in first person viewpoint - Google Patents

Abstract

Acquiring a first sequence image of a hand photographed by a single depth camera worn by a user; Obtaining a first spatio-temporal feature vector from a plurality of frames in the acquired first sequence image; Constructing a random forest based on a second spatio-temporal feature vector extracted from a frame of a second sequence image of a hand that knows information about the occurrence of the click action and the click position; And inputting the first spatiotemporal feature vector into the random forest to determine whether or not a click action of a hand occurs in the first sequence image and a click position. According to an embodiment of the present invention, A method for detecting clicks by a device is disclosed.

Description

[0001] CLICK DETECTING APPARATUS AND METHOD FOR DETECTING CLICK IN FIRST PERSON VIEWPOINT [0002]

The present invention relates to a click sensing device at a first-person viewpoint and a click sensing method therefor. More particularly, the present invention relates to a method for determining a click action and a click position of a user's hand in augmented reality or virtual reality situations.

A number of studies on hand tracking and hand gesture recognition in augmented reality (AR) and virtual reality (VR) environments have presented various interaction scenarios. Interaction scenarios include multi-click interactions in the mid-air [5], gesture-based inputs in the wearable AR environment [3], [4] (Refer to Reference [10]) of a VR scene of a robot and a direct-click of an object in a state of wearing a head mounted display (HMD) and a hand posture prediction sensor (Reference [1]) in a VR environment .

However, most studies are unsuitable for detecting the movement of the fingertip of a finger that is obscured by its palm or other finger in the first person view (or in an egocentric viewpoint). There are the following difficulties in detecting the click movement and the click position of the hand at the first person viewpoint.

First, when a click occurs in a situation where the user can move freely, the finger to be clicked may be blocked by another finger or may be blocked by the palm of the hand.

Second, users perform click operations in various ways for each user. For example, a user may perform a click operation using only one finger joint, while another user may perform a click operation using all the finger joints of the finger.

Third, the attitude of each user varies greatly. Since there is 4 degrees of freedom (DOF) per finger, the degree of freedom of the hand is 26 degrees of freedom in total, considering the degrees of freedom of the five fingers and the degree of freedom of the palm.

As described above, there are many difficulties in detecting the clicking action and the click position of the hand. However, a method of accurately detecting the clicking action and the click position of the hand while ensuring the free movement of the user at the first person viewpoint of the VR environment and the AR environment, It is necessary for development of AR field.

Meanwhile, a list of references referred to in the present specification is as follows.

[1] Leap motion. http://www.leapmotion.com/. Accessed Sep. 10, 2014.

[2] M. K. Bhuyan, Neog, D. R., and M. K. Kar. Fingertip detection for handpose recognition. International Journal of Computer Science and Engineering, 4 (3): 501-511, March 2012.

[3] A. Colaco, A. Kirmani, H. S. Yang, N.-W. Gong, C. Schmandt, and V. K. Goyal. Mime: Compact, low power 3d gesture sensing for interaction with head mounted displays. In UIST, pages 227-236, USA, 2013.

[4] T. Ha, S. Feiner, and W. Woo. Wearhand: Head-worn, RGB-D camerabased, bare-hand user interface with visually enhanced depth perception. In ISMAR, pages 219-228. IEEE, September 2014.

[5] G. Hackenberg, R. McCall, and W. Broll. Lightweight palm and finger tracking for real-time 3d gesture control. In Virtual Reality Conference (VR), 2011 IEEE, pages 19-26, march 2011.

[6] P. Krejov and R. Bowden. Multi-touchless: Real-time fingertip detection and tracking using geodesic maxima. 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 0: 1-7, 2013.

[7] V. Lepetit and P. Fua. Keypoint Recognition Using Random Forests and Random Ferns, pages 111-124. Springer, 2013.

[8] Y. Liao, Y. Zhou, H. Zhou, and Z. Liang. Fingertips detection algorithm based on skin color filtering and distance transformation. In QSIC, pages 276-281. IEEE, 2012.

[9] S. Melax, L. Keselman, and S. Orsten. Dynamics based 3d skeletal hand tracking. In Proceedings of the 2013 Graphics Interface Conference, GI'13, pages 63-70, Toronto, Ont., Canada, Canada, 2013.

[10] Z. Pan, Y. Li, M. Zhang, C. Sun, K. Guo, X. Tang, and S. Z. Zhou. A real-time multi-cue hand tracking algorithm based on computer vision. In Proceedings of the 2010 IEEE Virtual Reality Conference, VR '10, pages 219-222, Washington, DC, USA, 2010. IEEE Computer Society.

A click sensing apparatus and a click sensing method in a first person view according to an embodiment of the present invention aim at accurately detecting a click operation and a click position of a user's hand.

In addition, the click sensing device and the click sensing method in the first person view according to an embodiment of the present invention are intended to ensure free movement of the user.

In addition, according to an embodiment of the present invention, there is provided a click sensing device and a click sensing method in a first person view, in which a user is not required to install a separate device in addition to a single depth camera worn by a user.

According to an embodiment of the present invention,

Acquiring a first sequence image of a hand photographed by a single depth camera worn by a user; Obtaining a first spatio-temporal feature vector from a plurality of frames in the acquired first sequence image; Constructing a random forest based on a second spatio-temporal feature vector extracted from a frame of a second sequence image of a hand that knows information about the occurrence of the click action and the click position; And inputting the first space time feature vector to the random forest to determine whether a hand click occurs in the first sequence image and a click position.

Wherein the acquiring of the first space time feature vector comprises: sensing a position of a finger finger of a hand and a position of a finger joint in each of a plurality of frames in the first sequence image; And obtaining the first space time feature vector by arranging the position of the finger tip of the detected hand and the position of the finger joint in a frame order.

The step of constructing the random forest may include: randomly selecting a type of a split function of a split node in the tree of the random forest; And optimizing parameters of the partition function selected at the split node.

The partitioning function may include a function for returning a position vector derived from one frame, a function for returning a velocity vector derived from two frames, and a function for returning an acceleration vector derived from three frames.

The partition function corresponds to the following equation,

[Mathematical Expression]

는 분할 함수, m은 분할 함수를 나타내는 인덱스로서 무작위로 선택되는 값,

는 제 1 시퀀스 영상(V) 내 n번째 프레임에서 i 손가락의 공간 특징 벡터, i는 손가락을 가리키는 인덱스, j는 특정 손가락의 핑거 조인트들과 핑거 팁 중 어느 하나를 가리키는 인덱스,

는 단일 프레임에서 j에 해당하는 포인트의 위치 벡터를 반환하는 함수,

는 두 개의 프레임에서 j에 해당하는 포인트의 속도 벡터를 반환하는 함수,

는 세 개의 프레임에서 j에 해당하는 포인트의 가속도 벡터를 반환하는 함수, n은 제 1 시퀀스 영상 내 프레임의 개수 및 p, q, r은 선행 오프셋일 수 있다.In the above equation,

Is a partition function, m is a randomly selected index indicating the partition function,

I is an index indicating a finger, j is an index indicating one of the finger joints and finger tips of a specific finger, i is a spatial feature vector of the i-th finger in the n-th frame in the first sequence image V,

A function that returns a position vector of a point corresponding to j in a single frame,

Is a function that returns the velocity vector of a point corresponding to j in two frames,

Where n is the number of frames in the first sequence image and p, q, r may be the preceding offset.

Wherein the sensing the position of the finger tip comprises: extracting contour points of a hand in a frame in the first sequence image; Calculating a distance between a central point of the palm and the extracted contour points and calculating a distance variation with respect to the previous contour point for each contour point; Constructing a shape feature vector for each of the contour points with a distance variation amount at each of a predetermined number of previous contour points, a distance variation amount at a current contour point, and a distance variation amount at each of a predetermined number of subsequent contour points; Constructing a random forest based on a shape feature vector of an image of a hand that knows the location of the finger tip; And inputting a shape feature vector of each of the contour points to a random forest configured based on the shape feature vector to detect the position of the finger tip in the frame in the first sequence image.

Wherein the determining comprises comparing an average probability of occurrence of a click motion of a hand determined in each of the plurality of trees of the random forest with a predetermined threshold value to determine whether or not the click operation of the hand occurs; And determining a click position of a hand having the highest vote among the click positions of the hand determined in each of the plurality of trees of the random forest as a final click position.

The single depth camera worn by the user may include a head mounted display (HMD).

According to another aspect of the present invention,

An image acquiring unit acquiring a first sequence image of a hand photographed by a single depth camera worn by a user; An image sensing unit for acquiring a first spatio-temporal feature vector from a plurality of frames in the acquired first sequence image; And a second spatio-temporal feature vector extracted from a frame of a second sequence image of a hand that knows information about whether a click operation has occurred and a click position, and generates a first spatio-temporal feature And a controller for inputting a vector to the random forest and determining whether a click operation of the hand is generated and a click position in the first sequence image.

Some effects that can be achieved by the click sensing device and the click sensing method in the first person view according to an embodiment of the present invention are as follows.

i) It is possible to accurately detect the click operation and the click position of the user's hand.

ii) it can guarantee the free movement of the user.

iii) In addition to a single depth camera worn by the user, the user is not required to install a separate device.

However, the effects that can be achieved by the click sensing device and the click sensing method in the first person view according to the embodiment of the present invention are not limited to those mentioned above, It will be understood by those skilled in the art that the present invention can be understood by those skilled in the art.

FIG. 1 is a schematic diagram for explaining a click detection method according to an embodiment of the present invention. Referring to FIG.
2 is an exemplary diagram for explaining space-time features in an embodiment of the present invention.
Figure 3 is an exemplary diagram illustrating a tree of random forests configured based on space-time features.
4 (a) and 4 (b) are views for explaining a method of detecting a finger tip of a hand in a frame.
5 is a diagram illustrating an ROC curve of the click detection method according to an embodiment of the present invention.
6A to 6D are diagrams showing experimental results in various scenarios for comparing the click detection method according to the conventional technology and the embodiment of the present invention.
7 is a block diagram illustrating a configuration of a click sensing apparatus according to another embodiment of the present invention.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It is to be understood, however, that the intention is not to limit the invention to the specific embodiments, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In addition, numerals (e.g., first, second, etc.) used in the description of the present invention are merely an identifier for distinguishing one component from another.

Also, in this specification, when an element is referred to as being "connected" or "connected" with another element, the element may be directly connected or directly connected to the other element, It should be understood that, unless an opposite description is present, it may be connected or connected via another element in the middle.

In addition, components referred to in this specification as 'units', 'modules', and the like refer to hardware components such as software, FPGA, or ASIC, and these components perform certain roles. However, the components are not limited to software or hardware. The component may be configured to reside on an addressable storage medium. Further, two or more components may be merged into one component, or one component may be divided into two or more functions according to a more refined function. In addition, each of the components to be described below may additionally perform some or all of the functions of the other components in addition to the main functions of the component itself, and some of the main functions And may be performed entirely by components.

Hereinafter, exemplary embodiments according to the technical idea of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic view for explaining a click sensing method (or a touch sensing method) according to an embodiment of the present invention.

A click sensing device according to an embodiment of the present invention acquires a first sequence image V of a hand photographed by a single depth camera worn by a user. The first sequence image V may include a plurality of frame images.

의 엘리먼트로 포함된다. 클릭 감지 장치는 3D 핸드 포스쳐 추정기(3D hand posture estimator)(R)(참고 문헌 [9])를 이용하여 손의 핑거 조인트의 위치를 감지할 수 있으며, 구현예에 따라서는 참조 문헌 [9]이외에 공지된 다른 방법에 따라 손의 핑거 조인트의 위치를 감지할 수도 있다. 참조 문헌 [9]의 3D 핸드 포스쳐 추정기(R)는 각 핑거 조인트의 기저 벡터(basis vector)를 제공하기 때문에 스케일(scale) 및 회전(rotation)에 대해 불변한 시공간 특징을 획득하기 위해,

의 모든 3차원 벡터들은 아래의 수학식 1에 따라 베이스 조인트

를 원점으로 하는 로컬 좌표로 변환된다. 로컬 좌표로의 변환은 후술하는 핑거 팁(fingertip)의 3차원 벡터에 대해서도 적용된다.The click sensing device senses the x-coordinate, y-coordinate and z-coordinate of a finger joint with a three-dimensional position of the finger joint of the hand in each frame of the first sequence image (V) do. The total number of finger joints in the hand is 20, and the positions of twenty finger joints in one frame are three-dimensional vectors

. ≪ / RTI > The click sensing device can sense the position of a finger joint of a hand using a 3D hand posture estimator (R) (Ref. 9), and according to an embodiment, Alternatively, the position of the finger joint of the hand may be sensed according to other known methods. Since the 3D hand posture estimator R of Ref. 9 provides a basis vector of each finger joint, in order to obtain an invariant space-time characteristic for scale and rotation,

Lt; RTI ID = 0.0 > 1 < / RTI >

As the origin. The conversion to the local coordinates is also applied to the three-dimensional vector of a finger tip (to be described later).

[Equation 1]

의 변환 매트릭스의 인버스 매트릭스이다. 또한, R은 3×3의 회전 매트릭스(rotation matrix), T는 3×1의 이동 벡터(translation vector)를 나타내며, 이에 따라 M은 4×4의 변환 매트릭스이다.In Equation 1, v _l is the local coordinate of the finger joint, v _g is the global coordinate of the finger joint, M ^-1 is the base joint

Lt; / RTI > is the inverse matrix of the transformation matrix. Also, R represents a 3 × 3 rotation matrix and T represents a 3 × 1 translation vector, whereby M is a 4 × 4 transformation matrix.

의 엘리먼트로 포함된다. 본 발명의 실시예에서는 각 프레임에서 손의 핑거 팁을 감지하는데 있어, 랜덤 포레스트(random forest)(F_S)를 이용할 수 있는데, 이에 대해서는 후술한다.The click sensing apparatus modifies the reference documents [2], [8] so that the two-dimensional position of the fingertip of the hand in each frame of the first sequence image (V) And adds the z coordinate of each finger joint according to the depth value of the single depth camera to eventually detect the three-dimensional position of each finger tip. The position of the finger tip of the five fingers in one frame is a three-dimensional vector

. ≪ / RTI > In an embodiment of the present invention, a random forest (F _S ) may be used to detect a finger tip of a hand in each frame, as will be described later.

와 특징 벡터 s의 엘리먼트들을 선택적으로 조합하여 공간 특징 벡터(spatial feature vector) x를 아래의 수학식 2와 같이 구성한다.If a finger joint and a finger tip are detected in each frame, the click sensing device detects the feature vector

And the elements of the feature vector s are selectively combined to construct a spatial feature vector x according to Equation (2) below.

&Quot; (2) "

는

중에서

을 포함한다.In Equation (2), i denotes an index of a finger, t denotes an index of a frame (or time), t belongs to a range of 1 to n, and n denotes the total number of frames. If i is 3 in the above equation,

The

Between

.

The click sensing device combines x for a certain time in frame order to construct a space-time feature vector X = {x _it } as shown in FIG. Fig. 2 shows a space-time feature vector in which the spatial feature vectors of fingers with i = 3 are arranged in time.

The click sensing apparatus determines whether or not the user click action has occurred from the space time feature vector obtained from the first sequence image and whether the click action is generated (a) and the click position (a ') Extracts a space-time feature vector from a second sequence image that knows information in advance, and constructs a random forest (F _A ) by learning the extracted space-time feature vectors. The method of extracting the space-time feature vector from the second sequence image is the same as the method of extracting the space-time feature vector from the first sequence image.

In the present invention, the purpose of constructing a random forest (F _A ) based on the space-time feature vector is to detect a click operation and a click position in a classifier even if the finger tip is occluded. A random forest is an ensemble of binary decision trees. Each tree includes two kinds of nodes: a split node and a leaf node. Each split node of the random forest F _A in the present invention performs a split function of the specific task determined by a randomly selected parameter value h = (0: action detection, 1: position estimation) on the input data, It is determined whether it is routed to the lower left node or the lower right node. In the above, h represents the kind of specific business. The leaf node stores the probability of the click operation and the position of the finger tip in the three-dimensional space as an end node indicating the status of the click operation.

이 생성되며, 여기서,

은 분할 함수이고,

은 임계 값으로서, 입력 데이터 D를 두 개의 서브셋인 D^l과 D^r로 분할한다.

이고,

이다. V는 입력되는 시퀀스 영상을 의미한다.Each tree in the random forest (F _A ) grows by recursively partitioning the current training (or learning) data and sending it to two child nodes. At each node of the tree,

Is generated,

Is a partition function,

As the threshold, divides the input data D into two sub-sets, D ^l and D ^r .

ego,

to be. V is the input sequence image.

은 아래의 수학식 3과 같이 정의된다.Also,

Is defined as Equation (3) below.

&Quot; (3) "

는 시퀀스 영상(V) 내 n번째 프레임에서의 i 손가락의 공간 특징 벡터, i는 손가락을 가리키는 인덱스, j는 특정 손가락의 핑거 조인트들과 핑거 팁 중 어느 하나의 포인트의 특징 벡터를 가리키는 인덱스,

는 j에 해당하는 포인트의 위치 벡터를 반환하는 함수,

는 두 개의 프레임 사이의 시간 변화량에 따른, j에 해당하는 포인트의 위치 변화량, 즉 속도 벡터를 반환하는 함수이며,

는 세 개의 프레임 사이에서 j에 해당하는 포인트의 속도의 변화량, 즉, 가속도 벡터를 반환하는 함수이다. 또한, n은 시퀀스 영상 내 프레임의 개수, p, q, r은 선행 오프셋(preceding offset)을 의미한다.In the above equation (3), m is an index indicating a division function,

I is an index indicating a finger, j is an index indicating a feature vector of any one of finger joints and finger tips of a specific finger,

Is a function that returns the position vector of the point corresponding to j,

Is a function that returns a positional change amount of a point corresponding to j, i.e., a velocity vector, according to a time variation amount between two frames,

Is a function that returns the amount of change in velocity of a point corresponding to j between three frames, that is, an acceleration vector. Also, n is the number of frames in the sequence image, and p, q, and r are preceding offset.

m is determined based on a randomly selected task type h. For example, if h is 0, it means action detection task, so m is randomly determined to be 2 or 3. If h is 1, it means position estimation task, so m is determined as 1.

The random forest F _A according to an embodiment of the present invention is different from a general random forest in that split candidates of different kinds are stored according to the value of h in each split node of the tree. For example, if h is 0, the partition candidate that gives the largest information gain is stored as the best best partition candidate, and if h is 1, the smallest regression uncertainty The dividend candidate is stored as the optimal split candidate.

The above information gain is defined by Equation (4) below,

&Quot; (4) "

In Equation (4), H (·) is Shannon's Entropy, which is defined by Equation (5) below.

&Quot; (5) "

Regression uncertainty is defined by the following equation (6) based on the variance of the position vectors of the finger tips in the local coordinates.

&Quot; (6) "

의 위치로부터 핑거 팁 위치까지의 오프셋을 가리킨다. In Equation (6), [Sigma] x is a sample covariance matrix of a set of finger tip position vectors, and tr ([eta]) is a trace function. Here, the position vectors of the finger tips are calculated according to Equation (1)

Lt; / RTI > to the finger tip position.

The growth of the tree is repeated recursively for the split data D ^l and D ^r until the stop criterion is satisfied. The growth process of the tree can be stopped if the number of samples in the date set is smaller than the predetermined minimum number, or if the depth of the tree exceeds a predetermined value.

As shown in FIG. 3, the trees of the random forest F _A randomly select values of each parameter (e.g., h, m, i, j, p, q, In order to find the optimal partition function at the split node, we find the optimal parameter combination. Further, at the leaf node of the tree, the probability of the click operation and the offset vector of the click position located in the local coordinate of the base joint of the hand are stored in the learning step.

First to enter the space-time characteristics of the first sequence images to random forest (F _A), the probability of clicking operation state from a random forest (F _A) (e.g., a click operation right (true) probability and not the click operation (false ) ≪ / RTI > probability and the offset of the finger tip position within the local coordinate are determined as: < EMI ID = 7.0 >

&Quot; (7) "

이고,

는 클릭 동작

의 상태의 확률이며, 여기서,

이고,

은 클릭 동작이 발생하였을 때의 추정된 핑거 팁의 로컬 오프셋 위치 x, y, z이다. 클릭 동작

의 확률은 랜덤 포레스트(F_A)의 트리들의 리프 노드에 저장된 값들을 평균하여 계산된다. 클릭 감지 장치는 계산된 평균 값이 기 설정된 임계값보다 큰 경우, 클릭이 실제 발생한 것으로 판단한다. 또한, 랜덤 포레스트(F_A)의 트리들의 리프 노드에 저장된 로컬 오프셋 위치들 중 가장 많은 보우트(vote)의 로컬 오프셋 위치가 최종적인 핑거 팁의 클릭 위치, 즉, 로컬 오프셋 위치로 판단될 수 있다. 클릭 감지 장치는 최종적인 핑거 팁의 로컬 오프셋 위치를 글로벌 좌표로 변환한다.In Equation (7)

ego,

Click action

, Where < RTI ID = 0.0 >

ego,

Is the local offset position x, y, z of the estimated finger tip when the click action occurred. Click action

Is calculated by averaging the values stored in the leaf nodes of the trees of the random forest (F _A ). The click sensing device determines that a click actually occurs if the calculated average value is greater than a predetermined threshold. In addition, the local offset position of the most votes among the local offset positions stored in leaf nodes of the random forest F _A can be determined as the click position of the final finger tip, i.e., the local offset position. The click sensing device converts the local offset position of the final finger tip to global coordinates.

On the other hand, the click sensing device has been described as being capable of detecting the two-dimensional position of the finger tip of the hand in each frame of the first sequence image V by modifying reference documents [2] and [8] Will be described with reference to FIG.

은 0으로 설정된다.References [2] and [8] propose a method of extracting a plurality of contour points and a central point of the palm from the image of the hand, and sensing the finger tip of the hand based on the distance between the central point and each contour point . In an embodiment of the present invention, as shown in Fig. 4 (a), in order to construct a shape vector of a finger tip that is constant to the scale, the previous contour point at each contour point .

Is set to zero.

&Quot; (8) "

이고, w는 이미지 내 윤곽 포인트들의 개수이고, c_l은 이미지 내 l번째 윤곽 포인트의 2차원 위치를 나타낸다. 또한, d_l은 윤곽 포인트 c_l과 손바닥 중앙 포인트 p 사이의 거리를 나타내고, 아래의 수학식 9와 같이 계산된다. In Equation (8)

W is the number of contour points in the image, and c _l is the two-dimensional position of the lth contour point in the image. Further, d _l represents the distance between the contour point c _l and the palm center point p, and is calculated as shown in Equation 9 below.

&Quot; (9) "

이다. 실험적으로, offset은 7로 설정될 수 있으며, 이에 의하면, 모양 특징 벡터 Y는 총 15개의 엘리먼트를 포함한다.The shape feature vector at each contour point includes as an element a distance variation amount at each of a predetermined number of previous contour points, a distance variation amount at the current contour point, and a distance variation amount at each of a predetermined number of subsequent contour points. In other words, a sliding window is determined in consideration of the position of the contour point l, which is set to s _u = 1-offset shown in FIG. 4 (b), s _v = 1 + offset, Shape feature vector at point l

to be. Experimentally, the offset can be set to 7, whereby the shape feature vector Y contains a total of 15 elements.

과

로 분할하는데, 여기서,

이고,

이며,

과

는 모양 특징 벡터 Y의 엘리먼트이다.The click sensing device constructs a random forest (F _S ) based on shape feature vectors of images of the hand that already know the location of the finger tips. Here, the partition function of the split nodes of each tree of the random forest (F _S ) compares two elements randomly selected from the shape feature vector Y, similar to the two pixel test of Reference [7]. Thus, the feature data set Ds of the current split node is divided into two subsets:

and

Lt; RTI ID = 0.0 >

ego,

Lt;

and

Is an element of the shape feature vector Y.

When the random forest F _S is configured, the frames of the first sequence image may be input to the random forest F _S , and the final finger tip may be detected considering the probability of the finger tip and the probability of not being the Tinger tip.

FIG. 5 is a diagram illustrating an ROC curve of a click sensing method according to an exemplary embodiment of the present invention. FIGS. 6A to 6D are diagrams illustrating an example of a click sensing method in various scenarios for comparing a click sensing method according to an exemplary embodiment of the present invention Fig.

FIG. 5 shows the ROC curve of the click detection method according to the finger click operation. The processing time was calculated as 30.65 FPS (frame per second), and the accuracy of the click detection of each finger was 89.80% for the thumb, 96.90% for the index finger, %, Ring finger 92.82% and possession 94.37% respectively. This is a dramatically improved value compared to the failure rate of 77.09% of [6].

6A shows experimental results in a virtual environment in which static objects exist sparse, FIG. 6B shows experimental results in a virtual environment in which dynamic objects are sparsely present, FIG. 6C shows results of static FIG. 6D shows experimental results in a virtual environment in which objects are dense and FIG. 6D shows experimental results in a virtual environment in which dynamic objects are concentrated. In each graph, a represents the click area sensed according to the reference [2] and [8] when the experimenter clicks on the target object located at the center, and b is detected according to the reference [9] Represents a click area, and c represents a click area sensed according to an embodiment of the present invention.

As shown in FIGS. 6A to 6D, according to references [2] and [8], it is possible to confirm a tendency that a click position is sensed by biasing toward a user rather than a position of an actual target object. According to [9] It is possible to confirm the tendency of the click position to be detected by being shifted higher than the position of the target object. On the other hand, according to the embodiment of the present invention, it can be seen that the click position is uniformly detected around the target object without being shifted to either side.

FIG. 7 is a block diagram illustrating a configuration of a click sensing apparatus 700 according to another embodiment of the present invention.

7, the click sensing apparatus 700 may include an image acquiring unit 710, an image sensing unit 730, and a controller 750. The click sensing apparatus 700 may include an image sensing unit 710, an image sensing unit 730, The image acquisition unit 700, the image sensing unit 730, and the control unit 750 may be implemented by at least one microprocessor, and may operate according to a program stored in a memory (not shown).

The click sensing device 700 according to another embodiment of the present invention may be a wearable device, and may further include a single depth camera. Alternatively, depending on the implementation, the click sensing device 700 may be implemented as a separate computer from a single depth camera worn by the user.

The image acquiring unit 710 acquires a first sequence image of a hand photographed by a single depth camera worn by the user.

The image sensing unit 730 acquires a first space time feature vector from a plurality of frames in the first sequence image. Specifically, the image sensing unit 730 acquires the position of the finger tip of the hand and the position of the finger joint as a spatial feature vector in each of a plurality of frames in the first sequence image, and arranges the acquired spatial feature vector To obtain the first space time feature vector.

The control unit 750 constructs a random forest F _A based on the second space time feature vector extracted from the frame of the second sequence image of the hand that knows the click operation and the information about the click position in advance, The first space time feature vector is input to the random forest F _A to determine whether or not a hand click occurs in the first sequence image and a click position.

In order to detect the position of a finger tip of a hand in each of a plurality of frames in the first sequence image, the image sensing unit 730 may include a random forest F _S ) of a plurality of frames in the first sequence image to detect the position of the finger tip.

Meanwhile, the embodiments of the present invention described above can be written in a program that can be executed in a computer, and the created program can be stored in a medium.

The medium may include, but is not limited to, storage media such as magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.), optical reading media (e.g., CD ROMs,

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. You will understand. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

700: Click detection device
710:
730:
750:

Claims (10)

Acquiring a first sequence image of a hand photographed by a single depth camera worn by a user;
Obtaining a first spatio-temporal feature vector from a plurality of frames in the acquired first sequence image;
Constructing a random forest based on a second spatio-temporal feature vector extracted from a frame of a second sequence image of a hand that knows information about the occurrence of the click action and the click position; And
Inputting the first space time feature vector to the random forest to determine whether a hand click occurs in the first sequence image and a click position,
Wherein configuring the random forest comprises:
Randomly selecting a parameter of a split function of a split node in the tree of the random forest; And
And optimizing parameters of the partition function selected at the split node,
The partition function
A function for returning a position vector derived from one frame, a function for returning a velocity vector derived from two frames, and a function for returning an acceleration vector derived from three frames. Click detection method.
The method according to claim 1,
Wherein acquiring the first space-time feature vector comprises:
Sensing a position of a fingertip of a hand and a position of a finger joint in each of a plurality of frames in the first sequence image; And
And obtaining the first space time feature vector by arranging the position of the finger tip of the detected hand and the position of the finger joint in a frame order.
delete delete The method according to claim 1,
The partition function corresponds to the following equation,

[Mathematical Expression]

In the above equation,

Is a partition function, m is a randomly selected index indicating the partition function,

J is an index indicating a feature vector of a point of one of finger joints and finger tips of a specific finger, i is a spatial feature vector of an i-th finger in the n-th frame in the first sequence image V,

A function that returns a position vector of a point corresponding to j in a single frame,

Is a function that returns the velocity vector of a point corresponding to j in two frames,

Is a function for returning an acceleration vector of a point corresponding to j in three frames, n is the number of frames in the first sequence image and p, q, r are the preceding offsets.
3. The method of claim 2,
Wherein sensing the position of the finger tip comprises:
Extracting contour points of a hand in a frame in the first sequence image;
Calculating a distance between a central point of the palm and the extracted contour points and calculating a distance variation with respect to the previous contour point for each contour point;
Constructing a shape feature vector for each of the contour points with a distance variation amount at each of a predetermined number of previous contour points, a distance variation amount at a current contour point, and a distance variation amount at each of a predetermined number of subsequent contour points;
Constructing a random forest based on a shape feature vector of an image of a hand that knows the location of the finger tip; And
And detecting a position of a finger tip in a frame in the first sequence image by inputting a shape feature vector of each of the contour points to a random forest configured based on the shape feature vector. .
The method according to claim 1,
Wherein the determining step comprises:
Comparing a probability of occurrence of a click operation of a hand determined in each of the plurality of trees of the random forest with a preset threshold value to determine whether or not the click operation of the hand occurs; And
And determining a click position of a hand having the highest vote among the click positions of the hand determined in each of the plurality of trees of the random forest as a final click position.
The method according to claim 1,
The single depth camera worn by the user,
And a head mounted display (HMD).
8. A computer program stored on a medium in combination with hardware for executing the click detection method of any one of claims 1, 2, 5 to 8.
An image acquiring unit acquiring a first sequence image of a hand photographed by a single depth camera worn by a user;
An image sensing unit for acquiring a first spatio-temporal feature vector from a plurality of frames in the acquired first sequence image; And
Constructs a random forest based on a second space-time feature vector extracted from a frame of a second sequence image of a hand that knows information about whether a click operation has occurred and a click position, And a controller for inputting the first sequence image to the random forest and determining whether a click operation of the hand is generated and a click position in the first sequence image,
Wherein the controller randomly selects a parameter of a split function of a split node in the tree of the random forest and optimizes a parameter of a partition function selected at the split node, A function for returning a position vector derived from a frame of the frame, a function for returning a velocity vector derived from two frames, and a function for returning an acceleration vector derived from three frames.

Images