KR20240018122A - Apparatus and Method for Detecting Pointing Gesture - Google Patents

Abstract

An apparatus and method for detecting a pointing gesture are disclosed. A pointing gesture detection device according to an embodiment of the present invention includes a memory in which at least one program is recorded and a processor for executing the program, the program comprising: predicting a three-dimensional posture of the user's body from an input image, prediction A step of detecting the hand area and arm area through the posture information and a step of inferring whether a pointing gesture is performed based on deep learning using each of the detected hand area and arm area as input can be performed.

Description

Apparatus and Method for Detecting Pointing Gesture}

The described embodiments relate to techniques for recognizing pointing gestures.

Technologies related to pointing using RGB or RGBD cameras in the field of computer vision and image processing can be used as technologies such as robot manipulation and virtual cursors.

Conventional pointing gesture recognition technologies predict the 3D joint coordinates of the user's body and then calculate the intersection point in 3D space with a line (ray) connecting the joint coordinates, such as the line connecting the elbow and hand or the line connecting the eye and index finger. Through this, it is determined which object the user is pointing to in three-dimensional space.

However, most conventional methods are performed under the implicit assumption that the user is performing a pointing gesture.

In other words, since the pointing direction prediction algorithm always operates without determining whether a pointing gesture exists, it causes inefficiency problems due to unnecessary computational costs and power consumption for predicting the pointing direction even though it is not a pointing gesture.

In addition, as the pointing direction prediction result is output even though it is not a pointing gesture, you may be exposed to inconvenience and even risk due to unwanted malfunction.

For example, when selecting a menu or icon based on non-contact sensing on a kiosk or smart TV by predicting the pointing direction, a malfunction caused by the pointing direction prediction algorithm, which is always running even when a pointing gesture is not performed, may affect the use of the product. It may reduce usability. Additionally, malfunctions in industrial applications, such as operating infotainment through the HUD of a self-driving car or operating a drone, have the risk of leading to a major safety accident.

As mentioned above, although there is a clear need for technology to determine the presence or absence of a pointing gesture, gesture recognition technologies using conventional deep learning are limited to recognizing numbers (0~10), the alphabet (A~Z), etc. through static gesture recognition. It focuses on classification of symbols or recognition of dynamic gestures such as left, right, up, down, run, and stop.

Among conventional technologies, pointing gestures such as "Visual pointing gestures for bi-directional human robot interaction in a pick-and-place task." (2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 2015) There is a method to determine the operating conditions of the pointing direction prediction algorithm based on recognition of hands other than the hand, but this method requires unnecessary restrictions on the other hand and has limitations that make it difficult to utilize in driving situations.

Also, “Pointing gesture recognition based on 3d-tracking of face, hands and head orientation.” There are some conventional methods, such as (Proceedings of the 5th international conference on Multimodal interfaces. 2003), that determine whether a pointing gesture occurs based on a probability model using Hidden Markov Model (HMM), a traditional machine learning technique, etc. This has limitations in performance compared to recent neural network-based methods that show excellent performance in dependency modeling and data expression between input data. In addition, HMM is generally a modeling method for time series data, and has the disadvantage that the length of time series data is too short or the reliability of inference results is low in a single image.

The purpose of the described embodiment is to solve problems of inefficiency, malfunction, and risk caused by performing a pointing direction prediction operation without determining the presence or absence of a pointing gesture.

The purpose of the described embodiment is to solve the problem that the operating conditions of the pointing direction prediction algorithm are difficult to utilize due to physical or situational constraints.

The purpose of the described embodiment is to improve the performance of detecting the presence or absence of a pointing gesture in real time based on the user's body part even with a single image using deep learning technology.

A pointing gesture detection device according to an embodiment includes a memory in which at least one program is recorded and a processor for executing the program, the program comprising: predicting a three-dimensional posture of the user's body from an input image, predicted posture information A step of detecting the hand area and an arm area and a step of inferring whether a pointing gesture is performed based on deep learning using each of the detected hand area and arm area as input can be performed.

At this time, the input image can be acquired through an RGB camera or RGBD camera installed in equipment using non-contact sensing.

At this time, the program performs the predicting step, the detecting step, and the inferring step for each input image frame. However, if the inference result of a predetermined number or more of the input image frames within a certain period of time is a pointing gesture action, the pointing gesture is performed. Additional steps to determine that it has been detected may be performed.

At this time, the program further performs the step of determining the reliability of the landmark in the predicted 3D posture, and if the reliability of the landmark is not greater than a predetermined threshold, the step of detecting and inferring the corresponding input image frame is It can be omitted.

At this time, the 3D posture information is 3D joint coordinates representing each of the joints of the user's body, and in the detection step, the program determines that among the 3D joint coordinates of the user's body, the elbow, wrist, and palm are located on the same line. Assuming, the hand area and arm area can be detected.

At this time, when detecting the hand area, the program infers the position where the straight line connecting the 3-dimensional coordinates of the elbow and the 3-dimensional coordinates of the wrist extends in the direction of the hand by a predetermined multiple of the straight line as the hand position, focusing on the hand position. Steps of generating a bounding box in three-dimensional space and projecting the bounding box in three-dimensional space back into the pixel coordinate system through camera internal parameters may be performed to obtain two-dimensional bounding box coordinates corresponding to the hand area.

At this time, the size of the bounding box may be set differently depending on the user's age.

At this time, when detecting the arm area, the program can detect the arm area by generating a minimal square patch including the bounding box coordinates of the hand area, the 3D coordinates of the elbow, and the 3D coordinates of the shoulder.

At this time, in the inference step, the program calculates the average of the outputs for each of the deep learning network into which the detected hand area data is input and the deep learning network into which the detected arm area data is input, and uses the argmax operation to generate binary Classification can be performed.

A pointing gesture detection method according to an embodiment includes predicting a three-dimensional posture of the user's body from an input image, detecting a hand region and an arm region through predicted posture information, and detecting each of the detected hand region and arm region. It may include a step of inferring whether a pointing gesture is performed as an input based on deep learning.

At this time, the input image can be acquired through an RGB camera or RGBD camera installed in equipment using non-contact sensing.

At this time, the predicting step, the detecting step, and the inferring step are performed for each input image frame, and if the inference result of a predetermined number or more of the input image frames within a certain period of time is a pointing gesture, it is considered that the pointing gesture has been detected. A judgment step may be further included.

At this time, the pointing gesture detection method according to the embodiment further includes the step of determining the reliability of the landmark in the predicted 3D posture, and if the reliability of the landmark is not greater than a predetermined threshold, The detection step and the inference step can be omitted.

At this time, the 3D posture information is 3D joint coordinates representing each of the joints of the user's body, and the detecting step assumes that the elbow, wrist, and palm are located on the same line among the 3D joint coordinates of the user's body. The hand area and arm area can be detected.

At this time, the detection step is to detect the hand area, inferring the position where the straight line connecting the 3-dimensional coordinates of the elbow and the 3-dimensional coordinates of the wrist extends in the direction of the hand by a predetermined multiple of the straight line as the hand position. It may include generating a bounding box in a three-dimensional space as the center and projecting the bounding box in the three-dimensional space back into the pixel coordinate system through camera internal parameters to obtain two-dimensional bounding box coordinates corresponding to the hand area. there is.

At this time, the size of the bounding box may be set differently depending on the user's age.

At this time, in the detection step, when detecting the arm area, the arm area can be detected by generating a minimum square patch including the bounding box coordinates of the hand area, the 3D coordinates of the elbow, and the 3D coordinates of the shoulder.

At this time, the inference step calculates the average of the outputs of each of the deep learning network into which the detected hand area data is input and the deep learning network into which the detected arm area data is input, and performs binary classification through the argmax operation. can do.

A pointing gesture detection method according to an embodiment includes predicting 3D joint coordinates from an input image frame to the 3D posture of the user's body, detecting a hand region and an arm region through the predicted posture information, and detecting the detected hand It includes the step of inferring a deep learning-based ensemble using each of the region and the arm region as input, where the predicting step, detecting step, and inferring step are performed for each frame of the input image, and a deep learning-based ensemble for each predetermined number of frames is performed. If the inference results are pointing gestures, a step of determining that the user's pointing gesture has been detected may be further included.

At this time, the detecting step is to infer the position of the hand where the straight line connecting the three-dimensional coordinates of the elbow and wrist extends in the direction of the hand by a predetermined multiple of the straight line, and create a bounding box in three-dimensional space centered on the hand position. A step of generating, projecting the bounding box in 3D space back into the pixel coordinate system through camera internal parameters to obtain 2D bounding box coordinates corresponding to the hand area, and the bounding box coordinates of the hand area, the 3D coordinates of the elbow, and It may include detecting the arm area by generating a minimal square patch containing the 3D coordinates of the shoulder.

According to the described embodiment, it is possible to solve problems of inefficiency, malfunction, and risk caused by performing a pointing direction prediction operation without determining the presence or absence of a pointing gesture.

In other words, it is used as a prerequisite for the pointing direction prediction technology that is widely studied in the field of conventional computer vision, improving the efficiency and usability of the application system by predicting the pointing direction only when it is determined that a pointing gesture is being performed. In addition, risks in the industrial field due to unwanted malfunctions can be eliminated.

In addition, according to the described embodiment, it is possible to solve the problem that the pointing direction prediction algorithm operating condition is difficult to utilize due to body or situation constraints.

The purpose of the described embodiment is to improve the performance of detecting the presence or absence of a pointing gesture in real time based on the user's body part even with a single image using deep learning technology.

In addition, according to the described embodiment, it can be used as an assistive technology for early screening of children with ASD by determining whether the pointing gesture, which is a key element of the nonverbal communication characteristics of children with autism spectrum disorder (ASD), is performed. You can. In other words, compared to general children, children with ASD show great difficulty in using pointing gestures as a form of non-verbal communication. Therefore, in early screening of children with ASD, automatic pointing gesture detection technology is used to measure existing skills. It can be used as an assistive technology for early screening of children with ASD by alleviating the financial and time burden of the diagnosis process through qualified experts and by utilizing RGB cameras that can be easily installed not only in the ASD screening test space but also at home.

1 is a schematic block diagram of a pointing gesture detection device according to an embodiment.
Figure 2 is an example diagram in which a pointing gesture detection device according to an embodiment is utilized.
Figure 3 is a flowchart for explaining a pointing gesture detection method according to an embodiment.
Figure 4 is a diagram showing the configuration of a computer system according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Although terms such as “first” or “second” are used to describe various components, these components are not limited by the above terms. The above terms may be used only to distinguish one component from another component. Accordingly, the first component mentioned below may also be the second component within the technical spirit of the present invention.

The terms used in this specification are for describing embodiments and are not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” or “comprising” implies that the mentioned component or step does not exclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used in this specification can be interpreted as meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, an apparatus and method for detecting a pointing gesture according to an embodiment will be described in detail with reference to FIGS. 1 to 4 .

Hereinafter, an apparatus and method for detecting a pointing gesture according to an embodiment will be described in detail with reference to FIGS. 1 to 4 .

FIG. 1 is a schematic block diagram of a pointing gesture detection device according to an embodiment, and FIG. 2 is an example diagram of how the pointing gesture detection device according to an embodiment is utilized.

Referring to FIG. 1, the pointing gesture detection device 100 according to an embodiment can detect whether the user is making a pointing gesture based on deep learning in an input image.

In detail, the pointing gesture detection device 100 according to the embodiment includes an image-based body posture detection unit 110, a region of interest detection unit 120, a deep learning inference unit 130, and a pointing gesture detection unit 140. It can be included.

The image-based body posture detection unit 110 predicts the 3D posture of the user's body from an input image and may include an image acquisition unit 111 and a body posture detection unit 112.

The image acquisition unit 111 may acquire input images through the RGBD camera 11 and the RGB camera 12.

Here, in the case of the RGBD camera 11, since the distance value can be obtained in addition to the RGB value, 3D pose estimation can be easily performed with only a single camera.

Also, in the case of the RGB camera 12, ambiguity about the distance value may occur if only a single camera is used, but 3D coordinates can be inferred to some extent with a single image through a deep learning network learned through recent big data. This may be possible.

These RGBD cameras 11 and RGB cameras 12 may be attached to kiosks, smart TVs, self-driving cars, robots, drones, elevators, and ASD inspection spaces.

That is, the pointing gesture technology according to the embodiment is used in industrial fields that require human-robot interaction such as robot navigation, robot arm manipulation, or drone manipulation, or in various industrial fields such as kiosks, smart TVs, and elevators, using physical touch, It can be used as a prerequisite technology for a non-contact sensing-based instruction algorithm that replaces hardware such as joysticks and remote controls. In other words, by activating the non-contact sensing-based pointing algorithm only when it is determined that the user is performing a pointing gesture, not only does it improve efficiency by preventing unnecessary power consumption and memory use, but also reduces usability due to unwanted manipulation. It can be used to prevent the risk of accidents, etc.

The body posture detector 112 infers 3D posture information from the acquired image.

At this time, the 3D posture information may be 3D joint coordinates representing each joint of the user's body based on the camera coordinate system. For example, referring to FIG. 2 , the 3D posture information may be 3D joint coordinate information 210 representing each joint of the user's body.

In detail, the body posture detection unit 112 detects at least one user in the input image based on a top-down algorithm such as HRNet and Stacked Hourglass network, and then detects the at least one user. Precise 2D joint positions can be inferred through posture inference for each region.

At this time, if there are many people in the input image, bottom-up OpenPose and HigherHRNet may be used to improve inference speed.

As described above, the body posture detector 112 may back-project the detected 2D coordinates based on the pixel coordinate system into 3D coordinates using the distance value and camera internal parameters.

The 3D joint coordinates predicted in this way can be used as landmarks to detect areas of interest in the future.

At this time, depending on the embodiment, the body posture detection unit 112 determines the reliability of these landmarks, and if the reliability of the landmarks is not greater than a predetermined threshold, the 3D posture information for the corresponding image frame is not used, The 3D pose for the next input image frame is estimated.

The region of interest detection unit 120 detects at least one region of interest (ROI) as a body part required for deep learning inference using the predicted 3D joint coordinates of the user's body.

In the field of conventional action recognition, human areas within an image are commonly detected for time series input data and a deep learning network is learned using the detected full body area. On the other hand, in the embodiment, the deep learning network for detecting pointing gestures uses still images, especially body part areas of hands and arms, as data for learning and inference.

The reason why this embodiment is differentiated from the conventional action recognition or gesture recognition fields is as follows.

First, there is no large-scale public training data to determine the presence or absence of a pointing gesture, and therefore, there is a high possibility that the accuracy of deep learning inference will decrease due to the domain gap due to differences in data characteristics during learning and inference. . In the case of learning and inference of pointing behavior using time series data, the amount of change in the behavioral aspect and execution time of the pointing gesture is large, and collecting learning data that includes all of this spectrum consumes additional enormous human and temporal resources.

To solve this, you can try still image-based deep learning learning and inference, but still image-based gesture recognition lacks motion information on the time axis, high inter-class variance, and low inter-class variance. It causes additional problems such as intra-class variance.

Therefore, in order to solve these problems in the embodiment, the hand region and arm region, which are body parts specialized for detecting pointing gestures, can be used as input data for the deep learning network. In the case of pointing gesture recognition, unlike conventional gesture recognition technologies that distinguish between existing fine-grained classes, the purpose is simple binary classification of whether or not a pointing gesture is performed, so it is used for pointing behavior. By directly first purifying specialized input data and transmitting it to the deep learning network, the stability of learning can be increased by minimizing the domain gap between the training data and the data to be used during inference.

In other words, “Pointing: Where language, culture, and cognition meet.” As described in (Psychology Press, 2003), pointing behavior has clear characteristics that distinguish it from other movements, such as extension of the index finger, bending the remaining fingers toward the palm, and extension of the arm. By utilizing the appearance information of the hand and arm, the domain gap between data can be minimized, while at the same time, pointing gestures can be easily identified with just one still image that lacks motion information.

To this end, the region of interest detection unit 120 includes an arm area detection unit 121 and a hand area detection unit 122.

At this time, based on the assumption that among the predicted 3D joint coordinates, the elbow, wrist, and palm are on the same line (collinearity), the hand and arm regions, which are body parts required for multi-scale deep learning inference, are detected as regions of interest.

When using a CNN-based body part detector (e.g. hand detector) learned using conventional cropped body part data, it is dependent on the angle and size of the body part in the image, and in particular, the scale occupied by the body part in the image is different from the existing body part detector. If it is different from the data used for learning, it has the disadvantage of not providing stable detection performance.

Therefore, in the embodiment, "Hand keypoint detection in single images using multiview bootstrapping." was used to detect hand and arm regions, which are stable body parts. Similar to the method disclosed in (Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017), detection of the corresponding body region is performed by assuming that among the detected 3D body joints, the elbow, wrist, and hand are in a straight line. .

The hand area detector 121 may infer as the hand position the position where the line connecting the 3D coordinates of the elbow and the 3D coordinates of the wrist extends a predetermined multiple of the line in the direction of the hand.

For example, as shown in FIG. 2, through a heuristic judgment, it is assumed that the hand position 212 is in a position extended relative to the wrist by about 0.5 times the length 211 between the elbow and the wrist. Through this, the 3D position of the hand can be inferred.

Then, the hand area detector 121 creates a bounding box in three-dimensional space centered on the hand position.

At this time, the size of the bounding box may vary depending on the size of the user's hand, such as an adult or child. In an embodiment, it may be set to 140 mm based on heuristic judgment and experimental experience.

Then, the hand area detection unit 121 projects the bounding box in the three-dimensional space back into the pixel coordinate system through camera internal parameters to obtain two-dimensional bounding box coordinates corresponding to the hand area.

When detecting the arm area in the hand area detection unit 121, the arm area detection unit 122 generates a minimum square patch including the bounding box coordinates of the hand area, the 3D coordinates of the elbow, and the 3D coordinates of the shoulder to detect the arm area. can be detected. For example, as shown in FIG. 2, a square patch 222 including a hand area 221 and shoulder coordinates may be an arm area.

The deep learning inference unit 130 can infer whether a pointing gesture is performed based on deep learning using each of the detected hand and arm regions as input.

To this end, the deep learning inference unit 130 may include a model ensemble inference unit 131 and a time series result synthesis unit 132.

At this time, the model ensemble inference unit 131 synthesizes the deep learning-based static pose classification model inference results with multiple inputs of each detected hand region and arm region for a single image.

That is, as shown in FIG. 2, data in the hand area 221 are input to the first deep learning network 231, and data in the arm area 222 are input to the second deep learning network 232 for inference. It can be.

At this time, the input data for each hand and arm area can be resized to 224x224 for deep learning learning and inference, then scaled to 0 to 1 values for the RGB pixel range and normalized using the mean and variance of ImageNet. there is.

Additionally, the deep learning networks 231 and 232 according to the embodiment can freely utilize various deep learning networks such as VGG, ResNet, and ViT. And, the last layer of the deep learning network may be composed of a fully connected layer with output dimension 2 for binary classification.

As shown in FIG. 2, the model ensemble inference unit 131 includes a first deep learning network 231 into which detected hand region data is input and a second deep learning network 232 into which detected arm region data is input. The outputs for each are ensembled (240). Specifically, the average of the output values can be calculated and binary classification can be performed through the argmax operation.

This can be seen as being mechanically consistent with ensemble inference using a type of multi-scale input or multi-modal input, and is classified as a method of synthesizing prediction results through late fusion.

The time series result synthesis unit 132 synthesizes the result values on the time series axis in order to alleviate the volatility of the result values of each frame unit and derive more stable results. In other words, frame-by-frame inference results can be stored in a buffer of size 3 frames and synthesized.

The pointing gesture determination unit 140 determines whether the user is currently performing a pointing gesture based on the inference result of the deep learning inference unit 130. That is, if the inference result of a predetermined number of input image frames within a certain period of time is a pointing gesture, it may be determined that the pointing gesture has been detected.

As described above, when it is determined that it is a pointing gesture by the pointing gesture detection device 100 according to another embodiment, non-contact sensing such as kiosks, smart TVs, self-driving cars, robots, drones, elevators, and ASD examination spaces Pointing gesture recognition can be performed on this required equipment.

For example, as shown in FIG. 2, it can be used in an application system such as detecting an object 250 pointed by a user in a three-dimensional space linked through a ray casting method 230 connecting the eye and index finger.

Figure 3 is a flowchart for explaining a pointing gesture detection method according to an embodiment. Here, the pointing gesture detection method according to the embodiment may be performed by the above-described device 100.

Referring to FIG. 3, the pointing gesture detection method according to the embodiment includes predicting a three-dimensional posture of the user's body from an input image (S310), detecting a hand region and an arm region through the predicted posture information (S310). S340 to S345) and a step of inferring whether a pointing gesture is performed based on deep learning using each of the detected hand and arm regions as input (S350 to S370).

First, the device 100 estimates at least one 3D posture of a user existing within the camera's field of view (S310).

At this time, the input image can be acquired through an RGB camera or RGBD camera installed in equipment using non-contact sensing.

At this time, the device 100 analyzes landmark reliability for detecting the hand region and arm region among at least one estimated 3D posture (S320).

At this time, landmarks for detecting the hand area and arm area may include the shoulder, elbow, and wrist.

Then, the device 100 determines whether the analyzed landmark reliability is greater than a predetermined threshold T ₁ (S330).

If the landmark reliability is not greater than the predetermined threshold T ₁ as a result of the determination in S330, pointing gesture detection for the corresponding frame is omitted, and the process proceeds to S310 to perform posture estimation for the next frame.

On the other hand, if the landmark reliability is greater than the predetermined threshold T ₁ as a result of the determination in S330, the device 100 detects the region of interest based on the selected landmark (S340, S345). That is, the hand area is detected in S340, and the arm area is detected in S345 based on the predicted hand area coordinates.

At this time, the 3D posture information is 3D joint coordinates representing each of the joints of the user's body, and the detecting steps (S340, S345) are performed by determining that among the 3D joint coordinates of the user's body, the elbow, wrist, and palm are on the same line. Assuming their location, the hand area and arm area can be detected.

At this time, the step of detecting the hand area (S340) is a step of inferring the position where the straight line connecting the 3-dimensional coordinates of the elbow and the 3-dimensional coordinates of the wrist extends in the direction of the hand by a predetermined multiple of the straight line as the hand position, with the hand position as the center. The steps of creating a bounding box in 3D space and projecting the bounding box in 3D space back into the pixel coordinate system through camera internal parameters can be performed to obtain 2D bounding box coordinates corresponding to the hand area. .

At this time, the size of the bounding box may be set differently depending on the user's age.

At this time, in the step of detecting the arm area (S345), the arm area can be detected by generating a minimal square patch including hand area coordinates, elbow coordinates, and shoulder coordinates.

Then, the device 100 performs hand region-based deep learning inference (S350) and arm region-based deep learning inference (S355) based on a deep learning network for at least each of the detected hand region and arm region.

The device 100 synthesizes the hand region-based deep learning inference results and the arm region-based deep learning inference results through a model ensemble (S360). That is, the average of the outputs of each of the deep learning network into which the detected hand area data is input and the deep learning network into which the detected arm area data is input is calculated, and binary classification can be performed through the argmax operation.

At this time, the frame-by-frame inference results from S310 to S360 may be stored in a buffer.

Then, the device 100 determines the presence or absence of a pointing gesture by synthesizing the stored time series data (S370). That is, if the inference result of more than a predetermined number of input image frames within a period of time is a pointing gesture, it may be determined that the pointing gesture has been detected.

Figure 4 is a diagram showing the configuration of a computer system according to an embodiment.

The pointing gesture detection device 100 according to the embodiment may be implemented in a computer system 1000 such as a computer-readable recording medium.

Computer system 1000 may include one or more processors 1010, memory 1030, user interface input device 1040, user interface output device 1050, and storage 1060 that communicate with each other via bus 1020. You can. Additionally, the computer system 1000 may further include a network interface 1070 connected to the network 1080. The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in the memory 1030 or storage 1060. The memory 1030 and storage 1060 may be storage media including at least one of volatile media, non-volatile media, removable media, non-removable media, communication media, and information transfer media. For example, memory 1030 may include ROM 1031 or RAM 1032. Through the above process, the points that differentiate the present invention from conventional methods and implementation examples of the present invention have been described at a level that can be implemented by those with knowledge in the same industry. A body part-based pointing gesture detection system using still images according to an embodiment of the present invention detects body parts through a pose estimation algorithm to determine whether or not a pointing gesture is performed, and uses the detected parts as input to perform deep We present a methodology that is differentiated from conventional techniques through the process of running inference, model ensemble, and time series result synthesis.

Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing the technical idea or essential features. You will understand that it exists. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: Pointing gesture detection device
110: Image-based body posture extraction unit 111: Image acquisition unit
112: body posture detection unit 120: region of interest detection unit
121: arm area detection unit 122: hand area detection unit
130: Deep learning inference unit 131: Model ensemble inference unit
132: Time series result synthesis unit 140: Pointing gesture detection unit

Claims (20)

a memory in which at least one program is recorded; and
Contains a processor that executes a program,
The program is,
Predicting a 3D posture of a user's body from an input image;
Detecting a hand area and an arm area through predicted posture information; and
A pointing gesture detection device that performs a step of inferring whether a pointing gesture is performed based on deep learning using each of the detected hand and arm areas as input. The method of claim 1, wherein the input image is:
A pointing gesture detection device obtained through an RGB camera or RGBD camera installed on equipment where non-contact sensing is used. The method of claim 1, wherein the program:
The prediction step, detection step, and inference step are performed for each input image frame,
A pointing gesture detection device further performs the step of determining that a pointing gesture has been detected when the inference result of a predetermined number of input image frames within a certain period of time is a pointing gesture. According to claim 3, the program:
Further performing the step of determining the reliability of the landmark in the predicted 3D posture,
A pointing gesture detection device that omits the detection and inference steps for the corresponding input image frame when the reliability of the landmark is not greater than a predetermined threshold. The method of claim 1, wherein the 3D posture information is:
These are three-dimensional joint coordinates representing each joint of the user's body,
The program is,
In the detection step, a pointing gesture detection device that detects the hand area and arm area by assuming that the elbow, wrist, and palm are located on the same line among the three-dimensional joint coordinates of the user's body. According to claim 5, the program:
In detecting the hand area, inferring as the hand position a position where a straight line connecting the 3-dimensional coordinates of the elbow and the 3-dimensional coordinates of the wrist extends in the direction of the hand by a predetermined multiple of the straight line;
Creating a bounding box in three-dimensional space centered on the hand position; and
A pointing gesture detection device that performs the step of acquiring two-dimensional bounding box coordinates corresponding to the hand area by projecting the bounding box in three-dimensional space back into the pixel coordinate system through camera internal parameters. The method of claim 6, wherein the bounding box is:
A pointing gesture detection device whose size is set differently depending on the user's age. According to claim 6, the program:
A pointing gesture detection device that, in detecting the arm area, detects the arm area by generating a minimum square patch including the bounding box coordinates of the hand area, the 3-dimensional coordinates of the elbow, and the 3-dimensional shoulder. According to claim 8, the program:
In the inference step, the average is calculated for the outputs of each of the deep learning network into which the detected hand area data is input and the deep learning network into which the detected arm area data is input, and binary classification is performed through the argmax operation. Pointing gesture detection device. Predicting a 3D posture of a user's body from an input image;
Detecting a hand area and an arm area through predicted posture information; and
A pointing gesture detection method comprising the step of inferring whether a pointing gesture is performed based on deep learning using each of the detected hand and arm regions as input. The method of claim 10, wherein the input image is:
A pointing gesture detection method obtained through an RGB camera or RGBD camera installed on equipment using non-contact sensing. The method of claim 10, wherein the predicting, detecting, and inferring steps include:
It is performed for each input video frame,
A pointing gesture detection method further comprising determining that a pointing gesture has been detected when the inference result of a predetermined number or more of the input image frames within a certain period of time is a pointing gesture motion. According to claim 12,
Further comprising the step of determining the reliability of the landmark in the predicted 3D posture,
A pointing gesture detection method in which, when the reliability of a landmark is not greater than a predetermined threshold, the detection and inference steps for the corresponding input image frame are omitted. The method of claim 10, wherein the 3D posture information is:
These are three-dimensional joint coordinates representing each joint of the user's body,
The detection step is,
A pointing gesture detection method that detects the hand area and arm area by assuming that the elbow, wrist, and palm are located on the same line among the 3D joint coordinates of the user's body. The method of claim 14, wherein the detecting step includes:
In detecting the hand area, inferring as the hand position a position where a straight line connecting the 3-dimensional coordinates of the elbow and the 3-dimensional coordinates of the wrist extends in the direction of the hand by a predetermined multiple of the straight line;
Creating a bounding box in three-dimensional space centered on the hand position; and
A pointing gesture detection method comprising the step of projecting a bounding box in three-dimensional space back into the pixel coordinate system through camera internal parameters to obtain two-dimensional bounding box coordinates corresponding to the hand area. The method of claim 15, wherein the bounding box is:
A pointing gesture detection method whose size is set differently depending on the user's age. The method of claim 15, wherein the detecting step includes:
A pointing gesture detection method that detects the arm area by generating a minimum square patch containing the bounding box coordinates of the hand area, the 3D coordinates of the elbow, and the 3D shoulder coordinates. The method of claim 17, wherein the inferring step is:
A pointing gesture detection method that calculates the average of the outputs of each of the deep learning network into which the detected hand area data is input and the deep learning network into which the detected arm area data is input, and performs binary classification through the argmax operation. . Predicting 3D joint coordinates from an input image frame with a 3D posture of the user's body;
Detecting a hand area and an arm area through predicted posture information; and
It includes a deep learning-based ensemble inference step using each of the detected hand and arm regions as input,
The predicting, detecting, and inferring steps are performed for each frame of the input image,
A pointing gesture detection method further comprising determining that a user's pointing gesture has been detected when the deep learning-based ensemble inference results for a predetermined number of frames are pointing gestures. The method of claim 19, wherein the detecting step includes:
inferring the position of the hand where a straight line connecting the three-dimensional coordinates of the elbow and wrist extends in the direction of the hand by a predetermined multiple of the straight line;
Creating a bounding box in three-dimensional space centered on the hand position;
Obtaining two-dimensional bounding box coordinates corresponding to the hand area by projecting the bounding box in three-dimensional space back into the pixel coordinate system through camera internal parameters; and
A method for detecting a pointing gesture, comprising the step of detecting an arm area by generating a minimal square patch including bounding box coordinates of the hand area, 3D coordinates of the elbow, and 3D shoulder coordinates.

Images