KR20230040849A - Method and apparatus for classifying action based on hand tracking - Google Patents

Abstract

Disclosed are a method and a device for classifying a behavior based on hand tracking. According to an embodiment of the present disclosure, a method for classifying a behavior based on hand tracking includes the steps of: obtaining an image of a non-face-to-face user; detecting a hand region of a non-face-to-face user image based on a pre-learned behavior classification algorithm and classifying the behavior of the non-face-to-face user; and determining whether the non-face-to-face user uses a non-allowed object, based on an action classification result. Therefore, even when an image in which the hand is obscured by an obstacle is acquired, high accuracy can be obtained.

Description

Method and apparatus for classifying action based on hand tracking {METHOD AND APPARATUS FOR CLASSIFYING ACTION BASED ON HAND TRACKING}

The present disclosure uses a deep learning-based learning model that classifies the action of a person in an image through a key point for grasping the posture of the hand and a hand region image, and hand tracking that determines whether the person performs an unacceptable action. It relates to a method and apparatus for classifying based actions.

Recently, non-face-to-face methods are being applied in all fields of society. In other words, non-face-to-face can be called digital communication. In order to communicate digitally in a non-face-to-face manner, digital transformation must first take place. Online education, video conferencing, and telecommuting are examples of digital transformation.

In particular, online exams are often conducted suddenly, so there is a risk of fraud such as cheating, but it has the advantage of reducing the social cost and time of large-scale field exams in line with the era of the Fourth Industrial Revolution.

In general, companies are also conducting online qualification exams to obtain license qualifications from recruitment exams. It is a trend that not only companies but also kindergartens, middle and high schools, and universities conduct online classes and conduct midterm and final exams online.

However, in the case of online exams, various cheating can occur. Therefore, in general, the student's access time, IP, etc. are checked to determine whether there is cheating. Furthermore, in order to check whether cheating has occurred, it is possible to check the student's video captured in real time through a webcam installed in the student's computer.

In this case, since test supervisors have to supervise the images of all test takers, it is difficult to process in real time, and there are inefficient problems such as high cost and time required for checking cheating, and low accuracy.

Accordingly, there is a need for an algorithm capable of automatically discriminating cheating through video and a method for improving the accuracy of the algorithm and reducing cost.

The foregoing background art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention.

Prior art 1: Korean Patent Registration No. 10-2195401 (2020.12.19)

One task of an embodiment of the present disclosure is to determine whether the person has performed an unacceptable action through a deep learning-based learning model that classifies the action of a person in an image based on a key point for grasping a hand posture and an image of a hand region , it is intended to automatically detect cheating.

The purpose of the embodiments of the present disclosure is not limited to the above-mentioned tasks, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present invention. will be. It will also be seen that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

A hand tracking-based action classification method according to an embodiment of the present disclosure includes acquiring an image of a non-face-to-face user, detecting a hand region of a non-face-to-face user image based on a pre-learned action classification algorithm, and Classifying the user's behavior, and determining whether the non-face-to-face user uses a disallowed object based on the behavior classification result.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to an embodiment of the present disclosure, a deep learning-based learning model that classifies actions of a person in an image based on a key point for grasping a hand posture and an image of a hand region accurately determines in real time whether the person has performed an unacceptable action. By judging, it is possible to detect cheating and respond immediately.

In addition, since a large delay time does not occur in processing a video of a non-face-to-face user transmitted in real time, real-time operation can be performed without difficulty in an actual service environment.

In addition, by using the ROI image of the hand and joint keypoints together, rather than simply using the image, information on the shape of the hand and its surroundings can be obtained from the image, and the shape of the hand can be learned more accurately from the keypoint. , it can have high accuracy even when an image in which the hand is obscured by an obstacle is obtained.

Effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram schematically illustrating a hand tracking-based action classification system according to an embodiment.
2 is a block diagram schematically illustrating a behavior classification apparatus according to an exemplary embodiment.
3 is a diagram schematically illustrating a network structure of an action classification algorithm according to an embodiment.
4 is an exemplary diagram of keypoints inferred by a hand region detection network of an action classification algorithm according to an embodiment.
5 is a diagram visualizing input data of an action detection network of an action classification algorithm according to an embodiment.
6 is an exemplary diagram of a structure of an action classification algorithm that takes a predetermined frame section of a user image as an input according to an embodiment.
7 is a diagram schematically illustrating a network structure of an action classification algorithm including a gaze tracking network according to an embodiment.
8 is an exemplary diagram illustrating a result of classifying an action including gaze tracking according to an exemplary embodiment.
9 is a diagram showing experimental results (loss and accuracy graphs of a first data set) of an action classification algorithm according to an embodiment.
10 is a diagram showing an experimental result (a mismatch matrix when only two classes are learned) of an action classification algorithm according to an embodiment.
11 is a diagram showing experimental results (loss and accuracy graphs of a second data set) of an action classification algorithm according to an embodiment.
12 is a diagram showing experimental results of an action classification algorithm (a mismatch matrix when learning using five classes) according to an embodiment.
13 is a flowchart illustrating a method of classifying an action according to an exemplary embodiment.
14 is a flowchart illustrating a method of classifying an action including eye tracking according to an exemplary embodiment.

Advantages and features of the present disclosure, and methods of achieving them, will become clear with reference to the detailed description of embodiments in conjunction with the accompanying drawings.

However, it should be understood that the present disclosure is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all conversions, equivalents, and substitutes included in the spirit and technical scope of the present disclosure. . The embodiments presented below are provided to complete the present disclosure and to fully inform those skilled in the art of the scope of the disclosure. In describing the present disclosure, if it is determined that a detailed description of related known technologies may obscure the gist of the present disclosure, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present disclosure. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded. Terms such as first and second may be used to describe various components, but components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof are omitted. I'm going to do it.

1 is a diagram schematically illustrating a hand tracking-based action classification system according to an embodiment.

Referring to FIG. 1 , a hand tracking based action classification system 1 may include an action classification device 100 , a user terminal 200 , a server 300 and a network 400 .

The hand tracking-based action classification system 1 photographs a non-face-to-face user through a photographing device provided at a location where the non-face-to-face user is located, and uses a deep learning-based learning model to infer whether a non-face-to-face user performs an unacceptable action in an image. will be.

For example, the hand tracking-based action classification system 1 may determine cheating, such as using an unacceptable object in an image of a user taking an online test.

In an embodiment, a behavior class may be defined as using normal and non-permissible objects in an image of a non-face-to-face user. For example, the data of the normal class may consist of an image of taking notes and a natural behavior during test taking, such as turning one's hair or touching one's face.

That is, in order to determine the class defined above, it is necessary to detect the region where the non-face-to-face user's hand is located in an image in which all surrounding environments of the non-face-to-face user are captured. In addition, in an embodiment, keypoints for each knuckle as well as image information of the hand should be detected and the two pieces of information should be used together to accurately grasp the hand posture. Key points may mean representative landmark points for hand identification. In the following, the key points are unified and described.

Therefore, the hand tracking-based behavior classification system 1 detects the non-face-to-face user's hand region and hand keypoints in the non-face-to-face user image, and the object and hand captured in the image are identified based on the detected hand region image and hand keypoints. Cheating can be recognized through the shape.

In one embodiment, the hand tracking-based action classification system 1 acquires a non-face-to-face user image, and a hand region detection network that extracts a cropped image of the non-face-to-face user's hand region and keypoint information on the hand shape, and a hand image. It can be composed of a behavior classification network that receives data that combines , and keypoint information and infers whether a non-face-to-face user is currently using a disallowed object.

That is, the hand tracking-based action classification system 1 can infer whether a non-face-to-face user is currently using a disallowed object based on an action classification algorithm composed of a hand region detection network and an action classification network.

Accordingly, the hand-tracking-based action classification system 1 can enable automation by allowing AI to judge in real time the existing task, for example, in which a test supervisor had to directly monitor and supervise all test takers' images.

In addition, the hand tracking-based action classification system 1 can train a model with higher accuracy by utilizing not only the image of the hand but also location information of specific joints.

Meanwhile, in an embodiment, users may access an application or website implemented in the user terminal 200 and perform processes such as generating and learning a network of the behavior classification apparatus 100 .

The user terminal 200 includes a desktop computer, a smart phone, a laptop computer, a tablet PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop computer, a media player, a micro server, and a global positioning system (GPS) device operated by a user. , e-book readers, digital broadcast terminals, navigation devices, kiosks, MP3 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but are not limited thereto.

In addition, the user terminal 200 may be a wearable terminal such as a watch, glasses, hair band, and ring having a communication function and a data processing function. The user terminal 200 is not limited to the above, and a terminal capable of web browsing may be borrowed without limitation.

In an embodiment, the hand tracking-based action classification system 1 may be implemented by the action classification device 100 and/or the server 300 .

In one embodiment, the action classification device 100 may be implemented in the server 300, where the server 300 is configured to operate the hand tracking-based action classification system 1 including the action classification device 100. It may be a server or a server implementing part or all of the behavior classification apparatus 100 .

In one embodiment, the server 300 obtains an image of a non-face-to-face user, extracts a cropped image of the non-face-to-face user's hand region and keypoint information about the hand shape, and based on data combining the hand image and the keypoint information It may be a server that controls the operation of the behavior classification device 100 for the overall process of inferring whether a non-face-to-face user is currently using a disallowed object.

Also, the server 300 may be a database server that provides data for operating the behavior classification apparatus 100 . In addition, the server 300 may include a web server, an application server, or a deep learning network providing server.

In addition, the server 300 may include a big data server and an AI server required to apply various artificial intelligence algorithms, a calculation server that performs calculations of various algorithms, and the like.

Also, in this embodiment, the server 300 may include the aforementioned servers or network with these servers. That is, in this embodiment, the server 300 may include or network with the above web server and AI server.

In the hand tracking based action classification system 1 , the action classification device 100 and the server 300 may be connected by a network 400 . Such a network 400 may be wired networks such as LANs (local area networks), WANs (wide area networks), MANs (metropolitan area networks), ISDNs (integrated service digital networks), wireless LANs, CDMA, Bluetooth, satellite communication However, the scope of the present disclosure is not limited thereto. In addition, the network 400 may transmit and receive information using short-range communication and/or long-distance communication.

Also, the network 400 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 400 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure enterprise private network. Access to network 400 may be provided through one or more wired or wireless access networks. Furthermore, the network 400 may support an Internet of Things (IoT) network and/or 5G communication in which information is exchanged and processed between distributed components such as things.

2 is a block diagram schematically illustrating a behavior classification apparatus according to an exemplary embodiment.

Referring to FIG. 2 , the behavior classification apparatus 100 may include a communication unit 110 , a user interface 120 , a memory 130 and a processor 140 .

The communication unit 110 may provide a communication interface required to provide a transmission/reception signal between external devices in the form of packet data in conjunction with the network 400 . In addition, the communication unit 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals to and from other network devices through wired or wireless connections.

That is, the processor 140 may receive various data or information from an external device connected through the communication unit 110 and may transmit various data or information to the external device.

In one embodiment, the user interface 120 may include an input interface into which user requests and commands for controlling the operation of the behavior classification device 100 (eg, network parameter change, network learning condition change, etc.) are input. can

In one embodiment, the user interface 120 may include an output interface for outputting an action classification result or warning a user determined to have cheated. That is, the user interface 120 may output results according to user requests and commands. An input interface and an output interface of the user interface 120 may be implemented in the same interface.

The memory 130 may store various types of information necessary for the control (operation) of the operation of the behavior classification device 100 and store control software, and may include a volatile or non-volatile recording medium.

The memory 130 is connected to one or more processors 140 by an electrical or internal communication interface and, when executed by the processor 140, causes the processor 140 to control the behavior classification device 100 (cause) Codes can be saved.

Here, the memory 130 may include non-temporary storage media such as magnetic storage media or flash storage media, or temporary storage media such as RAM, but the scope of the present invention is not limited thereto. The memory 130 may include built-in memory and/or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, Non-volatile memory such as NAND flash memory, or NOR flash memory, SSD. It may include a compact flash (CF) card, a flash drive such as an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as an HDD.

Also, information related to an algorithm for performing learning according to the present disclosure may be stored in the memory 130 . In addition, various information necessary within the scope of achieving the object of the present disclosure may be stored in the memory 130, and the information stored in the memory 130 may be updated as received from a server or an external device or input by a user. may be

The processor 140 may control overall operations of the behavior classification apparatus 100 . Specifically, the processor 140 is connected to the configuration of the action classification device 100 including the memory 130, and executes at least one command stored in the memory 130 to control the overall operation of the action classification device 100. can be controlled with

Processor 140 can be implemented in a variety of ways. For example, the processor 140 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor Processor, DSP) may be implemented as at least one.

The processor 140, as a kind of central processing unit, may control the operation of the behavior classification device 100 by driving control software loaded in the memory 130. The processor 140 may include any type of device capable of processing data. Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example.

3 is a diagram schematically showing a network structure of an action classification algorithm according to an embodiment, FIG. 4 is an example of keypoints inferred in a hand region detection network of an action classification algorithm according to an embodiment, and FIG. It is a diagram visualizing input data of an action detection network of an action classification algorithm according to an embodiment.

In an embodiment, the processor 140 may perform an algorithm for determining cheating in a user image of a non-face-to-face user (eg, a user taking an online test). The processor 140 may detect a non-face-to-face user's hand region and a key point of the hand in the non-face-to-face user image, and recognize cheating through an object captured in the image and a shape of the hand.

That is, the processor 140 may learn an action classification algorithm capable of recognizing cheating by detecting the hand region and keypoint of the non-face-to-face user in the non-face-to-face user image, and based on the previously learned action classification algorithm It is possible to determine whether to use a disallowed object according to hand tracking-based action classification.

Referring to FIG. 3 , in an embodiment, an action classification algorithm determines whether a disallowed object is used according to an action classification result through a hand region detection network that detects a hand region and a hand keypoint and an object captured in an image and a shape of the hand. It can be broadly classified into inferring behavior classification networks.

In one embodiment, an open source cross platform (eg, Mediapipe) may be used to detect a non-face-to-face user's hand region and hand keypoint.

This open-source cross-platform has a graph-based framework and provides machine learning solutions for live and streaming media.

Cross-platform, also called multi-platform, is a term that means that a computer program, operating system, computer language, programming language, computer software, etc. can run on many types of computer platforms. A cross-platform application can run on more than one platform. This kind of software is also called multiplatform software.

The open source cross-platform in one embodiment, for example, can provide solutions such as Face Detection, Face Mesh, Pose, Hands, Object Detection, etc., in particular, tracking the shape of the hand and fingers to determine cheating. A Hands solution is available. For example, Hands can use machine learning to infer 21 three-dimensional keypoints for each hand knuckle in one frame.

The processor 140 may perform, for example, 21 hand keypoint detection as shown in FIG. 4 based on the hand region detection network of the action classification algorithm. The hand region detection network of this action classification algorithm may also be configured in detail with two models (a hand detection model and a hand keypoint detection model).

Among them, the first hand detection model (eg, Palm Detection Model) can find the palm or the back of a person's hand in an image. For example, a network model for finding the palm or the back of the hand may receive as input a 128 X 128 X 3 tensor with an RGB value of [-1.0, 1.0] and output a 1 X 896 X 18 tensor.

In order to infer the key point of the hand, first, the position where the hand appears in the entire image must be detected. However, it is difficult to accurately locate the hand region without being affected by different hand sizes, complex hand shapes, and degree of occlusion. In the case of the face, it has patterns that are easy to distinguish, such as eyes and mouth, but it is difficult to find such visual features in the hand.

Accordingly, the processor 140 may first detect an area where shape changes are small, such as a palm or a fist. Since the palm is a smaller object than the whole hand, the non-maximum suppression algorithm can work relatively well. In addition, since the horizontal to vertical ratio of the palm is approximately 1:1, the anchor box used in the non-maximum suppression algorithm can be simply defined as a square.

Object detection algorithms typically sample a large number of regions, determine whether these regions contain an object of interest, and adjust the edges of the regions to accurately predict the ground truth Bounding Boxes of the object. do. To find these bounding boxes, the anchor box method can be used.

In the anchor box method, bounding boxes with different sizes and aspect ratios can be created with each pixel at the center, and such bounding boxes are called anchor boxes, through which an object is detected.

At this time, the processor 140 may expand the palm region of interest (ROI) detected in the non-face-to-face user image to a size sufficient to sufficiently cover the shape when all fingers are spread out. In other words, the ROI of the palm area is expanded to the ROI of the entire hand area.

The second hand keypoint detection model (eg, Hand Landmark Model) receives the ROI image of the entire extended hand region as an input and can predict 3D coordinate values for keypoints of the hand. Here, the ROI image may be input as a color image (RGB image).

At this time, the processor 140 rotates a line segment connecting the center of the wrist and the middle finger in the ROI image parallel to the y-axis of the image in order to lower the capacity of the hand region detection network of the action classification algorithm. can

According to an embodiment, in an entire pipeline for processing input of continuous images, a hand tracking operation may be performed in such a way that hand position information identified in a previous frame is also used in a next frame. At this time, if the processor 140 cannot find the hand in the given ROI image, it can perform again from the process of detecting the palm or the back of the hand in the user image.

In one embodiment, since the capacity of the network of the behavior classification algorithm is reduced through the above process, it can be operated in real time even in a mobile environment.

Hereinafter, the action classification network of the action classification algorithm will be described. The behavior classification network determines whether the hand appearing in the current frame is using a disallowed object with the ROI image and keypoint information of the hand detected in the non-face-to-face user image, and can be called a kind of classifier.

In one embodiment, pretrained MobileNetV2 may be used as a backbone network for real-time performance of the behavior classification network of the behavior classification algorithm. MobileNet is a model that greatly reduces the amount of computation by introducing the concept of Depthwise Separable Convolution, and aims to operate sufficiently well even in an environment with somewhat insufficient computing resources, such as a mobile device. The next version, MobileNetV2, borrows the Inverted Residual structure and reduces the amount of computation more than the existing MobileNet.

Therefore, in one embodiment, MobilenetV2 learned with ImageNet can be used in consideration of the learning time and the similarity of data sets.

In one embodiment, the action classification network of the action classification algorithm may receive 160 X 160 X 24 tensors as input. In this case, the first three channels are RGB channels of the hand ROI image, and the remaining 21 channels that follow may be a heatmap in which 21 key points are marked one by one.

Referring to FIG. 5, FIG. 5(b) shows keypoint 0 for the hand in the hand ROI image shown in FIG. 5(a), and FIG. 5(c) shows the hand shown in FIG. 5(a). It shows 21 key points for the hand in the ROI image. 5(d) and 5(e) are visualized on the hand ROI image by combining the key points of each channel into one channel.

In one embodiment, a pipeline in which several nodes are connected like a graph and operate can continuously process image frames transmitted in real time. The inputs of the behavioral classification network can be made real-time within this pipeline.

More specifically, a node executing an action classification network may first receive a 160 X 160 X 3 hand ROI image (RGB image) as an input from a node that generates an ROI image of a size including all hands. In addition, the network 140 may receive coordinate values of keypoints as packets from a node that infers 21 keypoints by giving the same ROI image as an input of the hand keypoint detection network.

The processor 140 may generate an empty grid having the same size as the ROI image for each of the 21 keypoints, mark coordinates of the keypoints, and apply Gaussian blur. When 21 heat maps with one keypoint are created as described above, the processor 140 can configure a total of 24 channels of input tensors by adding them to the back of the RGB channels of the hand ROI image received as a packet.

In one embodiment, the 160 X 160 X 24 input tensor may be given as an input to a behavior classification network. For example, an action classification network that uses the input layer pretrained with ImageNet without modification can receive 160 X 160 X 3 tensors as input.

Therefore, in one embodiment, the action classification network can adjust the input tensor to a size of 160 X 160 X 3 by placing a 1 X 1 2D convolution layer at the front end. In addition, in one embodiment, the output tensor has a value encoded with each pixel of the ROI image and 21 keypoint information in the corresponding pixel by a 1 X 1 convolution operation that converts 24 channels into 3 channels.

That is, in one embodiment, the 160 X 160 X 3 tensor generated as described above passes through the next layers. And in one embodiment, the last fully-connected layer of the action classification network may be composed of an FC layer having the number of outputs suitable for the class for action classification.

6 is an exemplary diagram of a structure of an action classification algorithm that takes a predetermined frame section of a user image as an input according to an embodiment.

Referring to FIG. 6 , in an embodiment, when processing input of consecutive images, a motion of a hand is predicted using only one image frame, and position information of the hand identified in the previous frame is transferred to the next frame. The hand is traced in a way that is also used in the frame.

However, if the model's backbone network (for example, MobileNetV2) is changed to another network, various scenarios suitable for the new backbone network can be applied.

First, the action classification network of the action classification algorithm can be replaced with a model that receives a video input, that is, a certain frame section as an input. If you switch to a network used in the video classification field, such as SlowFast, I3D, or X3D, a certain section of non-face-to-face user images received in real time is defined as an input and the user's behavior within that section is defined. can do. In this case, the entire input/output pipeline of the action detection network can be configured as shown in FIG. 6 .

7 is a diagram schematically illustrating a network structure of an action classification algorithm including a gaze tracking network according to an embodiment, and FIG. 8 is an exemplary diagram illustrating a result of classification of an action including gaze tracking according to an embodiment.

Referring to FIGS. 7 and 8 , in one embodiment, eye tracking may be additionally performed to detect cheating.

The processor 140 may track the user's gaze in an image including a frontal or side face. That is, the processor 140 may derive the gaze direction in the 3D space by using the position of the iris in the image space. Further, the processor 140 may implement a process of recognizing a specified specific gaze gazing pattern, such as screen gazing and side gazing.

In other words, given the two-dimensional position of the iris in the image space and the posture of the face in the three-dimensional space in the camera space, the processor 140 can detect a gaze in the three-dimensional space and recognize a stable gaze-gaze pattern.

That is, in one embodiment, it is possible to determine whether or not cheating by combining gaze tracking information in addition to tracking the hand of a non-face-to-face user. In other words, the processor 140 may calculate not only information about the hand of the non-face-to-face user but also where the gaze is directed.

The processor 140 infers key points of a person's face captured in the image through a gaze detection network (eg, an Iris graph, one of the functions provided by Mediapipe), and uses the key points corresponding to the contours of the eyes from among them. Area images can be cropped.

In addition, the processor 140 may predict the pupil (iris) and key points for the pupil again from the cropped eye region image. Since the iris keypoint has 3-dimensional space coordinates, a vector indicating which direction the pupil of the non-face-to-face user is facing within the space in the image can be calculated through the cross product of the keypoint values of the iris and the pupil.

The processor 140 may determine whether the non-face-to-face user is looking outside the range of the monitor roughly determined in advance through the user's gaze vector.

In addition, the processor 140 may additionally define a bounding box occupied by a hand in a 3D space with a key point for a non-face-to-face user's hand. In other words, the processor 140 may calculate whether the gaze vector of the non-face-to-face user is facing the bounding box in the 3D space. As the action taken by the hand at this time, the result of whether or not to use the non-permissible object predicted through the above-described action classification algorithm can be used.

For example, as a result of determining whether to use a non-permissible object, if the result is that the non-face-to-face user is using a mobile phone, if the value that the non-face-to-face user's gaze is also directed toward the hand is obtained, it is more effective to determine whether or not to use a non-permissible object. can

More specifically, the processor 140 may detect the two-dimensional position of the iris of both eyes in the image space and the three-dimensional face posture in the camera space with respect to the face included in the user image. The processor 140 may use this to determine the gaze direction in the 3D space and perform gaze tracking to recognize gaze patterns such as screen gaze and side gaze.

Processor 140 may track a face based on a given image. The processor 140 may detect a two-dimensional position of a predefined finite keypoint on the face. For example, the processor 140 may track a face using a learning model trained to reach a certain level of accuracy by giving a label to each keypoint through a learning-based model such as a convolutional neural network (CNN).

The processor 140 can infer the two-dimensional position of the iris by determining the iris keypoint, and when a canonical landmark mesh of the face exists in the three-dimensional space, the face is defined as a Procrustes problem. The three-dimensional posture of can be inferred. Here, the Procrutesian problem is to find an orthogonal matrix that maps a given set of points closest to another given set of points. The one-to-one correspondence of points between the two sets must be known a priori.

When the focal length of a camera used to capture an image is known, the processor 140 may define a perspective projection matrix for transforming a camera space into an image space. Further, the processor 140 may calculate the inverse of the projection matrix and calculate a straight line on which the position of the iris in camera space is placed using the previously obtained position of the iris in the image space.

, 형상비(aspect ration)

, 주요 포인트(principal point) (

,

)가 주어졌을 때, 원근 투영 행렬

는 다음 수학식 1과 같이 정의될 수 있다.If this is explained with reference to the equation, the focal length

, aspect ratio

, the principal point (

,

) is given, the perspective projection matrix

Can be defined as in Equation 1 below.

를 계산하여, 변수

를 기반으로 다음 수학식 2와 같이, 이미지 공간 상의 홍채의 위치

를 통해 카메라 공간에서 원점과 홍채의 위치

를 통과하는 직선을 도출할 수 있다.Processor 140 is the inverse of the perspective projection matrix

By calculating the variable

Based on Equation 2, the position of the iris on the image space

The location of the origin and iris in camera space via

A straight line passing through can be derived.

의 적절한 값을 찾아서 카메라 공간 상의 홍채의 위치

를 구할 수 있다.In one embodiment,

The position of the iris in camera space by finding an appropriate value of

can be obtained.

In addition, the processor 140 calculates the position of the iris in the camera space by obtaining the intersection of the straight line passing through the center of the camera and the center of the iris and the geometry modeling the eye surface defined in the face space based on the given face posture. can be calculated.

를 구하기 위해서는 얼굴 공간에서 정의된 눈의 표면 기하를 카메라 공간으로 변환하고, 변환된 기하와 직선의 교점을 찾을 수 있다.Processor 140 represents an intersection with the surface geometry of the eye in camera space.

In order to obtain , the surface geometry of the eye defined in face space can be converted to camera space, and the intersection point of the converted geometry and straight line can be found.

, 눈 평면의 법선

이 주어졌을 때, 이 값들(

및

)을 얼굴의 자세를 나타내는 회전 행렬

, 이동 벡터

를 통해 카메라 공간으로 변환하여 다음 수학식 3을 만족하는

를 찾을 수 있다.Processor 140 calculates a point on one eye plane in face space.

, the normal of the eye plane

Given , these values (

and

) is a rotation matrix representing the pose of the face.

, the moving vector

Converted to camera space through

can be found.

는 다음 수학식 4와 같이 계산될 수 있다.that satisfies this formula

Can be calculated as in Equation 4 below.

가 주어진다고 가정하였을 때, 카메라 공간에서의 시선의 방향

는 다음 수학식 5와 같이 정의할 수 있다.The processor 140 may calculate the gaze direction in the camera space using the difference between the position of the center of the eyeball and the iris. That is, in one embodiment, the central position of one eyeball in the facial space

Assuming that is given, the direction of gaze in camera space

Can be defined as in Equation 5 below.

In addition, the processor 140 may calculate the position of the gaze placed on the predetermined screen area. At this time, assuming that the Y-axis of rotation of the face is parallel to the plane of the screen, the plane of the screen is calculated and the intersection point with the straight line forming the line of sight can be obtained.

이 다음 수학식 6에서와 같이 정의된다고 했을 때, 카메라 공간에서 화면 좌표계로 변환하는 행렬

을 정의할 수 있다. 화면 좌표계는 카메라 좌표계의 X축의 회전으로 정의되며, 원점의 위치를 서로 공유할 수 있다.In one embodiment, the pose of the face in camera space

Assuming that this is defined as in Equation 6 below, a matrix that converts from camera space to screen coordinate system

can define The screen coordinate system is defined as the rotation of the X axis of the camera coordinate system, and the location of the origin can be shared with each other.

을 이용하여 화면 상의 시선의 위치를 구하는 수식은 다음 수학식 7과 같다. 그리고

는 시선을 이루는 직선 상의 점 중 Z 성분이 0이 되는 것으로 정의될 수 있다.In one embodiment, the transformation matrix

The formula for obtaining the position of the gaze on the screen using Equation 7 is shown in Equation 7 below. and

may be defined as having a Z component of zero among points on a straight line constituting the line of sight.

가 미리 정한 화면의 축 정렬 경계상자(axis-aligned bounding box) 내에 들어오는지 검사하여 화면을 주시하는지 판단할 수 있다. 그리고 프로세서(140)는 시선의 방향을 나타내는 벡터

를 정해진 기준 벡터(reference vector)

와의 사잇각

을 알아내어, 일정 임계치가 넘었을 때 측면을 주시한다고 판단할 수 있다. 사잇각을 구하는 수식은 다음 수학식 8과 같이 나타낼 수 있다.The processor 140 is a point on a plane parallel to the screen detected above.

It can be determined whether the user is watching the screen by examining whether or not is within the axis-aligned bounding box of the screen which is pre-determined. And the processor 140 is a vector representing the direction of the gaze

to a set reference vector (reference vector)

angle between

By finding out, it can be determined that the user is looking at the side when a certain threshold value is exceeded. The formula for obtaining the included angle can be expressed as in Equation 8 below.

Hereinafter, experimental results for implementation performance of the behavior classification algorithm will be described with reference to FIGS. 9 to 12 .

9 is a diagram showing the experimental results of the behavior classification algorithm (loss and accuracy graph of the first data set) according to an embodiment, and FIG. 10 is the experimental result of the behavior classification algorithm according to an embodiment (when only two classes were learned). 11 is a diagram showing an experiment result (loss and accuracy graph of the second data set) of an action classification algorithm according to an embodiment, and FIG. 12 is a diagram showing an action classification algorithm according to an embodiment. This is a diagram showing the experimental results (the error matrix when learning using 5 classes).

In one embodiment, an experiment was conducted to verify the performance of the behavior classification device 100. For example, the first data set and the second data set may be used for network training of an action classification algorithm.

The first data set is an image frame extracted from a video taken by three graduate students. There are only two classes, one using normal and one non-permitted object, and the training data set consists of 10,766 frames and the test data set consists of 5,054 frames by combining both classes. Since frames are extracted at regular intervals from videos taking only one of the actions of using a mobile phone corresponding to normal or non-permitted objects, the amount of data set is large, but the change in action is not large.

The second data set is a more extensive version of the data set in which class types and the number of participants are increased compared to the first data set. It consists of 198 images per 18 participants for 5 classes (normal, keyboard typing, cell phone use, handwriting, reference book use).

Normal classes are behaviors that can be taken naturally while taking an online test, such as clasping hands, touching ears, hair, forehead, and collar, resting chin, and putting hands on the desk without taking any action. This is included.

Keyboard typing classes include keyboards such as mechanical and membrane used in general desktops and laptop keyboards. The mobile phone use class was used with either the left or right hand, and both hands were used, and the handwriting class was data taken when writing on paper or notebooks using only a pencil, ballpoint pen, or mechanical pencil. The last reference book use class consists of turning the pages and holding the book with both hands.

In one embodiment, some limitations are placed on filming so that the images collected as a data set can produce a scene that is as similar as possible to an actual scenario, an online test situation. Eighteen participants took pictures of themselves using a camera installed at a random location in a place where they judged that they could take the test alone without the intervention of others.

At this time, the camera was installed by applying the concept of latitude and longitude to diversify the location of the installed camera and the angle at which the camera faces the participant. In other words, latitude means the height of the location where the camera is placed, and longitude means whether the camera is placed on the left, front, or right side of the participant. Participants must shoot with a combination of three latitude and three longitude settings of the camera for one class.

Meanwhile, the experimental conditions of an embodiment may be set as follows. Using tensorflow 2.3, you can learn 20 epochs on the first data set and 100 epochs on the second data set. Also, the batch size can be set to 8, the optimization can be set to Adam optimizer, the initial running rate is set to 0.3, and if the verification accuracy does not improve during 2 epochs, it can be set to decrease by 0.5 times.

First, looking at the case of learning using the first data set, in one embodiment, NVIDIA TITAN Xp was used, and it took about 19 hours for learning. The loss during learning changed as shown in the graph shown in Fig. 9(a), and the change in accuracy appeared as shown in Fig. 9(b).

In addition, referring to FIG. 9(c), the verification loss measured with the test data set for each epoch can be checked, and the verification accuracy can be checked with reference to FIG. 9(d). In one embodiment, it can be confirmed that an accuracy of 88.8% was achieved in the test data set after learning was completed.

10 shows an error matrix capable of confirming true positive, true negative, false positive, and false negative, and the learning result of the first data set can be confirmed through FIG. 10 (a).

Next, examining the case of learning using the second data set, it can be largely divided into two types.

The first is to train binary classification using only the normal class and the mobile phone use class, the same as the data included in the first data set, and the second is to train multi-class classification using all five classes of the second data set. it was learned

In an embodiment, for fairness of learning data, the number of data for each class of the second data set is equal to the smallest class. So, in the first case where only the normal class and the mobile phone use class were learned, the number of training data is 790 each, for a total of 1,580. data is used

In the case of the model trained with only two classes, normal and mobile phone usage, the accuracy in the test data set was 86.6%, and the error matrix at this time can be confirmed through FIG. 10(b).

In one embodiment, when learning is performed using the first data set, other images of people in the training data set may be included in the test data set. When learning is performed using the second data set, the test data set may be configured only with persons not included in the training data set.

Therefore, the model trained with the second data set was tested on a person I never saw, but the accuracy did not drop significantly compared to when it was trained using the first data set, which consisted of only three people and overfitting occurred. It can be seen that the generalization performance is better.

In one embodiment, the change in loss and accuracy can be confirmed through FIG. 11, and the training took about 7 hours and used an NVIDIA GeForce RTX 2080 SUPER.

On the other hand, in the case of the model trained on all five classes, the accuracy in the test data set was 47.1%. As described above, the number of data used is 1,600 in total because the number of data is equally matched to the class with the smallest amount of data for fairness of training data.

Although the number of data is almost the same as the model trained only with normal and mobile phone usage, it may not be trained enough because there are as many as 5 classes. If more data are collected for each class and training is conducted, the performance of the model will be sufficiently improved.

12 shows the error matrix of a model trained with 5 classes. In one embodiment, training took about 10 hours and NVIDIA TITAN Xp was used.

13 is a flowchart illustrating a method of classifying an action according to an exemplary embodiment.

Referring to FIG. 13 , in step S100, the behavior classification apparatus 100 acquires an image of a non-face-to-face user.

In step S200, the action classification apparatus 100 classifies the action of the non-face-to-face user by detecting the hand region of the non-face-to-face user image based on the previously learned action classification algorithm.

At this time, the action classification apparatus 100 may detect a hand region image and hand keypoint information based on the acquired non-face-to-face user image.

That is, the behavior classification apparatus 100 may detect a region of interest (ROI) image of the entire region of the hand in the non-face-to-face user image, and predict three-dimensional coordinate values for each keypoint of the hand in the detected ROI image of the entire region of the hand. there is.

In addition, the action classification apparatus 100 may detect an ROI image of the palm or back of the hand region in the non-face-to-face user image, and expand the ROI of the palm or back of the hand region to the ROI of the entire hand region.

In addition, the action classification apparatus 100 determines the position of the center of the wrist and the middle finger in the ROI image of the entire area of the detected hand, and rotates a line segment connecting the center of the wrist and the middle finger in parallel with the y-axis of the image to obtain the detected hand An ROI image of the entire area can be obtained.

Further, the action classification apparatus 100 may classify a non-face-to-face user's action by detecting an object in the hand region and a posture of the hand based on the hand region image and hand keypoint information.

At this time, the action classification apparatus 100 may determine the posture of the hand of the non-face-to-face user based on the ROI image and the 3D coordinate values for each keypoint of the hand.

In step S300, the action classification apparatus 100 determines whether the non-face-to-face user uses a disallowed object based on the action classification result.

That is, in an embodiment, when a non-face-to-face user image is input, the pre-learned behavior classification algorithm queries the ROI image of the entire hand area and hand keypoint information detected based on the non-face-to-face user image as input, , it may be a learning model learned to output whether or not to use a disallowed object according to an object in the hand region and an action class mapped to the posture of the hand.

This pre-learned behavior classification algorithm is trained through a training phase, in which, when a non-face-to-face user image is input, the ROI image of the entire hand region is detected in the non-face-to-face user image; Predicting 3D coordinate values for each keypoint of the hand in the ROI image of the entire region of one hand, and detecting an object in the hand region and a posture of the hand based on the 3D coordinate values for each keypoint of the hand and the ROI image. , Classifying the non-face-to-face user's behavior, and inferring whether the non-face-to-face user uses a non-allowed object according to the behavior classification result.

At this time, the step of classifying the behavior of the non-face-to-face user in the training phase is the step of generating an empty grid having the same size as the ROI image of the entire hand area, and the empty grid based on the three-dimensional coordinate values for each keypoint of the hand. The method may include marking each keypoint coordinate point and applying Gaussian blur to a grid marked with each keypoint coordinate point.

In addition, the step of classifying the behavior of the non-face-to-face user in the training phase is based on a channel based on the color value of the ROI image of the entire hand region and an input tensor composed of channels for each keypoint, Classifying the behavior of the non-face-to-face user steps may be included.

14 is a flowchart illustrating a method of classifying an action including gaze tracking according to an exemplary embodiment.

In one embodiment, after performing steps S100 to S300 according to FIG. 13, an additional process according to gaze information may be performed.

That is, in steps S100 to S300, when it is determined that the non-face-to-face user uses a disallowed object based on the result of the non-face-to-face user's action classification based on the hand region image and the hand keypoint information, in step S400, the action is classified. The device 100 determines whether the non-face-to-face user's gaze direction is toward the hand position.

At this time, the behavior classification apparatus 100 may detect a face region image and face keypoint information based on the acquired non-face-to-face user image, and determine the direction of the gaze of the non-face-to-face user based on the face region image and face keypoint information. there is.

In addition, the action classification apparatus 100 derives the two-dimensional position of the iris in the image space and the pose of the face in the three-dimensional space in the camera space based on the face region image and the facial key point information, and through the direction of the iris in the three-dimensional space The direction of the gaze of the non-face-to-face user may be detected.

In step S500 thereafter, when the non-face-to-face user's line of sight direction is toward the hand position, the action classification apparatus 100 may additionally determine that the non-face-to-face user is using a disallowed object.

That is, in one embodiment, it is possible to determine whether or not cheating by combining gaze tracking information in addition to tracking the hand of a non-face-to-face user. In other words, the processor 140 may calculate not only information about the hand of the non-face-to-face user but also where the gaze is directed.

For example, as a result of determining whether to use a non-permissible object, if the result is that the non-face-to-face user is using a mobile phone, if the value that the non-face-to-face user's gaze is also directed toward the hand is obtained, it is more effective to determine whether or not to use a non-permissible object. can

However, in one embodiment, it is not limited to this, and the non-face-to-face user's action classification result based on the hand region image and hand keypoint information and the non-face-to-face user's gaze direction based on the face region image and face keypoint information Based on this, it is possible to determine whether a non-face-to-face user uses a non-allowed object.

That is, according to an embodiment, if it is determined that the user has used a disallowed object according to the hand tracking-based action classification result, the action classification result is performed based on the information on the hand and the information on the gaze, rather than additionally detecting the gaze direction. It may be inferred to determine whether to use a disallowed object.

Meanwhile, in step S600, the behavior classification apparatus 100 may output a warning to the non-face-to-face user according to the non-allowed object use determination.

In one embodiment, a warning may be output to a terminal being used by a non-face-to-face user, or a warning may be output to a location where a non-face-to-face user is located, and at the same time, a notification for a user who has performed an unacceptable action may be output to an administrator terminal. can

The warning method is not specifically limited, and according to an embodiment, a warning output after determining hand tracking-based cheating and a warning output after additionally determining eye-tracking-based cheating may be different.

Embodiments according to the present disclosure described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the purpose of the present disclosure, or may be known and usable to those skilled in the art in the field of computer software. An example of a computer program may include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

In the specification of the present disclosure (particularly in the claims), the use of the term "above" and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present disclosure, as including the invention to which individual values belonging to the range are applied (unless otherwise stated), each individual value constituting the range is described in the detailed description of the invention Same as

Unless an order is explicitly stated or stated to the contrary for steps comprising a method according to the present disclosure, the steps may be performed in any suitable order. The present disclosure is not necessarily limited to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in this disclosure is simply to explain the present disclosure in detail, and the scope of the present disclosure is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present disclosure should not be limited to the above-described embodiments, and not only the claims to be described later, but also all ranges equivalent to or equivalent to these claims are within the scope of the spirit of the present disclosure. will be said to belong to

1: Image Restoration System
100: image restoration device
110: Communication Department
120: user interface
130: memory
140: processor
200: user terminal
300: server
400: network

Claims (20)

A hand tracking-based action classification method for determining an action of a non-face-to-face user based on a hand image of a user using content in a non-face-to-face manner, at least part of each step being performed by a processor,
Obtaining an image of a non-face-to-face user;
Classifying the behavior of the non-face-to-face user by detecting a hand region of the non-face-to-face user image based on a pre-learned behavior classification algorithm; and
Based on the action classification result, determining whether the non-face-to-face user uses a non-allowed object,
Hand tracking-based behavior classification method. According to claim 1,
The step of classifying the behavior of the non-face-to-face user,
Detecting a hand region image and hand keypoint information based on the acquired non-face-to-face user image; and
Classifying an action of the non-face-to-face user by detecting an object in the hand region and a posture of the hand based on the hand region image and the hand keypoint information,
Hand tracking-based behavior classification method. According to claim 2,
The step of detecting the hand region image and hand keypoint information,
detecting a region of interest (ROI) image of an entire hand region from the non-face-to-face user image; and
Predicting three-dimensional coordinate values for each hand keypoint in the ROI image of the entire detected hand region,
Hand tracking-based behavior classification method. According to claim 3,
The step of detecting the ROI image of the entire region of the hand,
detecting an ROI image of a palm or back of the hand region in the non-face-to-face user image; and
Expanding the ROI of the palm or back of the hand region to the ROI of the entire hand region,
Hand tracking-based behavior classification method. According to claim 4,
The step of detecting the ROI image of the entire region of the hand,
detecting positions of the center of the wrist and the middle finger in the detected ROI image of the entire region of the hand; and
Obtaining an ROI image of the entire detected hand region by rotating the identified line segment connecting the center of the wrist and the middle finger in parallel with the y-axis of the image,
Hand tracking-based behavior classification method. According to claim 4,
The step of classifying the behavior of the non-face-to-face user,
Determining the posture of the hand of the non-face-to-face user based on the ROI image and the three-dimensional coordinate values for each of the hand keypoints,
Hand tracking-based behavior classification method. According to claim 1,
The pre-learned behavior classification algorithm,
When the non-face-to-face user image is input, the ROI image of the entire hand region and hand keypoint information detected based on the non-face-to-face user image are queried as input, and the action mapped to the object in the hand region and the posture of the hand A learning model learned to output whether or not to use a disallowed object according to the class,
Hand tracking-based behavior classification method. According to claim 7,
The pre-learned behavior classification algorithm is trained through a training phase,
The training phase is
detecting an ROI image of an entire hand region from the non-face-to-face user image when the non-face-to-face user image is input;
predicting three-dimensional coordinate values for each hand keypoint in the detected ROI image of the entire region of the hand;
Classifying an action of the non-face-to-face user by detecting an object in the hand region and a posture of the hand based on the ROI image and the three-dimensional coordinate values for each keypoint of the hand; and
Including the step of inferring whether the non-face-to-face user uses a non-allowed object according to the action classification result,
Hand tracking-based behavior classification method. According to claim 8,
Classifying the behavior of the non-face-to-face user in the training phase,
generating an empty grid having the same size as the ROI image of the entire region of the hand;
marking each keypoint coordinate point on the empty grid based on the three-dimensional coordinate value for each keypoint of the hand; and
Applying Gaussian blur to a grid marked with each keypoint coordinate point,
Hand tracking-based behavior classification method. According to claim 9,
Classifying the behavior of the non-face-to-face user in the training phase,
Classifying the action of the non-face-to-face user based on an input tensor composed of a channel based on a color value of the ROI image of the entire hand region and a channel for each keypoint,
Hand tracking-based behavior classification method. According to claim 1,
When it is determined that the non-face-to-face user uses a disallowed object based on the non-face-to-face user's action classification result based on the hand region image and the hand keypoint information, the non-face-to-face user's gaze direction is the direction of the hand determining whether it is facing a location; and
Further comprising the step of additionally determining that the non-face-to-face user is using a non-allowed object when the direction of the non-face-to-face user's gaze is toward the position of the hand,
Hand tracking-based behavior classification method. According to claim 11,
Determining whether the non-face-to-face user's gaze direction is toward the hand position,
detecting a face region image and face keypoint information based on the acquired non-face-to-face user image; and
Determining the direction of the gaze of the non-face-to-face user based on the face region image and the face keypoint information,
Hand tracking-based behavior classification method. According to claim 12,
Determining the direction of the gaze of the non-face-to-face user,
deriving a two-dimensional position of the iris in an image space and a face posture in a three-dimensional space in a camera space based on the face region image and the facial keypoint information; and
Detecting the direction of the gaze of the non-face-to-face user through the direction of the iris on the three-dimensional space,
Hand tracking-based behavior classification method. According to claim 1,
The step of determining whether the non-face-to-face user uses a non-allowed object,
Based on the non-face-to-face user's action classification result based on the hand region image and the hand keypoint information and the gaze direction of the non-face-to-face user based on the face region image and face keypoint information, the non-face-to-face user Further comprising the step of determining whether to use a non-permissible object of
Hand tracking-based behavior classification method. According to claim 1,
Further comprising outputting a warning to the non-face-to-face user according to the non-allowed object use determination.
Hand tracking-based behavior classification method. A hand tracking-based action classification device for determining a non-face-to-face user's action based on a hand image of the user using non-face-to-face content,
Memory; and
a processor coupled with the memory and configured to execute computer readable instructions contained in the memory;
The at least one processor,
Obtaining an image of a non-face-to-face user;
Based on a pre-learned behavior classification algorithm, detecting a hand region of the non-face-to-face user image and classifying the behavior of the non-face-to-face user; and
Based on the action classification result, it is set to perform an operation of determining whether the non-face-to-face user uses a non-allowed object
A hand tracking-based behavior classification device. 17. The method of claim 16,
The operation of classifying the behavior of the non-face-to-face user,
An operation of detecting a hand region image and hand keypoint information based on the acquired non-face-to-face user image, and
Based on the hand region image and the hand keypoint information, detecting an object in the hand region and a posture of the hand, and classifying the action of the non-face-to-face user,
A hand tracking-based behavior classification device. 17. The method of claim 16,
The pre-learned behavior classification algorithm,
When the non-face-to-face user image is input, the ROI image of the entire hand region and hand keypoint information detected based on the non-face-to-face user image are queried as input, and the action mapped to the object in the hand region and the posture of the hand A learning model learned to output whether or not to use a disallowed object according to the class,
A hand tracking-based behavior classification device. 17. The method of claim 16,
The at least one processor,
When it is determined that the non-face-to-face user uses a disallowed object based on the non-face-to-face user's action classification result based on the hand region image and the hand keypoint information, the non-face-to-face user's gaze direction is the direction of the hand an operation to determine whether the location is facing, and
When the non-face-to-face user's gaze direction is toward the hand position, further determining that the non-face-to-face user is using a non-allowed object is further performed,
A hand tracking-based behavior classification device. 17. The method of claim 16,
The operation of determining whether the non-face-to-face user uses a non-allowed object,
Based on the non-face-to-face user's action classification result based on the hand region image and the hand keypoint information and the gaze direction of the non-face-to-face user based on the face region image and face keypoint information, the non-face-to-face user Further comprising the operation of determining whether to use a non-permissible object of
A hand tracking-based behavior classification device.

Images