KR20220153929A - Deep learning method and apparatus based on tsn for real-time hand gesture recognition in video - Google Patents

Abstract

Disclosed are a method and an apparatus for temporal segment network (TSN)-based deep learning for real-time hand gesture recognition in a video. According to an embodiment, a method for TSN-based deep learning for real-time hand gesture recognition in video is performed by a computer device. The method comprises the steps of: segmenting the entire video according to a time series image or a spatial image in a TSN for hand gesture recognition; performing deep learning on the time series image and deep learning on the spatial image, respectively; integrating the time series images on which the deep learning is performed and the spatial images on which the deep learning is performed; and performing final classification using the integrated spatial images and the integrated time series images. The present invention recognizes motions in which several 2D actions are connected in the 3D video and recognizes hand gestures in real time based on the deep learning.

Description

TSN-based deep learning method and apparatus for real-time hand gesture recognition in video

The following embodiments relate to a hand gesture recognition method and apparatus, and more specifically, to a TSN-based deep learning method and apparatus for real-time hand gesture recognition in a video.

Mobile phones are a huge part of our lives, and the percentage of Internet traffic generated by mobile phones is surpassing that of desktops. Because of this shift, the way humans interact with these devices has also evolved from keyboard/mouse to gestures, voice and brain computer interfaces. For example, in the case of a voice interface, a combination of a hand gestural interface and voice is currently used because the voice interface is inaccurate or difficult to use in a noisy outdoor environment or in a quiet environment.

In addition, there is an increasing demand to replace keyboard or panel touch input by using recognition of various hand gestures as program inputs. For example, if hand gesture recognition can be replaced with a command for autonomous driving in the course of driving an autonomous vehicle with a complex surrounding environment, the risk of an accident can be reduced because actions such as pressing a button can be avoided.

Existing TSN (Temporal Segment Networks) techniques take the entire video as an input and output the subject of the video, but these techniques are not suitable for applications that recognize hand gestures in real time and use them as commands. Therefore, a technique capable of recognizing and outputting hand gestures for each time interval is required.

Korean Patent Registration No. 10-2121654 describes a deep learning-based automatic gesture recognition method and system.

Korean Patent Registration No. 10-2121654

Embodiments describe a TSN-based deep learning method and apparatus for real-time hand motion recognition in a video. It provides real-time deep learning technology based on TSN (Temporal Segment Networks), a hand gesture recognition technique.

Embodiments are to provide a TSN-based deep learning method and apparatus for recognizing hand gestures in real time in a video, capable of recognizing motions in which several 2D actions are connected in a 3D video and recognizing hand gestures in real time based on deep learning.

A deep learning method based on Temporal Segment Networks (TSN) for real-time hand gesture recognition among videos performed by a computer device according to an embodiment includes the steps of zoning an entire video according to a time-series image or a spatial image in the TSN for hand gesture recognition. ; performing deep learning on the time series image and deep learning on the spatial image, respectively; integrating time-series images on which deep learning is performed and integrating spatial images on which deep learning is performed; and final classification using the integrated spatial images with the integrated time-series images.

The zoning may include dividing a video into a plurality of short snippets; and zoning each of the short pieces of information according to a time-series image or a spatial image.

In the final classification step, it is possible to classify whether the video includes a hand gesture using information pre-stored using the integrated time-series images and the integrated spatial images.

A deep learning apparatus based on Temporal Segment Networks (TSN) for real-time hand gesture recognition among videos according to another embodiment includes an image segmentation unit that zones an entire video according to a time-series image or a spatial image in the TSN for hand gesture recognition; an image deep learning unit that performs deep learning on the time series image and deep learning on the spatial image, respectively; an image integration unit that integrates time-series images on which deep learning has been performed and spatial images on which deep learning has been performed; and a hand gesture recognizing unit that finally classifies the spatial images integrated with the integrated time-series images.

The video segmentation unit may include a first segmentation unit that divides the video into a plurality of short snippets; and a second zoning unit for zoning each of the short pieces of information according to a time-series image or a spatial image.

According to the embodiments, it is possible to provide a TSN-based deep learning method and apparatus for recognizing hand gestures in real time in a video, which can recognize motions in which several 2D actions are connected in a 3D video and recognize hand gestures in real time based on deep learning. can

1A is a diagram schematically illustrating a TSN structure for real-time hand gesture recognition according to an embodiment.
1B is a diagram illustrating an example of a TSN structure for recognizing two streams of hand gestures in real time according to an embodiment.
2 is a module for explaining a 2D module according to an embodiment.
3 is a diagram for explaining a 3D module according to an exemplary embodiment.
4 is a flowchart illustrating a TSN-based deep learning method for hand gesture recognition according to an embodiment.
5 is a block diagram illustrating a TSN-based deep learning device for hand gesture recognition according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in many different forms, and the scope of the present invention is not limited by the embodiments described below. In addition, several embodiments are provided to more completely explain the present invention to those skilled in the art. The shapes and sizes of elements in the drawings may be exaggerated for clarity.

The following embodiments relate to a deep learning method and apparatus based on TSN for real-time hand gesture recognition in a video, and hand gesture recognition in a 3D video in which hand gesture recognition of a simple 2D image recognizes a motion in which several 2D actions are connected. We propose a real-time deep learning technology based on TSN (Temporal Segment Networks), which is a technique.

1A is a diagram schematically illustrating a TSN structure for real-time hand gesture recognition according to an embodiment.

Referring to FIG. 1A, in the TSN 130 for recognizing hand motion (hand motion), the entire video 110 is zoned, CNNs for time-series images and CNNs for the same spatial image are separately performed, and each is integrated (140). It goes through the process of final classification (150). Here, the entire video 110 can be divided into a plurality of short snippets 120 and then segmented into a time series image and the same spatial image.

For example, when the image V is divided into m regions, it may be divided into {S ₁ , S ₂ , ..., S _m }. At this time, TSN can be expressed as the following equation.

[Equation 1]

Here, T _m is a randomly sampled area among m areas, and F( T _m ; W) is a CNN deep learning function that takes T _m as an input and W as a parameter, and outputs the classification result. G is the zoning integration function, which serves to integrate the results of all zoning. H is a function that classifies which hand gesture the video contains.

1B is a diagram illustrating an example of a TSN structure for recognizing two streams of hand gestures in real time according to an embodiment.

Referring to Fig. 1b, the framework of the two-stream hand gesture recognition network (top) and the structure of the inflated 3D ResNet50 (bottom) are shown. As an example of an extended 3D module, a two-stream hand gesture recognition network based on 3D ResNets can be proposed. Hand gesture video input from the network framework is divided into segments, and short snippets can be selected from each segment for each frame. Here, = 2, = 24 can be set. Class scores from different short snippets can be fused to create a final prediction. The network has two inflated 3D ResNets and can share parameters. For each inflated 3D ResNet during training, we can run a softmax operation on the output of the fully connected layer and then use the cross entropy loss to compute the error between the model output and the label.

2 is a module for explaining a 2D module according to an embodiment. 3 is a diagram for explaining a 3D module according to an embodiment.

Referring to FIG. 2, an example of a 2D module (residual module) is shown, and referring to FIG. 3, an example of an extended 3D module is shown. Internally, each frame is a 2D image, and a method of converting a 2D image into a 3D image in a deep learning process may go through the following process. In a 2D process, all filters and pooling kernels can be extended to give a temporal dimension. Filters are usually N*N, and you can extend them to N*N*N. So we can add a temporal dimension to the 2D module and use it as an input to the learning module.

4 is a block diagram for explaining an example of an internal configuration of a computer system according to an exemplary embodiment.

For example, a TSN-based deep learning device for recognizing hand gestures according to embodiments of the present invention may be implemented through the computer system (device) 400 of FIG. 4 . As shown in FIG. 4, the computer system 400 includes a processor 410, a memory 420, a permanent storage device 430, a bus ( 440), an input/output interface 450 and a network interface 460.

Processor 410 may include or be part of any device capable of processing any sequence of instructions. Processor 410 may include, for example, a computer processor, a processor in a mobile device or other electronic device, and/or a digital processor. Processor 410 may be included, for example, in a server computing device, server computer, series of server computers, server farm, cloud computer, content platform, mobile computing device, smartphone, tablet, set top box, media player, and the like. Processor 410 may be connected to memory 420 through bus 440 .

Memory 420 may include volatile memory, permanent, virtual or other memory for storing information used by or output by computer system 400 . Memory 420 may include, for example, random access memory (RAM) and/or dynamic RAM (DRAM). Memory 420 may be used to store any information, such as state information of computer system 400. The memory 420 may also be used to store instructions of the computer system 400 including, for example, instructions for TSN-based deep learning for hand gesture recognition. Computer system 400 may include one or more processors 410 as needed or appropriate.

Bus 440 may include a communication infrastructure that enables interaction between the various components of computer system 400 . Bus 440 may carry data between components of computer system 400, for example, between processor 410 and memory 420, for example. Bus 440 may include a wireless and/or wired communications medium between components of computer system 400, and may include parallel, serial, or other topological arrangements.

Persistent storage device 430 is a component such as a memory or other persistent storage device as used by computer system 400 to store data for some extended period of time (e.g., relative to memory 420). may include Persistent storage 430 may include non-volatile main memory as used by processor 410 in computer system 400 . Persistent storage device 430 may include, for example, flash memory, hard disk, optical disk, or other computer readable media.

Input/output interface 450 may include interfaces to a keyboard, mouse, voice command input, display, or other input or output device. Configuration commands and/or information for TSN-based deep learning for recognizing hand gestures may be received through the input/output interface 450 .

Network interface 460 may include one or more interfaces to networks, such as a local area network or the Internet. Network interface 460 may include interfaces for wired or wireless connections. Configuration commands and/or information for TSN-based deep learning for hand gesture recognition may be received through the network interface 460 .

Also, in other embodiments, computer system 400 may include more components than those of FIG. 4 . However, there is no need to clearly show most of the prior art components. For example, the computer system 400 is implemented to include at least some of the input/output devices connected to the above-described input/output interface 450, or a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, It may further include other components such as databases and the like.

5 is a block diagram illustrating a TSN-based deep learning device for hand gesture recognition according to an embodiment.

FIG. 5 is a diagram illustrating an example of components that may be included in the processor 410 of the computer system 400 according to the embodiment of FIG. 4 . Here, the processor 410 of the computer system 400 may include the TSN-based deep learning device 500 for hand gesture recognition according to an embodiment. The TSN-based deep learning apparatus 500 for hand gesture recognition according to an embodiment includes an image zoning unit 510, an image deep learning unit 520, an image integration unit 530, and a hand gesture recognizing unit 540. It can be done. Here, the image zoning unit 510 may include a first zoning unit 511 and a second zoning unit 512 .

The processor 410 and components of the processor 410 may perform steps S110 to S140 included in the TSN-based deep learning method for hand gesture recognition of FIG. 6 . For example, the processor 410 and components of the processor 410 may be implemented to execute an instruction according to an operating system code included in the memory 420 and at least one program code described above. Here, the at least one program code may correspond to a code of a program implemented to process the water-based pollutant monitoring method.

The TSN-based deep learning method for hand gesture recognition may not occur in the shown order, and some of the steps may be omitted or additional processes may be further included.

6 is a flowchart illustrating a TSN-based deep learning method for hand gesture recognition according to an embodiment.

Referring to FIG. 6 , a deep learning method based on Temporal Segment Networks (TSN) for real-time recognition of hand gestures in a video performed by a computer device according to an exemplary embodiment converts the entire video into a time-series image or spatial image in TSN for hand gesture recognition. Segmenting according to images (S110), performing deep learning on time-series images and deep learning on spatial images, respectively (S120), integrating time-series images on which deep learning has been performed, and spatial images on which deep learning has been performed It may include a step of integrating (S130), and a final classification step (S140) using the integrated time-series images and the integrated spatial images.

Here, the step of zoning (S110) may include dividing the video into a plurality of short snippets, and zoning each of the short snippets according to a time-series image or a spatial image.

Below, each step of the TSN-based deep learning method for hand gesture recognition according to an embodiment will be described.

A TSN-based deep learning method for recognizing a hand gesture according to an embodiment may be described in more detail by taking a TSN-based deep learning apparatus for recognizing a hand gesture according to an embodiment as an example.

In step S110, the image segmentation unit 510 may zone the entire video according to time-series images or spatial images in the TSN for hand gesture recognition.

Here, the image zoning unit 510 may include a first zoning unit 511 and a second zoning unit 512 . More specifically, the first zoning unit 511 may divide the video into a plurality of short snippets. Then, the second zoning unit 512 may zone each piece of short information according to a time-series image or a spatial image.

In step S120, the image deep learning unit 520 may perform deep learning on time-series images and deep learning on spatial images, respectively.

In step S130, the image integrator 530 may integrate time-series images on which deep learning is performed and spatial images on which deep learning is performed.

In step S140, the hand gesture recognition unit 540 may perform final classification using the integrated time-series images and the integrated spatial images. At this time, the hand gesture recognizing unit 540 may classify whether the hand gesture of the video is included using pre-stored information using integrated time-series images and integrated spatial images.

As described above, according to the embodiments, hand gestures can be recognized in real time by recognizing motions in which several 2D actions are connected in a 3D video and performing deep learning after temporal or spatial zoning.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or the components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (5)

In the deep learning method based on TSN (Temporal Segment Networks) for real-time hand motion recognition in a video performed by a computer device,
Segmenting the entire video according to a time-series image or a spatial image in TSN for hand gesture recognition;
performing deep learning on the time series image and deep learning on the spatial image, respectively;
integrating time-series images on which deep learning is performed and integrating spatial images on which deep learning is performed; and
Final classification using the integrated spatial images with the integrated time-series images
Including, TSN-based deep learning method for hand gesture recognition. According to claim 1,
The zoning step is
Dividing a video into a plurality of short snippets; and
Zoning each of the short pieces of information according to a time-series image or a spatial image.
Including, TSN-based deep learning method for hand gesture recognition. According to claim 1,
In the final classification step,
Classifying whether or not the hand gesture of the video is included using pre-stored information using the integrated time-series images and the integrated spatial images
Characterized by, TSN-based deep learning method for hand gesture recognition. In a deep learning device based on TSN (Temporal Segment Networks) for real-time hand motion recognition in a video,
In TSN for hand gesture recognition, an image segmentation unit that zones the entire video according to time-series images or spatial images;
an image deep learning unit that performs deep learning on the time series image and deep learning on the spatial image, respectively;
an image integration unit that integrates time-series images on which deep learning has been performed and spatial images on which deep learning has been performed; and
Hand gesture recognition unit for final classification using the integrated time-series images and the integrated spatial images
Including, TSN-based deep learning device for hand gesture recognition. According to claim 4,
The video zoning unit,
a first zoning unit dividing a video into a plurality of short snippets; and
A second zoning unit for zoning each of the short pieces of information according to a time-series image or a spatial image.
Including, TSN-based deep learning device for hand gesture recognition.

Images