KR102587234B1 - Deep learning method and apparatus based on tsn for real-time hand gesture recognition in video - Google Patents

Abstract

A TSN-based deep learning method and device for real-time hand gesture recognition in videos is presented. A deep learning method based on TSN (Temporal Segment Networks) for real-time hand gesture recognition among videos performed by a computer device according to an embodiment includes the step of zoning the entire video according to a time series image or spatial image in the TSN for hand gesture recognition. ; performing deep learning on the time series image and deep learning on the spatial image, respectively; Integrating time series images on which deep learning was performed and integrating spatial images on which deep learning was performed; and a final classification step using the integrated time series images and the integrated spatial images.

Description

TSN-based deep learning method and device for real-time hand gesture recognition in videos {DEEP LEARNING METHOD AND APPARATUS BASED ON TSN FOR REAL-TIME HAND GESTURE RECOGNITION IN VIDEO}

The following embodiments relate to a hand gesture recognition method and device, and more specifically, to a TSN-based deep learning method and device for real-time hand gesture recognition in videos.

Mobile phones are a huge part of our lives, and the percentage of Internet traffic generated from mobile phones exceeds that of desktops. Due to this transition, the way humans interact with these devices has evolved from keyboard/mouse to gesture, voice, and brain-computer interfaces. For example, in the case of voice interfaces, there are problems with voice interfaces being inaccurate or difficult to use in noisy outdoor environments or environments that require silence, so a combination of a hand gestural interface and voice is currently used.

In addition, there is an increasing demand to replace keyboard or panel touch input by using recognition of various hand gestures as input to programs. For example, if hand gesture recognition can be replaced with autonomous driving commands during the operation of a self-driving car in a complex surrounding environment, actions such as pressing buttons can be avoided, thereby reducing the risk of accidents.

The existing TSN (Temporal Segment Networks) technique uses the entire video as input and outputs the subject of the video, but this technique is not suitable for applications that recognize hand gestures in real time and use them as commands. Therefore, a technique that can recognize and output hand gestures for each time interval is needed.

Korean Patent No. 10-2121654 describes technology related to this deep learning-based automatic gesture recognition method and system.

Korean Patent No. 10-2121654

Embodiments describe a TSN-based deep learning method and device for real-time hand gesture recognition in videos, and more specifically, hand gesture recognition of simple two-dimensional images in three-dimensional videos that recognize motions where multiple 2D actions are connected. It provides real-time deep learning technology based on TSN (Temporal Segment Networks), a hand gesture recognition technique.

Embodiments provide a TSN-based deep learning method and device for real-time hand gesture recognition in a video, which can recognize motion linked to multiple 2D actions in a 3D video and recognize hand gestures in real time based on deep learning.

A deep learning method based on TSN (Temporal Segment Networks) for real-time hand gesture recognition among videos performed by a computer device according to an embodiment includes the step of zoning the entire video according to a time series image or spatial image in the TSN for hand gesture recognition. ; performing deep learning on the time series image and deep learning on the spatial image, respectively; Integrating time series images on which deep learning was performed and integrating spatial images on which deep learning was performed; and a final classification step using the integrated time series images and the integrated spatial images.

The zoning step includes dividing the video into a plurality of short snippets; and zoning each of the short pieces of information according to a time series image or a spatial image.

In the final classification step, pre-stored information using the integrated time-series images and the integrated spatial images can be used to classify whether the video includes hand gestures.

According to another embodiment, a deep learning device based on TSN (Temporal Segment Networks) for real-time hand gesture recognition among videos includes an image zoning unit for zoning the entire video according to a time series image or spatial image in the TSN for hand gesture recognition; An image deep learning unit that performs deep learning on the time series image and deep learning on the spatial image, respectively; An image integration unit that integrates time series images on which deep learning has been performed and spatial images on which deep learning has been performed; and a hand gesture recognition unit that performs final classification using the integrated time series images and the integrated spatial images.

The video zoning unit includes a first zoning unit that divides the video into a plurality of short snippets; and a second zoning unit for zoning each of the short pieces of information according to a time series image or a spatial image.

According to embodiments, a TSN-based deep learning method and device for real-time hand gesture recognition in videos is provided, which can recognize motion connected to multiple 2D actions in a 3D video and recognize hand gestures in real time based on deep learning. You can.

FIG. 1A is a diagram schematically showing a TSN structure for real-time hand gesture recognition according to an embodiment.
FIG. 1B is a diagram illustrating an example of a TSN structure for real-time hand gesture recognition of two streams according to an embodiment.
Figure 2 is a module for explaining a two-dimensional module according to an embodiment.
Figure 3 is a diagram for explaining a 3D module according to an embodiment.
Figure 4 is a flowchart showing a TSN-based deep learning method for hand gesture recognition according to an embodiment.
Figure 5 is a block diagram showing a TSN-based deep learning device for hand gesture recognition according to an embodiment.

Hereinafter, embodiments will be described with reference to the attached drawings. However, the described embodiments may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. In addition, various embodiments are provided to more completely explain the present invention to those with average knowledge in the art. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

The following examples relate to a TSN-based deep learning method and device for real-time hand gesture recognition in videos. Hand gesture recognition in a simple 2D image can be used to recognize hand gestures in a 3D video that recognizes the motion of multiple 2D actions connected. We propose a real-time deep learning technology based on Temporal Segment Networks (TSN).

FIG. 1A is a diagram schematically showing a TSN structure for real-time hand gesture recognition according to an embodiment.

Referring to FIG. 1A, in the TSN 130 for hand motion recognition, the entire video 110 is segmented, the CNN for the time series image and the CNN for the same space image are separately processed and integrated (140). It goes through a final classification process (150). Here, the entire video 110 can be divided into a plurality of short snippets 120 and then divided into time series images and co-space images.

For example, if image V is divided into m sections, it can be divided into {S ₁ , S ₂ , ···, S _m }. At this time, TSN can be expressed as the following equation.

[Equation 1]

Here, T _m is a randomly sampled area among m areas, and F( T _m ; W) is a function of CNN deep learning with T _m as input and W as a parameter, and the classification result as output. G is the zoning integration function, which integrates the results of all zones. H is a function that classifies which hand gestures the video contains.

FIG. 1B is a diagram illustrating an example of a TSN structure for real-time hand gesture recognition of two streams according to an embodiment.

Referring to Figure 1b, it shows the framework of a two-stream hand gesture recognition network (top) and the structure of an expanded 3D ResNet50 (bottom). As an example of an extended 3D module, a two-stream hand gesture recognition network based on 3D ResNets can be proposed. In the network framework, the input hand gesture video is divided into segments and short snippets can be selected from each segment every frame. Here you can set = 2, = 24. Class scores from different short snippets can be fused to generate the final prediction. The network has two dilated 3D ResNets and can share parameters. During the training process, for each inflated 3D ResNet, we can run a soft max operation on the output of the fully connected layer and then use the cross-entropy loss to calculate the error between the model output and the labels.

Figure 2 is a module for explaining a two-dimensional module according to an embodiment. And FIG. 3 is a diagram for explaining a 3D module according to an embodiment.

Referring to FIG. 2, an example of a two-dimensional module (residual module) is shown, and FIG. 3 shows an example of an expanded three-dimensional module. Internally, each frame is a 2D image, and converting a 2D image to 3D during the deep learning process can be done through the following process. In a 2D process, all filters and pooling kernels can be extended to give a time dimension. The filter is usually N*N and can be extended to N*N*N. Therefore, we can add a time dimension to the 2-dimensional module and use it as input to the learning module.

Figure 4 is a block diagram for explaining an example of the internal configuration of a computer system according to an embodiment.

For example, a TSN-based deep learning device for hand gesture recognition according to embodiments of the present invention may be implemented through the computer system (device) 400 of FIG. 4. As shown in FIG. 4, the computer system 400 is a component for executing a TSN-based deep learning method for hand gesture recognition and includes a processor 410, a memory 420, a persistent storage device 430, and a bus ( 440), an input/output interface 450, and a network interface 460.

Processor 410 may include or be part of any device capable of processing any sequence of instructions. Processor 410 may include, for example, a computer processor, a processor in a mobile device or other electronic device, and/or a digital processor. Processor 410 may be included in, for example, a server computing device, server computer, series of server computers, server farm, cloud computer, content platform, mobile computing device, smartphone, tablet, set-top box, media player, etc. Processor 410 may be connected to memory 420 through bus 440.

Memory 420 may include volatile memory, persistent, virtual, or other memory for storing information used by or output by computer system 400. Memory 420 may include, for example, random access memory (RAM) and/or dynamic RAM (DRAM). Memory 420 may be used to store any information, such as state information of computer system 400. The memory 420 may also be used to store instructions of the computer system 400, including instructions for TSN-based deep learning for hand gesture recognition, for example. Computer system 400 may include one or more processors 410 as needed or appropriate.

Bus 440 may include a communications infrastructure that enables interaction between various components of computer system 400. Bus 440 may carry data between components of computer system 400, for example, between processor 410 and memory 420. Bus 440 may include wireless and/or wired communication media between components of computer system 400 and may include parallel, series, or other topological arrangements.

Persistent storage device 430 is a component, such as a memory or other persistent storage device, as used by computer system 400 to store data for some extended period of time (e.g., compared to memory 420). may include. Persistent storage device 430 may include non-volatile main memory, such as used by processor 410 within computer system 400. Persistent storage device 430 may include, for example, flash memory, hard disk, optical disk, or other computer-readable medium.

The input/output interface 450 may include interfaces for a keyboard, mouse, voice command input, display, or other input or output device. Configuration commands and/or information for TSN-based deep learning for hand gesture recognition may be received through the input/output interface 450.

Network interface 460 may include one or more interfaces to networks, such as a local area network or the Internet. Network interface 460 may include interfaces for wired or wireless connections. Configuration commands and/or information for TSN-based deep learning for hand gesture recognition may be received through the network interface 460.

Additionally, in other embodiments, computer system 400 may include more components than those of FIG. 4 . However, there is no need to clearly show most prior art components. For example, the computer system 400 is implemented to include at least some of the input/output devices connected to the input/output interface 450 described above, or a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, It may further include other components such as a database, etc.

Figure 5 is a block diagram showing a TSN-based deep learning device for hand gesture recognition according to an embodiment.

FIG. 5 is a diagram illustrating examples of components that the processor 410 of the computer system 400 according to the embodiment of FIG. 4 may include. Here, the processor 410 of the computer system 400 may include a TSN-based deep learning device 500 for hand gesture recognition according to an embodiment. The TSN-based deep learning device 500 for hand gesture recognition according to an embodiment includes an image zoning unit 510, an image deep learning unit 520, an image integration unit 530, and a hand gesture recognition unit 540. It can be done. Here, the image zoning unit 510 may include a first zoning unit 511 and a second zoning unit 512.

The processor 410 and its components may perform steps (S110 to S140) included in the TSN-based deep learning method for hand gesture recognition in FIG. 6. For example, the processor 410 and its components may be implemented to execute instructions according to the code of the operating system included in the memory 420 and at least one program code described above. Here, at least one program code may correspond to the code of a program implemented to process a water pollution source monitoring method.

The TSN-based deep learning method for hand gesture recognition may not occur in the order shown, and some of the steps may be omitted or additional processes may be included.

Figure 6 is a flowchart showing a TSN-based deep learning method for hand gesture recognition according to an embodiment.

Referring to FIG. 6, a deep learning method based on TSN (Temporal Segment Networks) for real-time hand gesture recognition among videos performed by a computer device according to an embodiment converts the entire video into a time series image or spatial image in TSN for hand gesture recognition. A step of zoning according to the image (S110), a step of performing deep learning on the time series image and deep learning on the spatial image respectively (S120), integrating the time series images on which deep learning was performed, and spatial images on which deep learning was performed It may include a step of integrating them (S130), and a step of final classification using the integrated time series images and the integrated spatial images (S140).

Here, the zoning step (S110) may include dividing the video into a plurality of short snippets and zoning each piece of short information according to a time series image or a spatial image.

Below, each step of the TSN-based deep learning method for hand gesture recognition according to an embodiment is described.

The TSN-based deep learning method for hand gesture recognition according to an embodiment can be explained in more detail by taking a TSN-based deep learning device for hand gesture recognition according to an embodiment as an example.

In step S110, the image segmentation unit 510 may segment the entire video in TSN for hand gesture recognition according to time series images or spatial images.

Here, the image zoning unit 510 may include a first zoning unit 511 and a second zoning unit 512. More specifically, the first zoning unit 511 may divide the video into a plurality of short snippets. Thereafter, the second zoning unit 512 may zone each piece of short information according to a time series image or a spatial image.

In step S120, the image deep learning unit 520 may perform deep learning on time series images and deep learning on spatial images, respectively.

In step S130, the image integration unit 530 may integrate time series images on which deep learning has been performed and spatial images on which deep learning has been performed.

In step S140, the hand gesture recognition unit 540 may perform final classification using the integrated time series images and integrated spatial images. At this time, the hand gesture recognition unit 540 can classify whether the video contains a hand gesture using pre-stored information using integrated time series images and integrated spatial images.

As described above, according to the embodiments, hand movements can be recognized in real time by recognizing the motion of multiple 2D actions connected in a 3D video and performing deep learning after temporal or spatial zoning.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (5)

In a deep learning method based on TSN (Temporal Segment Networks) for real-time hand gesture recognition among videos performed by a computer device,
In TSN for hand gesture recognition, zoning the entire video according to time series image or spatial image;
performing deep learning on the time series image and deep learning on the spatial image, respectively;
Integrating time series images on which deep learning was performed and integrating spatial images on which deep learning was performed; and
Final classification step using the integrated time series images and the integrated spatial images
Including,
The zoning step is,
Dividing the video into a plurality of short snippets; and
It includes zoning each of the short pieces of information according to a time series image or a spatial image,
The steps of performing deep learning on the time series image and deep learning on the spatial image, respectively, are:
In order to convert each frame, which is a two-dimensional image, into a three-dimensional image in the deep learning process, all filters and pooling kernels are expanded to give a time dimension, thereby converting each frame into a three-dimensional image.
Hand gestures are recognized through a two-stream hand gesture recognition network based on two expanded 3D ResNets, and the short information is selected from each segment in each frame and fuses the class scores of different short information to generate time series images and spatial images. to integrate them
TSN-based deep learning method for hand gesture recognition. delete According to paragraph 1,
The final classification step is,
Classifying whether the video contains hand gestures using pre-stored information using the integrated time series images and the integrated spatial images.
A TSN-based deep learning method for hand gesture recognition, characterized by . In a deep learning device based on TSN (Temporal Segment Networks) for real-time hand gesture recognition in videos,
An image zoning unit that segments the entire video according to time series or spatial images in TSN for hand gesture recognition;
An image deep learning unit that performs deep learning on the time series image and deep learning on the spatial image, respectively;
An image integration unit that integrates time series images on which deep learning has been performed and spatial images on which deep learning has been performed; and
A hand gesture recognition unit that performs final classification using the integrated time series images and the integrated spatial images.
Including,
The video zoning unit,
A first zoning unit that divides the video into a plurality of short snippets; and
It includes a second zoning unit for zoning each of the short pieces of information according to a time series image or a spatial image,
The video deep learning unit,
In order to convert each frame, which is a two-dimensional image, into a three-dimensional image in the deep learning process, all filters and pooling kernels are expanded to give a time dimension, thereby converting each frame into a three-dimensional image.
Hand gestures are recognized through a two-stream hand gesture recognition network based on two expanded 3D ResNets, and the short information is selected from each segment in each frame and fuses the class scores of different short information to generate time series images and spatial images. to integrate them
TSN-based deep learning device for hand gesture recognition. delete

Images