KR102819296B1 - Behavior Inference Method Based on Transposition Transformation and Computing Device for Performing the Same - Google Patents

Abstract

The present invention relates to an action inference method considering spatiotemporal characteristics of an object. According to one embodiment of the present invention, a method for action inference performed in a computer device extracts spatial information from input data as a feature map, connects the extracted feature maps, transposes the connected feature maps, and outputs an inference result of a neural network model learned using the transposed feature map as an input. In the present invention, when CNN is used in a network for inference, feature points that a space has in each time period can be learned, and when a transformer is used, relationships between spaces in time can be learned. Through this transformation process, action recognition can be performed with high accuracy while maintaining spatial information by focusing on temporal information.

Description

{Behavior Inference Method Based on Transposition Transformation and Computing Device for Performing the Same}

The present invention relates to a method for action inference that takes into account spatiotemporal characteristics of an object.

Action recognition is one of the important research topics in the field of computer vision and can be utilized in various application fields. For action recognition, it is essential to include both temporal and spatial information. Temporal information is important for understanding the flow of continuous actions, and spatial information contributes to identifying the posture or location at a specific moment.

Existing action recognition methods have mainly taken two approaches. The first approach is to learn spatiotemporal information simultaneously by utilizing clips or videos. In this method, the action recognition system is configured by extracting spatiotemporal information through 3D convolution or processing images into patches and putting them into a transformer. This method can achieve a high recognition rate by simultaneously processing spatiotemporal information, but the amount of computation increases rapidly, making it difficult to recognize actions in real time in embedded systems with limited resources.

In cases where spatial information is prioritized, the feature points of the object whose action is to be recognized in the image are extracted, and then these feature maps are stacked to include temporal information, and then temporal information is added through additional processing. For example, there are methods of passing it through a GNN (Graph Neural Network) or reapplying a transformer or CNN.

However, since these methods are designed to inherently prioritize spatial information, the importance of temporal information is often overlooked. For example, in situations where temporal information is more important for the same action, such as sitting down and standing up, a design that focuses on spatial information can cause problems in accurately recognizing actions.

The purpose of the present invention is to minimize computational burden by efficiently providing input data that meets the purpose of a network even with limited resources.

In addition, the present invention seeks to minimize errors that may occur in action recognition by achieving harmony with spatial information through a design that gives priority to temporal information.

Through this, the present invention aims to simultaneously improve accuracy and efficiency in a real-time action recognition system. Furthermore, this approach can enable rapid and accurate action recognition even in embedded systems or resource-constrained environments.

In order to achieve the above-described purpose, a behavioral inference method performed in a computer device according to one embodiment of the present invention extracts spatial information from input data as a feature map, connects the extracted feature maps, transposes the connected feature maps, and outputs an inference result of a neural network model learned using the transposed feature map as an input.

Connected feature maps can be transposed along the height direction.

Connected feature maps can be transposed along the width direction.

The output of the first transformed feature map transposed in the height direction and the output of the second transformed feature map transposed in the width direction can be integrated to output the inference result.

The criteria for prepositional transformation can be determined based on the behavior of the object to be inferred.

The input data can be reduced to the first model to extract feature maps.

A computer device including a memory communicating with a processor, which can extract spatial information from input data as a feature map, connect the extracted feature maps, transpose the connected feature maps, and output the inference result of a neural network model trained using the transposed feature map as input.

The present invention extracts spatial information to generate a feature map of a frame input in a real-time system, and based on this, reflects temporal information to provide a higher-dimensional action recognition result.

In addition, the present invention enables recognition of actions that take into account the flow of time corresponding to each space by transforming a feature map through a design focused on temporal information.

The present invention can learn the features that space has in terms of time by using CNN in a network for inference, and the relationships between spaces in terms of time by using a transformer. Through this transformation process, it is possible to perform action recognition with high accuracy while maintaining spatial information by focusing on temporal information.

The present invention can increase the accuracy of an action recognition system and reduce action recognition errors in a real-time system through a structure that balances time and space information.

FIG. 1 is a block diagram showing the configuration of a system according to one embodiment of the present invention.
Figure 2 is a flowchart showing an inference process according to one embodiment of the present invention.
FIG. 3 is a diagram illustrating a connection process according to one embodiment of the present invention.
FIG. 4 is a diagram illustrating a pre-transformation process according to one embodiment of the present invention.
FIG. 5 is a diagram showing integration of feature maps according to one embodiment of the present invention.
FIG. 6 is a diagram showing integration of output according to one embodiment of the present invention.
FIG. 7 is an exemplary diagram showing the implementation of a service server as a computing device according to one embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art can invent various devices that implement the principles of the invention and are included in the concept and scope of the invention, even though they are not explicitly described or illustrated in this specification. In addition, it should be understood that all conditional terms and embodiments listed in this specification are, in principle, expressly intended only for the purpose of making the concept of the invention understood, and are not limited to the specifically listed embodiments and conditions.

The above-described objects, features and advantages will become more apparent through the following detailed description with reference to the attached drawings, whereby a person having ordinary skill in the art to which the invention pertains will be able to easily practice the technical idea of the invention.

In addition, when describing the invention, if it is judged that a detailed description of a known technology related to the invention may unnecessarily obscure the gist of the invention, the detailed description will be omitted. Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is an exemplary diagram showing a service system according to one embodiment of the present invention.

Referring to FIG. 1, the system of the present invention is a system composed of a user terminal (1000) and a server (300), and can enable behavior-based event detection in various application fields through behavior recognition.

The user terminal (1000) is a device that collects input data (100) in real time through a device such as a camera or sensor.

Input data (100) is mainly provided in the form of a video clip or image sequence, and the user terminal (1000) can perform preprocessing work necessary to transmit the data to the server (300).

The user terminal (1000) can capture input data in frames and transmit it to the server (300). In addition, it is designed to operate in a limited resource environment such as an embedded system, so that data can be collected and transmitted in real time. In this process, the resolution of the frame can be adjusted to save resources, or only important information can be extracted and transmitted.

The main purpose of the user terminal (1000) is to efficiently collect data required for real-time action recognition and transmit this data to the server (300). Through this, the server (300) can support accurate action recognition.

The server (300) is a central processing unit that executes a neural network model and can perform action recognition inference based on input data transmitted from a user terminal (1000). The server (300) can include various neural network models (CNN, Transformer, etc.) for transpose patch action recognition according to the present invention.

The server (300) can receive input data transmitted from the user terminal (1000) and perform action recognition based on the input data through a neural network model. In this process, the server (300) can simultaneously process spatial information and temporal information of the input data to ultimately generate an action recognition result. The server (300) can return this result to the user terminal (1000) to provide real-time feedback to the user.

The main purpose of the server (300) is to generate inference results in real time for input data provided from the user terminal (1000) through accurate and rapid action recognition. The server (300) executes a complex neural network model based on high processing capability and can derive optimal recognition results by considering both temporal and spatial information.

Furthermore, in this embodiment, the server (300) may be replaced with a device such as a user terminal (1000) or an edge device capable of operating a neural network model, in addition to a computer system that provides a service in a relationship with another computer or device (client) through a network, and thus, it is also possible to perform the behavioral inference method according to this embodiment through communication between devices.

That is, the device performing the role of a server (300) does not necessarily have to be a central server (300), and a neural network model can be run on a user terminal (1000) or an edge device, and behavioral patterns can be analyzed and recognized in real time through cooperation between devices.

The system according to the present invention can recognize various actions in real time and detect events or provide warnings according to specific behavior patterns. For example, in a public safety monitoring system, actions such as a person falling can be detected in real time and immediate measures can be taken. By monitoring the patient's movements, abnormal behaviors that may occur during fall prevention or rehabilitation treatment can be detected in real time and reported to medical staff.

Alternatively, a smart home system can recognize the behavior of residents and implement automated environmental controls such as lighting or temperature control. Additionally, if abnormal behavior is detected, security alerts can be activated.

Below, the specific service process is described with reference to Fig. 2.

Figure 2 is a flowchart showing an inference process according to one embodiment of the present invention.

Referring to FIG. 2, the server (300) first extracts spatial information as a feature map.

The server (300) extracts spatial information necessary for action recognition from input data and creates a feature map (S100).

When a frame comes in as input in a real-time system, the first network can analyze the spatial features of the frame and extract important information in the form of a feature map.

These feature maps are used as the basis data for action recognition and can play a key role in efficient processing, especially in resource-constrained environments such as embedded systems. This process can help the model identify key objects or actions within the frame.

Next, the server (300) connects the extracted feature maps (S200).

The server (300) can combine feature maps extracted from multiple frames to create a new feature map reflecting temporal information.

FIG. 3 is a diagram illustrating a connection process according to one embodiment of the present invention.

Referring to FIG. 3, the server (300) can stack feature maps of each frame reflecting spatial information and provide characteristics that include changes over time.

This goes beyond simply listing spatial information to including temporal information, that is, continuous changes between each frame. This process is essential to reflect temporal correlation, which is an important element in action recognition.

The server (300) can condense images input in sequence into feature maps (320) through a convolution layer (310). The condensed feature maps (320) can be generated as a connected feature map (320') by stacking them in time or frame order.

The condensed feature maps (320) are information extracted from each frame, and feature maps generated from multiple frames are arranged in chronological order.

The condensed feature maps (320) reflect the temporal flow of data by stacking the features of each frame along the time axis, thereby enabling the capture of temporal changes or motion patterns.

The stacked feature maps can be temporally or frame-wise concatenated to form a single integrated concatenated feature map (320').

The server (300) transposes the connected feature map (S300).

The server (300) can change the form of input data by transposing the connected feature map to more clearly reflect temporal changes at each spatial location.

Transpose can rearrange the dimensions of feature maps to enable neural network models to learn temporal information more effectively. For example, CNN-based networks can capture temporal features at each spatial location, while transformer-based networks can learn temporal relationships between each spatial location.

The transformation can be performed in the height or width direction to simultaneously preserve spatial information and emphasize temporal information.

FIG. 4 is a diagram illustrating a pre-transformation process according to one embodiment of the present invention.

Referring to FIG. 4, the server (300) transposes the connected feature map (320') according to the direction of the determined transposition transformation (height direction or width direction) to generate a transposed feature map (320 T).

The server (300) can dynamically determine the direction of the pretransformation depending on the type of behavior of a specific object for which it is desired to infer.

The server (300) can dynamically determine the direction of the preprocessing transformation according to the behavior type of the object, rather than performing it in a fixed manner.

For example, if an object's behavior primarily involves vertical movement (e.g., jumping or sitting/standing up), it may be appropriate to perform the transposition along the height direction. Conversely, if an object's behavior involves horizontal movement (e.g., running or walking side to side), it may be more effective to perform the transposition along the width direction.

That is, for an example of a transposition transformation based on an object's behavior, for an action that emphasizes vertical movement, it may be appropriate to perform a transposition transformation based on the height direction. This can better capture the feature that the body position changes vertically over time. The neural network model can use this transposition feature map to more accurately recognize actions based on vertical changes.

In the case of human actions such as walking or running left and right, which are actions with prominent horizontal changes, it may be appropriate to perform transposition transformation based on the width direction. This allows the neural network to effectively learn actions that move left and right over time. For example, it can well reflect changes in body position and posture while moving left and right.

Alternatively, complex movements may occur, resulting in complex behaviors that include both vertical and horizontal changes.

In this case, a transposition transformation that considers both the height direction and the width direction may be required. By performing a transposition transformation based on the height direction first and then performing a transposition transformation based on the width direction, complex behaviors can be analyzed more precisely.

In this embodiment, the server (300) can provide flexibility to dynamically determine the criteria for preprocessing according to a specific behavior type of an object. This enables customized data transformation according to the behavioral characteristics of an object, which can greatly improve the accuracy of behavior recognition.

The next server (300) outputs the inference result of the neural network model learned using the transposed feature map as input.

The server (300) can input the pre-transformed feature map into a learned neural network model to ultimately derive an action recognition result.

The transposed feature map is used as input to the trained neural network model (CNN, transformer, etc.). The neural network model can process this input and produce the final result of action recognition. At this stage, the model can accurately recognize actions while considering both temporal and spatial information, and the output result is used as the final judgment of the action recognition system.

Specifically, the server (300) according to the present embodiment integrates the pre-transformed feature map into an integrated feature map (S400).

Referring to FIG. 5, the server (300) can integrate two feature maps.

The first transformed feature map (320T') may be, for example, a feature map that has been transposed based on the height direction. It may be a map generated by extracting important patterns or features in the height direction of input data and transforming them.

In addition, the second transformed feature map (320T) is a feature map that is transposed based on the width direction. It is a map created by extracting important patterns or features in the width direction of input data and transforming them.

These two transformations interpret the spatial structure of the input data differently, which allows them to learn features in different directions. Therefore, by integrating each feature map, the neural network can infer important information in both the height and width directions of the data without missing it.

The server (300) can determine a method of combining the first transformation feature map (height direction) and the second transformation feature map (width direction).

Concatenate, as the first integration method, can simply be a way to concatenate two feature maps. For example, the features in the height direction and the features in the width direction can be concatenated into a new dimension. This allows the neural network to learn patterns obtained from both directions while maintaining all the information in the two feature maps. This method preserves all the information in the two transformed feature maps without losing any information.

Alternatively, the second integration method, Add, can be performed by adding two feature maps. In this case, two feature maps of the same dimension can be added element by element to create a single integrated map.

Using the sum operation, the two feature maps are combined to more intuitively reflect the importance of each feature. This method can provide the advantage of considering patterns in both directions while compressing information.

It is also possible to apply a hybrid method that connects the integrated feature map for the transpose map with the first transformed feature map (320T') and the second transformed feature map (320T) as a third integrated method.

For example, an integrated map can be created by performing a sum operation on the first transformation feature map (height direction) and the second transformation feature map (width direction).

That is, by adding the element values of the two maps, a single integrated intermediate map can be generated, thereby integrating pattern information in both directions and combining information by considering the importance of each feature map.

Afterwards, the server (300) performs final integration through a connection operation between the intermediate map and each pre-transformed feature map.

The server (300) performs the final integration by connecting the intermediate map combined by the sum operation to the first transformation feature map (320T') and the second transformation feature map (320T) in a new dimension. In this process, information from the two maps is combined to create a single integrated map, and feature information obtained from both directions can be preserved.

If only a simple sum operation is performed, some information may be lost as the feature information in each direction is compressed. Therefore, by performing the sum operation and then connecting the two direction maps in a new dimension, information loss can be prevented and the original information of each feature map can be preserved.

Additionally, in an additional embodiment, the server (300) can integrate each output generated from the feature map to provide a final output (S500).

Referring to FIG. 6, in this embodiment, the server (300) inputs the first transformation feature map (320T') and the second transformation feature map (320T) into the neural network (330), respectively, and can ultimately generate a single integrated output using the inference result of the neural network.

Furthermore, according to the present embodiment, the neural network model for inference may be a transformer-based model that extracts temporal relationships of feature values within an integrated feature map.

That is, it is also possible for the transformer model to effectively learn temporal relationships from transposed feature maps and infer actions.

Transformers can extract temporal relationships from input data (here, transposed feature maps).

The transposed feature map is the original data that has been transposed in a specific direction (height or width direction) to clearly reveal spatial changes according to temporal order. The transformer model can receive this transposed input and learn the correlation between each feature value according to temporal order.

The attention mechanism can dynamically calculate how important each element of the input data is to other elements. By evaluating the interaction between each input feature, it can assign weights that reflect the relationship.

Attention can compute how each temporal feature in the transposed feature map relates to other temporal features. For example, it learns how past actions affect current actions, or how current actions can predict future actions.

The attention mechanism of the transformer model can simultaneously consider the relationships between all temporal features, effectively recognizing complex temporal patterns and improving the accuracy of action recognition.

Transformer models can learn both rapid movements that occur within short time intervals, as well as changes in behavior that occur over longer time intervals.

Attention mechanisms can recognize the temporal context in which a specific action occurs. For example, by simultaneously learning the timing of arm swings and changes in leg movement according to speed during running, more sophisticated action recognition can be achieved.

The present invention can increase the accuracy of an action recognition system and reduce action recognition errors in a real-time system through a structure that gives priority to temporal information.

Referring to FIG. 7 below, the hardware configuration of a service server (300) according to one embodiment of the present invention will be described.

Referring to FIG. 7, in some embodiments of the present invention, the service server (300) may be implemented in the form of a computing device. At least one of each module constituting the service server (300) is implemented on a general-purpose computing processor and thus may include a processor (308), an input/output I/O (302), a memory (340), an interface (306), and a bus (314). The processor (308), the input/output device (302), the memory (340), and/or the interface (306) may be coupled to each other via the bus (314). The bus (314) corresponds to a path through which data is moved.

Specifically, the processor (308) may include at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a microprocessor, a digital signal processor, a microcontroller, an application processor (AP), and logic devices capable of performing functions similar thereto.

The input/output device (302) may include at least one of a keypad, a keyboard, a touchscreen, and a display device. The memory device (340) may store data and/or programs, etc.

The interface (306) may perform a function of transmitting data to or receiving data from a communication network. The interface (306) may be wired or wireless. For example, the interface (306) may include an antenna or a wired/wireless transceiver, etc. The memory (340) may further include a high-speed DRAM and/or SRAM, etc. as a volatile operating memory for improving the operation of the processor (308) but protecting personal information.

Additionally, the memory (340) stores programming and data configurations that provide the functionality of some or all of the modules described herein. For example, it may include logic to perform selected aspects of the learning method described above.

A program or application is loaded as a set of instructions including each step of performing the above-described method stored in memory (340) and causes the processor to perform each step.

The various embodiments described herein may be implemented in a recording medium readable by a computer or similar device, for example, using software, hardware, or a combination thereof.

In terms of hardware implementation, the embodiments described herein can be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. In some cases, the embodiments described herein can be implemented as a control module itself.

In a software implementation, the embodiments, such as the procedures and functions described in this specification, may be implemented as separate software modules. Each of the software modules may perform one or more of the functions and operations described in this specification. The software code may be implemented as a software application written in a suitable programming language. The software code may be stored in a memory module and executed by a control module.

The above description is merely an example of the technical idea of the present invention, and those skilled in the art will appreciate that various modifications, changes, and substitutions may be made without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention and the attached drawings are not intended to limit the technical idea of the present invention but to explain it, and the scope of the technical idea of the present invention is not limited by these embodiments and the attached drawings. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the rights of the present invention.

Claims (14)

In a method of behavioral inference performed on a computer device,
A step of extracting spatial information from input data entered in sequence as a feature map;
A step of connecting the extracted feature maps by considering the sequence;
A step of transposing the above connected feature map;
A step of integrating the above-mentioned pre-transformed feature map into an integrated feature map; and
A step of outputting the inference result of a neural network model learned using the integrated feature map as input,
The above transposition step transposes the connected feature map along a first direction and a second direction different from the first direction,
The above integrating step integrates the first transformed feature map transformed in the first direction and the second transformed feature map transformed in the second direction.
Behavioral reasoning methods. In the first paragraph,
The above pre-transformation step is,
An action inference method for transposing the above-mentioned connected feature map based on the height direction of the above-mentioned feature map. In the second paragraph,
The above pre-transformation step is,
An action inference method for transposing the above-mentioned connected feature map based on the width direction of the above-mentioned feature map. In the third paragraph,
The above output step is,
An action inference method that outputs an inference result by integrating the output of a first transformed feature map transposed based on the height direction and a second transformed feature map transposed based on the width direction. In the third paragraph,
The above preposition transformation criteria are a behavioral inference method determined based on the behavior of the object to be inferred. In the first paragraph,
The steps for extracting the above feature map are:
An action inference method for extracting a feature map that reduces the above input data into a first model. In the first paragraph,
A step of generating an intermediate map by adding the first transformed feature map and the second transformed feature map,
The above integrating step integrates the first transformed feature map, the second transformed feature map, and the intermediate map into an integrated feature map by performing a concatenation operation.
Behavioral reasoning methods. Processor, and
comprising a memory communicating with said processor,
The above memory stores instructions that cause the processor to perform operations,
The above actions are,
An operation that extracts spatial information from input data entered in sequence as a feature map.
An operation of connecting the extracted feature maps by considering the sequence;
The operation of transposing the above connected feature maps,
An operation of integrating the above-mentioned transposed feature map into an integrated feature map, and
Including an operation of outputting the inference result of a neural network model learned using the integrated integrated feature map as input,
The above transposition operation transposes the connected feature map along a first direction and a second direction different from the first direction,
The above integrating operation integrates the first transformed feature map transformed in the first direction and the second transformed feature map transformed in the second direction.
Computer devices. In Article 8,
The above transposition conversion operation is,
A computer device that transposes the above-mentioned connected feature map based on the height direction of the above-mentioned feature map. In Article 9,
The above transposition conversion operation is,
A computer device that transposes the above-mentioned connected feature map based on the width direction of the above-mentioned feature map. In Article 10,
The above output action is,
A computer device that outputs an inference result by integrating the output of a first transformed feature map transposed based on the height direction and a second transformed feature map transposed based on the width direction. In Article 10,
A computer device in which the criteria for the above prepositional transformation are determined based on the behavior of the object to be inferred. In Article 8,
The operation of extracting the above feature map is,
A computer device for extracting a feature map that reduces the above input data into a first model. In Article 8,
Including an operation of generating an intermediate map by adding the first transformed feature map and the second transformed feature map,
The above integrating operation is a computer device that integrates the first transformed feature map, the second transformed feature map, and the intermediate map into an integrated feature map by performing a connection operation.

Images