KR102558663B1 - Multi-task network model for performing various abnormal behavior detection - Google Patents

Abstract

According to an embodiment of the present disclosure, a method for generating a multi-task network model for performing a plurality of tasks implemented by a computing device including at least one processor is provided. The method further comprises determining a corresponding frame rate for each of the plurality of tasks; setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, each of the plurality of frame rate networks having a corresponding frame rate designating a frame rate for processing image input data; and training the multi-task network; can include

Description

Multi-task network model performing various abnormal behavior detection {MULTI-TASK NETWORK MODEL FOR PERFORMING VARIOUS ABNORMAL BEHAVIOR DETECTION}

The present invention relates to machine learning for image analysis, and more particularly, to a multi-task network model capable of processing images at various frame rates.

The act of detecting abnormal behavior through video becomes a fundamental technology for important applications that can be used in real environments, such as child protection, dangerous behavior alarms, and rescue signal reception. However, direct detection of deviant behavior while watching a video can be costly in order to hire a person, and the performance of deviant behavior detection is greatly influenced by the degree of fatigue of the person. Therefore, due to the excellent performance and fast detection speed of the deep learning algorithm in image analysis, the abnormal behavior detection process is being replaced by the deep learning method. However, existing studies are aimed at detecting a single abnormal behavior. In real applications, it should be possible to detect various abnormal behaviors. In reality, in most cases, a dangerous situation, a rescue signal, and the like are not defined as a single action.

Research on detecting a single deviant behavior is in progress, but research on a single network capable of detecting various deviant behaviors is still lacking. In addition, studies on learning to perform multiple tasks in one network are being conducted, but there has been no study applied to 4-dimensional data such as images.

4D data has frame information added to 3D data. Depending on the abnormal behavior, the appropriate frame rate may be different. If the multi-task learning technique using 3-dimensional data is simply applied to 4-dimensional data, it is difficult to reflect the difference in frame rate according to the task. Therefore, a new multi-task (abnormal behavior detection) learning technique applied to 4-dimensional data is required. To solve this problem, there is a demand for a network capable of detecting various abnormal behaviors in a single network.

An object of the present disclosure is to provide a method for generating a multi-task network model for performing a plurality of tasks and a method for performing a plurality of tasks.

According to an embodiment of the present disclosure for realizing the above object, a method for generating a multi-task network model for performing a plurality of tasks implemented by a computing device including at least one processor is provided. The method further comprises determining a corresponding frame rate for each of the plurality of tasks; setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, each of the plurality of frame rate networks having a corresponding frame rate designating a frame rate for processing image input data; and training the multi-task network; can include

In an alternative embodiment, determining a corresponding frame rate for each of the plurality of tasks includes: training a plurality of single-task networks for a first task included in the plurality of tasks, wherein the plurality of single-task networks respectively process input data according to different corresponding frame rates; performing performance evaluation of each of the plurality of single task networks for the first task; determining a network having the highest performance for the first task among the plurality of single task networks as a corresponding single task network for the first task; and determining the determined corresponding frame rate of the corresponding single task network as the corresponding frame rate of the first task. can include

In an alternative embodiment, setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network includes: determining a frame rate network having a corresponding frame rate equal to the corresponding frame rate of the first task among the plurality of frame rate networks; and adding a first task network for the first task on the determined frame rate network. can include

In an alternative embodiment, each of the plurality of frame rate networks includes a shared network and at least one task network, and a combination of a shared network included on a frame rate network having a corresponding frame rate equal to that of the first task and the first task network may have network characteristics of a corresponding single task network for the first task.

In an alternative embodiment, the network characteristics may include a channel configuration of the convolutional layer.

In an alternative embodiment, the corresponding frame rate includes a first frame rate and a second frame rate slower than the first frame rate, the plurality of frame rate networks includes a first frame rate network having the first frame rate and a second frame rate network having the second frame rate, and the number of channels of the plurality of convolution layers of a combination of a shared network included in the second frame rate network and a task network is equal to that of the shared network included in the first frame rate network. The number of channels of a plurality of convolutional layers of a combination of task networks may be greater than that of the number of channels.

In an alternative embodiment, the second frame rate may be an integer multiple of the first frame rate.

In an alternative embodiment, the number of channels of the plurality of convolution layers of the combination of the shared network and one task network included in the second frame rate network may be an integer multiple of the number of channels of the plurality of convolution layers of the combination of the shared network and one task network included in the first frame rate network.

In an alternative embodiment, the step of training the multi-task network includes: learning a combination of the first task network and a shared network included in a frame rate network having a corresponding frame rate equal to that of the first task using training data for the first task; can include

In an alternative embodiment, the input data is 4-dimensional or 5-dimensional data including frame information about an image, and each of the plurality of frame rate networks may process data having frame information corresponding to a corresponding frame rate among the input data.

In an alternative embodiment, the plurality of tasks may include detecting a plurality of anomalous behavior of an object on an image.

According to another embodiment of the present disclosure for realizing the above object, a method for performing a plurality of tasks implemented by a computing device including at least one processor is provided. The method further comprises processing input data through a multi-task network model according to claim 1 to generate a plurality of output values for the plurality of tasks; or processing input data through a shared network and a task network corresponding to the first task included in the multi-task network model according to claim 1 to generate an output value for the first task included in the plurality of tasks; can include

According to another embodiment of the present disclosure for realizing the above object, a computing device generating a multi-task network model for performing a plurality of tasks is provided. The computing device includes: a processor including at least one core; a memory containing program codes executable by the processor; and a network unit for receiving image input data. including,

The processor may determine a corresponding frame rate for each of the plurality of tasks, set a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, wherein each of the plurality of frame rate networks has a corresponding frame rate designating a frame rate for processing image input data, and train the multi-task network.

According to another embodiment of the present disclosure for realizing the above object, a computer program stored in a computer readable storage medium is provided. When the computer program is executed on one or more processors, the following operations of generating a multi-task network model for performing a plurality of tasks are performed: determining a corresponding frame rate for each of the plurality of tasks; setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, each of the plurality of frame rate networks having a corresponding frame rate designating a frame rate for processing image input data; and learning the multi-task network.

The present disclosure may provide a method for generating a multi-task network model for performing multiple tasks and a method for performing multiple tasks.

1 is a block diagram of a computing device for generating a multi-task network model or performing a plurality of tasks according to an embodiment of the present disclosure.
2 is a flowchart of a method of generating a multi-task network model for performing a plurality of tasks by processing image input data according to some embodiments of the present disclosure.
3 is a diagram for explaining a plurality of single task networks for determining corresponding frame rates for tasks according to some embodiments of the present disclosure.
4 is a diagram for explaining a multi-task network model according to some embodiments of the present disclosure.
5 is a diagram for explaining a frame rate network including a shared network and a task network according to some embodiments of the present disclosure.
6 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific details.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure, processor, object, thread of execution, program, and/or computer running on a processor. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component may be distributed between two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. Components may communicate via local and/or remote processes, e.g., according to a signal with one or more packets of data (e.g., data from one component interacting with another component in a local system, distributed system, and/or data transmitted via a signal to another system and over a network such as the Internet).

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified or clear from the context, “X employs A or B” is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses either A or B" may apply to either of these cases. Also, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the features and/or components are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Also, unless otherwise specified or where the context clearly indicates that a singular form is indicated, the singular in this specification and claims should generally be construed to mean "one or more".

In addition, the term “at least one of A or B” should be interpreted as meaning “when only A is included”, “when only B is included”, and “when A and B are combined”.

Those of skill should further appreciate that the various illustrative logical blocks, components, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

In the present disclosure, network functions, artificial neural networks, and neural networks may be used interchangeably.

On the other hand, the term "image" or "image data" used throughout the description and claims of the present invention refers to multi-dimensional data composed of discrete image elements (e.g., pixels in a two-dimensional image), or, in other words, a visible object (e.g., displayed on a video screen) or a digital representation of that object.

Throughout this specification, deep learning = model, network model, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links. A node may be composed of a combination of channels, filters, and nonlinear functions.

In a neural network, one or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of data of the output node may be determined based on data input to the input node. Here, a link interconnecting an input node and an output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are connected to one output node by respective links, the output node may determine an output node value based on values input to input nodes connected to the output node and a weight set for a link corresponding to each input node.

As described above, in the neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may constitute a layer. Some of the nodes constituting the neural network may form one layer based on distances from the first input node. For example, a set of nodes having a distance of n from the first input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed through to reach the corresponding node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of a layer in a neural network may be defined in a method different from the above. For example, a layer of nodes may be defined by a distance from a final output node.

An initial input node may refer to one or more nodes to which data is directly input without going through a link in relation to other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. Also, the hidden node may refer to nodes constituting the neural network other than the first input node and the last output node.

The neural network according to an embodiment of the present disclosure may have the same number of nodes of the input layer as the number of nodes of the output layer, and may be a neural network in which the number of nodes decreases and then increases again as the number of nodes progresses from the input layer to the hidden layer. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of the input layer may be less than the number of nodes of the output layer and the number of nodes decreases as the number of nodes progresses from the input layer to the hidden layer. In addition, the neural network according to another embodiment of the present disclosure may have more nodes in the input layer than nodes in the output layer, and the number of nodes increases as the number of nodes increases from the input layer to the hidden layer. A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. That is, it is possible to grasp the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). Deep neural networks include Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Auto Encoder WORKS), limited Boltzmann Machine (RBM), in -depth belief network (DBN), Q network, U network, Sham Network, hostile creation network Thork). The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In neural network learning, iteratively inputs training data to the neural network, calculates the output of the neural network for the training data and the error of the target, and backpropagates the error of the neural network from the output layer of the neural network to the input layer in order to reduce the error, thereby updating the weight of each node of the neural network. In the case of teacher learning, the learning data in which the correct answer is labeled is used for each learning data (ie, the labeled learning data), and in the case of comparative teacher learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning regarding data classification may be data in which each learning data is labeled with a category. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network and a label of the training data. As another example, in the case of comparative history learning for data classification, an error may be calculated by comparing input learning data with a neural network output. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's computation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate may be used in the early stage of neural network training to increase efficiency by allowing the neural network to quickly obtain a certain level of performance, and a low learning rate may be used in the late stage to increase accuracy.

In learning of a neural network, in general, training data may be a subset of real data (i.e., data to be processed using the trained neural network), and therefore, a learning cycle may exist in which errors for the training data decrease but errors for the actual data increase. Overfitting is a phenomenon in which errors for actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing the error of machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing training data, regularization, inactivating some nodes of the network during learning, dropout, and using a batch normalization layer may be applied.

1 is a block diagram of a computing device for generating a multi-task network model or performing a plurality of tasks according to an embodiment of the present disclosure.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100.

The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

The processor 110 may include one or more cores, and may include processors for data analysis and deep learning, such as a central processing unit (CPU), general purpose graphics processing unit (GPGPU), and tensor processing unit (TPU) of a computing device. The processor 110 may read a computer program stored in the memory 130 and process data for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning a neural network. The processor 110 may perform calculations for neural network learning, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating neural network weights using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the network function. For example, the CPU and GPGPU can process learning of network functions and specific tasks using network functions. In addition, in one embodiment of the present disclosure, learning of a network function and a specific task using a network function may be processed by using processors of a plurality of computing devices together. In addition, a computer program executed in a computing device according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program.

According to an embodiment of the present disclosure, the processor 110 may perform a plurality of tasks based on an image including at least one object of interest using a pretrained deep learning model. Here, the plurality of tasks may be understood as a task of predicting or determining a class related to a plurality of states, properties, characteristics, etc. of a target of interest. For example, the processor 110 may detect an abnormal behavior of a target of interest included in an image by using a pretrained deep learning model.

The deep learning model used by the processor 110 may include a multi-task network model that extracts a feature vector based on image input data including at least one object of interest and estimates a probability value corresponding to each of a plurality of tasks based on the derived feature vector. In this case, the multi-task network model may include a convolution neural network optimized for extracting a feature vector from an image. Here, the optimized convolutional neural network may be a 3D convolutional neural network for “horizontal values,” “vertical values,” and “RGAB” channel “values” of the video (or “image). In addition, the multi-task network model may include a fully-connected neural network that is connected to the convolutional neural network to generate output values suitable for a plurality of tasks. For example, the deep learning model may include a convolutional neural network that extracts a feature vector based on an image including at least one object of interest and a fully-connected neural network that outputs a probability value for each of a plurality of deviant behaviors based on a feature vector derived from the convolutional neural network. In addition, the “multiple” task “network” model can “process” the values output from each of the “multiple” convolutional “layers” through “non-linear” functions (eg, “RICO” functions). In addition to the description above, various types of neural networks suitable for classification tasks in the computer vision field may be applied to the neural network included in the deep learning model of the present disclosure.

According to an embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

According to an embodiment of the present disclosure, the memory 130 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM ( Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 130 on the Internet. The above description of the memory is only an example, and the present disclosure is not limited thereto.

The network unit 150 according to an embodiment of the present disclosure may use any type of known wired or wireless communication system.

The network unit 150 may receive an image expressing a target of interest from an external system. For example, the network unit 150 may receive an image of a CCTV installed at the entrance of a building from a CCTV image transmission system. The CCTV image may be data for learning or inference of a neural network model to be learned as a 2D feature or a 3D feature. Here, the CCTV image may be an image of several people passing through the building entrance. The image expressing the object of interest is not limited to the above example, and may include all images expressing the object of interest acquired through various photographing devices.

In addition, the network unit 150 may transmit and receive information processed by the processor 110, a user interface, and the like through communication with other terminals. For example, the network unit 150 may provide a user interface generated by the processor 100 to a client (eg, a user terminal). In addition, the network unit 150 may receive an external input of a user authorized as a client and transmit it to the processor 110 . In this case, the processor 100 may process operations such as outputting, correcting, changing, adding, and the like of information provided through the user interface based on the user's external input received from the network unit 150 .

Meanwhile, the computing device 100 according to an embodiment of the present disclosure may include a server as a computing system that transmits and receives information through communication with a client. In this case, the client may be any type of terminal capable of accessing the server. For example, the computing device 100 as a server may receive an image expressing a target of interest from an image capturing terminal, detect a specific action of the target of interest, and provide a user interface including the detected result to the user terminal. At this time, the user terminal may output a user interface received from the computing device 100 as a server, and receive or process information through interaction with the user.

In an additional embodiment, the computing device 100 may include any type of terminal that receives data resources generated by any server and performs additional information processing.

2 is a flowchart of a method of generating a multi-task network model for performing a plurality of tasks by processing image input data according to some embodiments of the present disclosure. 3 is a diagram for explaining a plurality of single task networks for determining corresponding frame rates for tasks according to some embodiments of the present disclosure. 4 is a diagram for explaining a multi-task network model according to some embodiments of the present disclosure. 5 is a diagram for explaining a frame rate network including a shared network and a task network according to some embodiments of the present disclosure.

The multi-task network model 300 according to the present disclosure may perform multiple tasks by processing an image including a target of interest. For example, the multi-task network model 300 can be used in an artificial intelligence platform that detects various abnormal behaviors of a target of interest appearing in an image. 4D data such as images may additionally include frame information in 3D data. The frame information may be information indicating which frame is being processed among a plurality of frames constituting an input image. This frame information may be related to the frame rate of the video. In performing various tasks, a suitable frame rate may vary. For example, a relatively fast frame rate may be suitable for detecting an abnormal behavior with fast motion (eg, a state in which people are fighting in a video). Conversely, a relatively slow frame rate may be suitable for detecting an abnormal behavior with a slow motion (eg, a person falling down). However, since the technology applied to the network model processing 3D data is difficult to reflect the frame rate, it is difficult to apply it directly to the network processing 4D data. The method according to the present disclosure may provide a method of generating a multi-task network model 300 capable of efficiently performing various tasks by processing images at various frame rates. Accordingly, the method according to the present disclosure can provide an effective method compared to conventional methods for processing 4D data. In the following, a method for generating the multi-task network model 300 is described in detail, according to some embodiments of the present disclosure.

According to some embodiments of the present disclosure, the method of generating the multi-task network model 300 may include determining a corresponding frame rate for each of a plurality of tasks (s110).

As described above, the multi-task network model 300 may process image input data, which is 4-dimensional data, in order to perform a plurality of tasks. For example, the image input data may be 4-dimensional data including vertical, horizontal, RGB channel values, and frame number values for single image data. Also, the multi-task network model 300 may process a plurality of images in parallel in order to perform a plurality of tasks. In this case, the image input data may be 5-dimensional data including vertical, horizontal, RGB channel values, frame information values, and image information values for a plurality of image data.

The multi-task network model 300 according to the present disclosure may process images at various frame rates (or frame rates) using frame information included in image input data. Specifically, each of the plurality of frame rate networks 310, 320, and 330 included in the multi-task network model 300 may have a corresponding frame rate designating a frame rate for processing image input data. For example, if an image has 64 frames per second (thus, the frame information can have values from 1 to 64), the first frame rate network 310 can process input data at a frame rate of 64, and the second frame rate network 320 can process input data at a frame rate of 32. In this case, the first frame rate network 310 can process input data having frame information of 1 to 64, while the second frame rate network 320 can process 1, 3, 5... It can process input data having frame information of 63. However, it is not limited thereto, and the frame rate network may process input data according to the frame rate in various ways.

As described above, the frame rate of an appropriate image may vary depending on the type of task to be performed. A corresponding frame rate designating an appropriate frame rate for each task may be determined so that the multi-task network model 300 can process input data at an appropriate frame rate for each of a plurality of tasks. In order to determine such a corresponding frame rate, after training a plurality of single task networks 200 having different corresponding frame rates for any one task, the corresponding frame rate of the single task network showing the best performance can be determined as the corresponding frame rate for the corresponding task.

Specifically, a method of determining an appropriate frame rate for each of a plurality of tasks through performance evaluation of a plurality of single task networks is as follows.

According to some embodiments of the present disclosure, determining the corresponding frame rate for each of the plurality of tasks may include training the plurality of single task networks 200 for a first task included in the plurality of tasks. Here, the plurality of single task networks 200 may process input data according to different corresponding frame rates. Also, determining the corresponding frame rate for each of the plurality of tasks may include performing performance evaluation of each of the plurality of single-task networks with respect to a first task.

The plurality of frame rates may include various numbers of frame rates. For example, as shown in FIG. 3 , the number of multiple frame rates may be three. For example, the input data 11 processed by the first single task network 210 may be all frames of an image. The input data 12 processed by the second single task network 220 may be half of an entire frame of an image. In this case, the corresponding frame rate of the second single task network 220 may be half of that of the first single task network 220 . Also, the input data 13 processed by the third single task network 230 may be 1/4 of an entire frame of an image. In this case, the corresponding frame rate of the third single task network 220 may be 1/4 of that of the first single task network 220 . Also, the corresponding frame rate of the second single task network 220 may be 1/2 of that of the second single task network 220 . In this case, when the input data is 64 frames, the first corresponding frame rate of the first single task network 210 may be 64 frame rate, the second corresponding frame rate of the second single task network 220 may be 32 frame rate, and the third corresponding frame rate of the third single task network 230 may be 16 frame rate. However, it is not limited thereto, and the plurality of frame rates may have various numbers and various rates.

When the number of frame rates is determined, the same number of single task networks as the number of frame rates may be trained for performance evaluation. For example, when the number of multiple frame rates is three, three single task networks can be used. For convenience of explanation, any one of the plurality of tasks may be referred to as a first task. In this case, three single-task networks can be learned using the learning data for the first task.

Here, several frame rates included in the plurality of frame rates may be integer multiples of each other. As in the above example, the plurality of frame rates may include a first frame rate and a second frame rate. In this case, the second frame rate may be an integer multiple (eg, twice) of the first frame rate. When the relationship between at least two frame rates is an integer multiple of each other, a single task network having each corresponding frame rate may have a network characteristic constructed using the integer multiple. This feature simplifies network configuration and has an advantage in extending additional networks.

Multiple single task networks may have different network characteristics according to corresponding frame rates. Here, the network characteristics may include a channel configuration of the convolution layer, a parameter value, a type of neural network, and the like. If the network characteristic is a channel configuration of a convolutional layer, the channel configuration may include the total number of channels and the arrangement of the number of channels in each of a plurality of convolutional layers (eg, increase or decrease of the number of channels according to the layer). As in the above example, the first frame rate is n times the second frame rate, and the plurality of single task networks 200 may include a first single task network 210 having the first frame rate as the corresponding frame rate and a second single task network 220 having the second frame rate as the corresponding frame rate. The first single task network 210 may include a plurality of first convolutional layers having a first number of channels, and the second single task network 220 may include a plurality of second convolutional layers having a second number of channels. In this case, the number of second channels may be n times the number of first channels. For example, referring to FIG. 3 , the number of channels of the plurality of convolution layers included in the first single task network 210 is h, and the number of channels of the plurality of convolution layers included in the second single task network 220 may be 2h. Also, the number of channels of the plurality of convolution layers included in the second single task network 220 may be 3h.

Also, the number of channels of each of the plurality of second convolution layers may be n times (eg, twice) the number of channels of each of the plurality of first convolution layers. For example, when a first layer among the plurality of second convolution layers has 10 channels, a first layer among the plurality of first convolution layers may have 5 channels. This channel configuration is just an example, and there may be various channel configurations.

In the case of using a low frame rate, since the number of frames processed during the same time of the video decreases, the accuracy of the output value may be lowered. To compensate for this, the method of the present disclosure may configure a network to perform inference using a larger number of channels for a slow frame rate. Different network characteristics of the plurality of single frame networks 200 according to the corresponding frame rates may be used for the plurality of frame rate networks 310 , 320 , and 330 included in the multi-task network model 300 . In this case, the multi-frame rate network can process the input data at the optimal frame rate for each task.

In further description of learning, a plurality of single task networks each having different network characteristics are learned for any one task (eg, the first task) included in the plurality of tasks. A dataset is distinguished between a training dataset and a validation dataset, and the training dataset can be used to train a plurality of single task networks. When learning of the plurality of single-task networks is completed, the confirmation dataset may be used to evaluate the performance of the learned plurality of single-task networks.

According to some embodiments of the present disclosure, determining the corresponding frame rate for each of the plurality of tasks may include determining a network having the highest performance for the first task among the plurality of single-task networks as the corresponding single-task network for the first task, and determining the corresponding frame rate of the determined corresponding single-task network as the corresponding frame rate of the first task.

As described above, a plurality of single task networks may have different networks according to corresponding frame rates, and as described below, a plurality of frame rate networks may have this characteristic. Accordingly, by determining corresponding frame rates through a plurality of single task networks, it is possible to configure a plurality of frame rate networks to process input data according to an optimal frame rate for a plurality of tasks. For example, after evaluating the performance of a plurality of learned single-task networks for a certain task, a single-task network having the highest performance may be determined as a corresponding task network for the corresponding task. For example, a network having the highest performance for the first task may be determined as a corresponding single task network for the first task. In other words, the corresponding single-task network of the first task may be a network having the highest performance for the first task among the plurality of single-task networks. In this case, the frame rate corresponding to the corresponding single task network for the first task may be determined as the corresponding frame rate of the first task. In other words, the corresponding frame rate of a specific task may mean a frame rate that can be processed by the multi-task network model 300 with the highest performance among a plurality of predetermined frame rates. This manner may be performed for a plurality of tasks so that a corresponding frame rate may be determined for each of the plurality of tasks. When a corresponding frame rate is determined for each of a plurality of tasks, the multi-task network model 300 may be generated using the corresponding frame rate. A method of generating the multi-frame network model 300 is described below.

According to some embodiments of the present disclosure, the method of generating the multi-task network model 300 may include setting a task network for each of a plurality of tasks on the frame rate networks 210, 220, and 230 having the determined corresponding frame rate among the plurality of frame rate networks 200 included in the multi-task network model 300 (step s120).

When a corresponding frame rate is determined for each of a plurality of tasks, a task network for each of the plurality of tasks can be set on any one of the plurality of frame rate networks so that each of the plurality of tasks can be processed on the optimal frame rate network. As described above, the plurality of frame rate networks may each process input data according to different corresponding frame rates. In this case, by using the corresponding frame rate determined by learning a plurality of single task networks, a multi-task network model 300 may be generated so that each task is processed on a network having the most suitable corresponding frame rate among the plurality of frame rate networks. A specific method of setting a task network on the multi-task network model 300 using the corresponding frame rate is as follows.

According to some embodiments of the present disclosure, setting a task network for each of a plurality of tasks on a frame rate network having a corresponding frame rate determined from among a plurality of frame rate networks 310 , 320 , and 330 included in the multi-task network model 300 includes determining a frame rate network having a corresponding frame rate equal to the corresponding frame rate of the first task among the plurality of frame rate networks 310 , 320 , and 330 , and determining a frame rate network for the first task on the determined frame rate network. It may include adding a first task network.

Similar to the examples described above, exemplary embodiments are described by referring to any one of a plurality of tasks as a first task. When the corresponding frame rate for the first task is determined, the first task network of the first task may be set on any one network among a plurality of frame rate networks included in the multi-task network model 300 . As described above, each of the plurality of frame rate networks may have one of the plurality of frame rates as a corresponding frame rate, as well as the plurality of single task networks. Accordingly, when the corresponding frame rate of the first task is determined, one network among a plurality of frame rate networks may be determined as a network corresponding to the corresponding frame rate of the first task (ie, a network having the most appropriate corresponding frame rate).

Specifically, for example, referring to FIG. 4 , when the number of the plurality of frame rates is three, the multi-task network model 300 may include a first frame rate network 310, a second frame rate network 320, and a third frame rate network 330. Similar to the above example, the three frame rates may include, for example, a first frame rate, a second frame rate, and a third frame rate. In this case, the first frame rate network may have the first frame rate as the corresponding frame rate. A second frame rate network may have a second frame rate at a corresponding frame rate. A third frame rate network may have a third frame rate as a corresponding frame rate. Using this relation, when the corresponding frame rate of the first task is the first frame rate, the corresponding frame rate network can be determined as the first frame rate network having the same corresponding frame rate. Also, when the corresponding frame rate of the first task is the second frame rate, the corresponding frame rate network may be determined as a second frame rate network having the same corresponding frame rate. Also, when the corresponding frame rate of the first task is the third frame rate, the corresponding frame rate network may be determined as a third frame rate network having the same corresponding frame rate.

When the corresponding frame rate network for the first task is determined, a task network for the first task (ie, the first task network) may be set on the determined corresponding frame rate network. Setting the task network here may include adding the task network on the corresponding frame rate network. When the first task network is added to the corresponding frame rate network for the first task, in the process of learning the multi-task network model 300 later, the first task network may be trained to process input data according to the corresponding frame rate of the corresponding frame rate network. Accordingly, when the multi-task network model 300 processes input data, the corresponding frame rate network may generate an output value using a frame rate suitable for the first task.

In some embodiments of the present disclosure, the plurality of frame rate networks 310, 320, 330 may each include a shared network and at least one task network. In addition to using a frame rate suitable for processing input data for the first task, the multi-task network model 300 may be configured to use network characteristics suitable for processing the first task. To this end, the shared network and the task network for any one task may be configured to have network characteristics of a single task network having the same corresponding frame rate.

In some embodiments of the present disclosure, a combination of the first task network and a shared network included in a frame rate network having a corresponding frame rate equal to that of the first task may have network characteristics of a corresponding single task network for the first task. Here, network characteristics may include various characteristics. For example, the network characteristics may include a channel configuration of the convolutional layer.

As in the above example, a plurality of single task networks may have different network characteristics. Since the corresponding frame rate for each of the plurality of tasks is determined using a plurality of single task networks, the multi-frame network model 300 preferably uses the network characteristics of the single task network used to determine the corresponding frame rate. For example, referring to FIG. 5, two frame rate networks 410 and 420 are shown. Each frame rate network may include a shared network (411, 421) and at least one task network (412, 422, 423). Here, when the first frame rate network 410 has the first frame rate and the second frame rate network 420 has the second frame rate, each of the frame rate networks 410 and 420 can implement network characteristics of a single task network having the same corresponding frame rate.

For example, as shown in FIG. 5 , the corresponding frame rate for the first task is the first frame rate, and the first task network 412 for the first task can be set on the first frame rate network 410. In this case, the combination of the shared network 411 and the first task network 412 of the first frame rate network 410 may have network characteristics of the first single task network 210 having the first frame rate. Here, when the network characteristic is a channel configuration of a plurality of convolutional layers, the combination of the shared network 411 and the first task network 410 may include a plurality of convolutional layers having the same number of channels h as that of the first single task network. That is, since the shared network 411 includes a plurality of convolutional layers having the number of channels of 1/2 x h, and the first task network 412 includes a plurality of convolutional layers having the number of channels of 1/2 x h, a combination thereof may include a plurality of convolutional layers having the same number of channels h as the first single task network.

As another example, as shown in FIG. 5 , the corresponding frame rate for the second task is the second frame rate, and the second task network 422 for the second task may be set on the second frame rate network 420. In this case, the combination of the shared network 421 and the second task network 422 of the second frame rate network 420 may have network characteristics of the second single task network 220 having the second frame rate. Here, when the network characteristic is a channel configuration of a plurality of convolutional layers, the combination of the shared network 421 and the second task network 422 may include a plurality of convolutional layers having the same number of channels 2h as the second single task network. That is, since the shared network 421 includes a plurality of convolutional layers having the number of channels h, and the second task network 410 includes a plurality of convolutional layers having the number of channels h, the combination thereof may include a plurality of convolutional layers having the same number of channels 2h as the second single task network 220. When the network characteristics are determined by such integer multiples, network expansion may be facilitated because networks having the same network characteristics may be added to expand the network. For example, when the number of channels in a previous network is m, the network can be expanded by adding the same number of channels as m. However, it is not limited thereto, and a combination of a shared network and a task network may have various network characteristics of a single task network.

According to some embodiments of the present disclosure, a method of generating a multi-task network model for performing a plurality of tasks may include training the multi-task network ( S130 ).

After each corresponding frame rate is determined for a plurality of tasks in the above-described manner, each task network for a plurality of tasks may be set on the multi-task network model 300 . After that, the multi-task network model 300 may be learned for each of the plurality of tasks.

According to some embodiments of the present disclosure, the step of training the multi-task network model 300 (s130) may include learning a combination of a shared network and the first task network included in a frame rate network having a corresponding frame rate equal to that of the first task by using learning data for the first task.

As described above, a plurality of frame rate networks included in the multi-task network model 300 may each have a shared network and at least one task network. Each frame rate network may contain only one shared network. Each frame rate network may have as many task networks as there are tasks with the same corresponding frame rate. For example, referring to FIG. 5 , the first frame rate network 410 may have one task network, and the second frame rate network 420 may have two task networks. In this case, the first frame rate network 410 can be trained on one task and the second frame rate network can be trained on two tasks. As described above, a combination of a shared network and at least one task network is learned to process input data at the same corresponding frame rate for one task, and may have the same network characteristics as a single task with the same corresponding frame rate. The multi-task network model designed in this way can perform back-propagation learning using gradient descent. In this case, for any one task, filters belonging to a convolution included in a combination of a shared network and a task network may be learned through backpropagation. For example, if the corresponding frame rate of the first task is the first frame rate, the first task network 412 of the first task may be configured on the first frame rate network 410 . In this case, learning of the first task may be performed on the multi-task network model 300, and in this case, a convolutional filter included in a combination of the shared network 411 and the first task network 412 may be learned through backpropagation. Through this, the shared network can learn about various tasks, and each task network can learn only about a specific task. This combination can show high efficiency in performing multiple tasks. However, it is not limited thereto, and various learning methods may be performed.

According to some embodiments of the present disclosure, a method for performing a plurality of tasks implemented by a computing device including at least one processor may include processing input data through the multi-task network model according to claim 1 to generate a plurality of output values for the plurality of tasks, or processing input data through a shared network and a task network corresponding to the first task included in the multi-task network model according to claim 1 to generate an output value for the first task included in the plurality of tasks.

The multi-task network model 300 generated by the above method can efficiently process multiple tasks on one network. For example, the multi-task network model 300 may process input data to generate a plurality of output values for a plurality of tasks. The multi-task network model 300 may obtain a plurality of output values, and the plurality of output values may be 2D vectors representing probability values for each of the plurality of tasks. For example, an output value generated by combining the shared network 411 of the first frame rate network and the first task network 412 may represent a probability value for the first task. An output value generated by a combination of the shared network 421 of the second frame rate network and the second task network 422 may represent a probability for the second task. An output value generated by a combination of the shared network 421 of the second frame rate network and the third task network 423 may represent a probability value for the third task. Accordingly, the multi-task network model 300 may perform a plurality of tasks using a plurality of output values.

Also, the multi-task network model 300 may use only some of the networks to perform one task. For example, to perform the first task, the multi-task network model 300 may use only a combination of the shared network 411 and the first task network 412 of the first frame rate network 410 . Also, to perform the second task, the multi-task network model 300 may use only a combination of the shared network 421 and the second task network 422 of the second frame rate network 420 . Also, to perform the third task, the multi-task network model 300 may use only a combination of the shared network 421 of the second frame rate network 420 and the third task network 423 .

In this way, the multi-task network model 300 may perform a single task or simultaneously perform a plurality of tasks. For example, a plurality of tasks may include detecting a plurality of abnormal behaviors of an object on an image. For example, in the case of a fast abnormal behavior (eg, a fight), the accuracy of detecting the abnormal behavior may be increased by processing input data at a high frame rate. In the case of an abnormal behavior with a slow motion (for example, a fainting person), the accuracy of detecting the abnormal behavior may be increased by processing the input data at a slow frame rate. Detecting multiple abnormal behaviors in the multi-task network model has the advantage of reducing the amount of computation because there is no need to separately store networks for detecting each abnormal behavior, and there is no need to operate each network separately during inference. Therefore, since the memory stored in the network and the amount of computation used for actual reasoning are reduced, the multi-task network model 300 can be used for IoT, mobile devices, etc. with a small computational capacity. In addition, the multi-task network model 300 can solve the problem of finding abnormal behavior in an already observed image, which requires a large amount of computation, and the problem of finding abnormal behavior in real-time CCTV surveillance, which requires fast inference. In addition, the multi-task network model 300 according to the present disclosure may provide a multiple deviant behavior detection algorithm that processes 4D data according to various frame rates by using frame information differently for each deviant behavior.

6 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may be implemented in combination with computer executable instructions and/or other program modules that may be executed on one or more computers and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, those skilled in the art will appreciate that the methods of the present disclosure may be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operative in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

A computer typically includes a variety of computer readable media. Computer readable media can be any medium that can be accessed by a computer, and such computer readable media includes volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and computer readable transmission media. Computer readable storage media includes volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other medium that can be accessed by a computer and used to store desired information.

A computer readable transmission medium typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery medium. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information within the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable transmission media.

An exemplary environment 1100 implementing various aspects of the present disclosure is shown comprising a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. System bus 1108 couples system components, including but not limited to system memory 1106 , to processing unit 1104 . Processing unit 1104 may be any of a variety of commercially available processors. Dual processor and other multiprocessor architectures may also be used as the processing unit 1104.

System bus 1108 may be any of several types of bus structures that may additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112 . A basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within computer 1102, such as during startup. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

Computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA), which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to removable diskette 1118), and an optical disk drive 1120 (e.g., for reading from or writing to removable diskette 1118). For example, for reading the CD-ROM disk 1122 or for reading from or writing to other high-capacity optical media such as DVD). Hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 may be coupled to system bus 1108 by hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. The interface 1124 for external drive implementation includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of computer readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will appreciate that other types of computer readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment and any such media may include computer executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored on the drive and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 1102 through one or more wired/wireless input devices, such as a keyboard 1138 and a pointing device such as a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. These and other input devices are often connected to processing unit 1104 through input device interface 1142, which is connected to system bus 1108, but may be connected by other interfaces such as parallel ports, IEEE 1394 serial ports, game ports, USB ports, IR interfaces, and the like.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communications. Remote computer(s) 1148 may be workstations, computing device computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, and generally include many or all of the components described for computer 1102, although for simplicity, only memory storage device 1150 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154 . Such LAN and WAN networking environments are common in offices and corporations and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to worldwide computer networks, such as the Internet.

When used in a LAN networking environment, computer 1102 connects to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communications to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, computer 1102 may include a modem 1158, be coupled to a communicating computing device on WAN 1154, or have other means for establishing communications over WAN 1154, such as over the Internet. A modem 1158, which may be internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored on remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between computers may be used.

Computer 1102 is operative to communicate with any wireless device or entity that is deployed and operating in wireless communication, e.g., printers, scanners, desktop and/or portable computers, portable data assistants (PDAs), communications satellites, any equipment or location associated with wireless detectable tags, and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, the communication may be a predefined structure as in conventional networks or simply an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) makes it possible to connect to the Internet without wires. Wi-Fi is a wireless technology, such as a cell phone, that allows such devices, eg, computers, to transmit and receive data both indoors and outdoors, i.e. anywhere within coverage of a base station. Wi-Fi networks use a radio technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, for example at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band).

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, instructions, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of program or design code (for convenience, referred to herein as software), or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. By way of example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure, and the general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present disclosure is not to be limited to the embodiments presented herein, but is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

Claims (14)

A method of generating a multi-task network model for performing a plurality of tasks implemented by a computing device comprising at least one processor, comprising:
determining a corresponding frame rate for each of the plurality of tasks;
setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, each of the plurality of frame rate networks having a corresponding frame rate designating a frame rate for processing image input data; and
training the multi-task network;
including,
Determining a corresponding frame rate for each of the plurality of tasks comprises:
training a plurality of single task networks for a first task included in the plurality of tasks, wherein the plurality of single task networks respectively process input data according to different corresponding frame rates;
performing performance evaluation of each of the plurality of single task networks for the first task;
determining a network having the highest performance for the first task among the plurality of single task networks as a corresponding single task network for the first task; and
determining the determined corresponding frame rate of the corresponding single task network as the corresponding frame rate of the first task;
including,
method. delete According to claim 1,
Setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network includes:
determining a frame rate network having a corresponding frame rate equal to that of the first task from among the plurality of frame rate networks; and
adding a first task network for the first task on the determined frame rate network;
including,
method. According to claim 3,
each of the plurality of frame rate networks includes a shared network and at least one task network;
A combination of a shared network included in a frame rate network having a corresponding frame rate equal to that of the first task and the first task network has network characteristics of a corresponding single task network for the first task, wherein the network characteristics include a channel configuration of a convolutional layer,
method. delete According to claim 4,
the corresponding frame rate includes a first frame rate and a second frame rate that is slower than the first frame rate;
The plurality of frame rate networks include a first frame rate network having the first frame rate and a second frame rate network having the second frame rate, and the number of channels of the plurality of convolution layers of the combination of the shared network and one task network included in the second frame rate network is greater than the number of channels of the plurality of convolution layers of the combination of the shared network and one task network included in the first frame rate network.
method. According to claim 6,
wherein the first frame rate is an integer multiple of the second frame rate;
method. According to claim 7,
The number of channels of the plurality of convolution layers of the combination of the shared network and one task network included in the second frame rate network is an integer multiple of the number of channels of the plurality of convolution layers of the combination of the shared network and one task network included in the first frame rate network,
method. According to claim 4,
Training the multi-task network is:
learning a combination of a shared network included in a frame rate network having a corresponding frame rate equal to that of the first task and the first task network by using learning data for the first task;
including,
method. According to claim 1,
The input data is 4-dimensional or 5-dimensional data including frame information for an image,
Each of the plurality of frame rate networks processes data having frame information corresponding to a corresponding frame rate among the input data.
method. According to claim 1,
The plurality of tasks include detecting a plurality of abnormal behaviors of an object on an image.
method. A method for performing a plurality of tasks implemented by a computing device comprising at least one processor, comprising:
processing input data through a multi-task network model to generate a plurality of output values for the plurality of tasks; or
processing input data through a shared network and a task network corresponding to the first task included in the multi-task network model to generate an output value for the first task included in the plurality of tasks;
including,
The method for generating the multi-task network model is:
determining a corresponding frame rate for each of the plurality of tasks;
setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, each of the plurality of frame rate networks having a corresponding frame rate designating a frame rate for processing image input data; and
training the multi-task network;
including,
Determining a corresponding frame rate for each of the plurality of tasks comprises:
training a plurality of single task networks for a first task included in the plurality of tasks, wherein the plurality of single task networks respectively process input data according to different corresponding frame rates;
performing performance evaluation of each of the plurality of single task networks for the first task;
determining a network having the highest performance for the first task among the plurality of single task networks as a corresponding single task network for the first task; and
determining the determined corresponding frame rate of the corresponding single task network as the corresponding frame rate of the first task;
including,
method. A computing device that creates a multi-task network model for performing a plurality of tasks,
a processor comprising at least one core;
a memory containing program codes executable by the processor; and
a network unit for receiving video input data;
including,
the processor,
determining a corresponding frame rate for each of the plurality of tasks;
Setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, each of the plurality of frame rate networks having a corresponding frame rate designating a frame rate for processing video input data, and
training the multi-task network;
When determining a corresponding frame rate for each of the plurality of tasks:
training a plurality of single-task networks for a first task included in the plurality of tasks, wherein the plurality of single-task networks respectively process input data according to different corresponding frame rates;
perform performance evaluation of each of the plurality of single task networks for the first task;
determining a network having the highest performance for the first task among the plurality of single task networks as a corresponding single task network for the first task; and
determining the determined corresponding frame rate of the corresponding single task network as the corresponding frame rate of the first task;
Device. A computer program stored on a computer-readable storage medium, which, when executed on one or more processors, causes the following operations to generate a multi-task network model for performing a plurality of tasks, the operations comprising:
determining a corresponding frame rate for each of the plurality of tasks;
setting a task network for each of the plurality of tasks on a frame rate network having the determined corresponding frame rate among a plurality of frame rate networks included in the multi-task network, each of the plurality of frame rate networks having a corresponding frame rate designating a frame rate for processing image input data; and
training the multi-task network;
including,
The operation of determining the corresponding frame rate for each of the plurality of tasks is:
training a plurality of single task networks for a first task included in the plurality of tasks, wherein the plurality of single task networks process input data according to different corresponding frame rates;
performing performance evaluation of each of the plurality of single task networks for the first task;
determining a network having the highest performance for the first task among the plurality of single task networks as a corresponding single task network for the first task; and
determining the determined corresponding frame rate of the corresponding single task network as the corresponding frame rate of the first task;
including,
A computer program stored on a computer-readable storage medium.