KR102263017B1 - Method and apparatus for high-speed image recognition using 3d convolutional neural network - Google Patents

Abstract

A high-speed image recognition method and apparatus using a 3D CNN (3-dimension convolutional neural network) are disclosed. A high-speed image recognition method using a 3D CNN (3-dimension convolutional neural network) comprises: inputting first image clips from among image clips constituting an input image to the 3D CNN, respectively, for each of the first image clips Obtaining result values of calculating a softmax function through 3D CNN, calculating a score margin using the obtained result values, comparing the calculated score margin with a preset threshold and determining whether to input the remaining video clips excluding the first video clips from among video clips constituting the input video to the 3D CNN in response to the step of performing and comparing the video clip. Therefore, it is possible to improve the operation speed for image recognition.

Description

High-speed image recognition method and apparatus using 3D CNN {METHOD AND APPARATUS FOR HIGH-SPEED IMAGE RECOGNITION USING 3D CONVOLUTIONAL NEURAL NETWORK}

The present invention relates to a method and apparatus for high-speed image recognition using a 3D CNN, and more particularly, a network operation for image recognition using a 3D CNN on some of the input image clips, and a subsequent image clip based on the result It relates to a technique for speeding up the operation speed by partially omitting the network operation for .

As artificial intelligence technology develops, deep learning techniques, which are machine learning algorithms that allow computers to perform human thinking through high-level abstraction, are being studied. These deep learning techniques use various artificial neural networks such as Deep Neural Network, Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), etc. make inferences about

In particular, the convolutional neural network is a network that has been attracting attention as showing excellent performance in image classification, and includes one or more convolutional layers.

Image recognition technology using a convolutional neural network is mainly used to identify an object included in an image or to recognize the behavior of an object (or a person). However, a 3D convolutional neural network, which is mainly used to recognize the behavior of an object, uses not a single 2D image but a 3D video image composed of a plurality of 2D images as an input.

Because the conventional 3D convolutional neural network requires a lot of resources to process a lot of computation and variables due to the use of a deep network, it is difficult to implement with limited resources of small devices including IoT (Internet of Things) devices. there is

An object of the present invention for solving the above problems is to provide a high-speed image recognition method using a 3D CNN.

Another object of the present invention for solving the above problems is to provide a high-speed image recognition apparatus using a 3D CNN.

One aspect of the present invention for achieving the above object provides a high-speed image recognition method using a 3D CNN.

The high-speed image recognition method using the 3D CNN includes inputting first image clips among image clips constituting an input image to a 3D CNN (3-dimension convolutional neural network), respectively, for each of the first image clips Obtaining result values of calculating a softmax function through the 3D CNN, calculating a score margin by using the obtained result values, and setting the calculated score margin with a preset threshold value Comparing and in response to the comparing, determining whether to input the remaining video clips excluding the first video clips among video clips constituting the input video to the 3D CNN. .

The score margin may be a difference value between a largest value and a second largest value among the result values.

The determining whether to input the remaining video clips to the 3D CNN may include, if the score margin is greater than the threshold, do not input video clips after the first video clips into the 3D CNN, and the result value It may include performing image recognition on the input image using only

The step of determining whether to input the remaining video clips to the 3D CNN may include inputting video clips after the first video clips to the 3D CNN if the score margin is less than the threshold value. can

The obtaining of the result values may further include accumulating and storing the result values obtained by calculating the softmax function in a memory.

The threshold value may be determined according to at least one of a type of a terminal performing image recognition, arithmetic capability, a type of an input image, a resolution of the input image, and the number of frames constituting the input image.

Each of the image clips constituting the input image may be composed of a preset number of temporally consecutive frames from among a plurality of frames constituting the input image.

Another aspect of the present invention for achieving the above object provides a high-speed image recognition apparatus using a 3D CNN.

A high-speed image recognition apparatus using a 3D CNN may include at least one processor, and a memory for storing instructions instructing the at least one processor to perform at least one step. have.

The at least one step may include inputting first video clips from among video clips constituting an input video to a 3D CNN (3-dimension convolutional neural network), respectively, and using the 3D CNN for each of the first video clips. obtaining results obtained by calculating a softmax function through the steps of, calculating a score margin using the obtained result values, comparing the calculated score margin with a preset threshold value, and In response to the comparing, the method may include determining whether to input the remaining video clips excluding the first video clips among video clips constituting the input video to the 3D CNN.

The score margin may be a difference value between a largest value and a second largest value among the result values.

The determining whether to input the remaining video clips to the 3D CNN may include, if the score margin is greater than the threshold, do not input video clips after the first video clips into the 3D CNN, and the result value It may include performing image recognition on the input image using only

The step of determining whether to input the remaining video clips to the 3D CNN may include inputting video clips after the first video clips to the 3D CNN if the score margin is less than the threshold value. can

The obtaining of the result values may further include accumulating and storing the result values obtained by calculating the softmax function in a memory.

The threshold value may be determined according to at least one of a type of a terminal performing image recognition, arithmetic capability, a type of an input image, a resolution of the input image, and the number of frames constituting the input image.

Each of the image clips constituting the input image may be composed of a preset number of temporally consecutive frames from among a plurality of frames constituting the input image.

In the case of using the high-speed image recognition method and apparatus using 3D CNN according to the present invention as described above, calculation speed can be improved and system resource requirements can be lowered by omitting calculations for subsequent video clips according to a score margin.

In addition, there is an advantage that image recognition can be performed using 3D CNN even in various devices with limited resources.

1 is an exemplary diagram for explaining a two-dimensional convolutional neural network according to an embodiment of the present invention.
2 is an exemplary diagram for explaining a 3D CNN according to an embodiment of the present invention.
3A and 3B are histograms for explaining a score margin value according to an embodiment of the present invention.
4 is a flowchart of a high-speed image recognition method using a 3D CNN according to the first embodiment of the present invention.
5 is a flowchart of a high-speed image recognition method dynamically using a 3D CNN according to a second embodiment of the present invention.
6 is a block diagram of a high-speed image recognition apparatus using a 3D CNN according to the first embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram for explaining a two-dimensional convolutional neural network according to an embodiment of the present invention.

Referring to FIG. 1 , a basic hierarchical structure of a two-dimensional convolutional neural network (CNN) may be confirmed. Specifically, the two-dimensional convolutional neural network receives an input image as an input and performs a convolution operation using a convolutional layer (10) and an activation function to output a feature map. An activation layer (11) that normalizes the output value of the convolutional layer (10), a pooling layer (pooling layer, 12) that extracts representative features by performing sampling or pooling on the output of the activation layer (11) may include In this case, several connection structures of the convolutional layer 10 , the activation layer 11 , and the pooling layer 12 may be repeatedly configured. In addition, in the convolutional neural network, a fully-connected layer 13 that combines several features extracted through a pooling layer 12 is connected to the rear end of the connection structure, and a softmax function It can be connected to a softmax layer 14 that normalizes the output of the pre-coupling layer 13 using the same.

The convolutional layer 10 may perform a convolution operation between the input image and the filter. The filter may be defined as a pixel unit area having a component value for performing a convolution operation with each pixel of the input image. In this case, the pixel unit area may be referred to as the size of the filter, and the filter may be generally expressed as a matrix. The convolutional layer 10 may repeat the convolution operation between the filter and the input image while sliding the filter in horizontal and vertical directions of the input image. In this case, an interval at which the filter moves at one time may be defined as a stride. For example, if the stride value is 2, a convolution operation with the input image may be performed while the filter is moved by an interval of two pixels. Also, as the convolutional layer 10 is repeated, the size of the output image (or feature map) may decrease. The convolutional layer performs a padding process to adjust the size of the output feature map. can be done Here, the padding process may be a process of filling the outer region of the input image with a specific value (eg, 0).

In the activation layer 11 , the activation function is a function that converts a feature extracted with a certain value (or matrix) into a nonlinear value, and a sigmoid function, a ReLU function, or the like may be used. In FIG. 1 , the activation layer 11 is illustrated separately from the convolutional layer 10 for convenience of description, but the activation layer 11 may be interpreted as being included in the convolutional layer 10 .

The pooling layer 12 is a layer that selects a feature representing the feature map by performing subsampling or pooling on the extracted feature map, and extracts the largest value for a certain area of the feature map. Max pooling, average pooling for extracting an average value, etc. may be performed. In this case, the pooling layer 12 is not necessarily performed after the activation layer 11, but may be selectively performed.

The precoupling layer 13 is generally located at the end of the CNN, and the precoupling layer 13 combines features extracted through the convolutional layer 10, the activation layer 11, and the pooling layer 12, You can determine which class it belongs to.

Specifically, the pre-coupling layer 13 may vectorize all pixels of the input feature map, multiply each parameter value, synthesize the calculation results, and output the class having the largest value as a result. The softmax layer 14 may express the operation result value in the pre-coupling layer 13 as a probability value between 0 and 1 using a softmax function. For example, the softmax function may be a function having a characteristic that all input values are normalized to values between 0 and 1 and the sum of output values is always 1. In FIG. 1 , the softmax layer 14 is illustrated separately from the precoupling layer 13 for convenience of description, but may be interpreted as being included in the precoupling layer 13 .

2 is an exemplary diagram for explaining a 3D CNN according to an embodiment of the present invention.

A three-dimensional convolutional neural network (hereinafter, 3D CNN) can be interpreted as an artificial neural network in which the two-dimensional convolutional neural network according to FIG. 1 is extended by one dimension along the time axis. The two-dimensional convolutional neural network according to FIG. 1 generally receives an image as input, and can be mainly used for purposes such as classifying the input image through spatial characteristics on the input image or identifying an object inside the input image. .

However, the 2D convolutional neural network has a limitation in that it cannot process video data including time information. On the other hand, since the 3D CNN performs convolution and pooling operations in consideration of the temporal component of the moving picture data, features can be extracted in consideration of the temporal properties of the moving picture data.

Specifically, referring to FIG. 2 , first, an input image 20 , which is moving image data composed of a plurality of frames (or pictures) along a time axis, is classified into a plurality of image clips 21 , and each image clip is divided into a 3D CNN It can be used as an input to (22). At this time, the video clip 21 is composed of a preset number of frames (the number of frames that the 3D CNN can process at once). For example, the video clip 21 may be composed of consecutive frames on the time axis. have. In addition, each frame (f=0, f=1 in the example according to FIG. 2) may be composed of K channels, and each channel may be composed of an image having a resolution of W·H. For example, if each frame is an image of RGB components, the number of channels may be three according to each component of R (Red), G (Green), and B (Blue).

The structure of the 3D CNN 22 is basically the same as or similar to the two-dimensional convolutional neural network according to FIG. 1 , but there may be a difference in using all image data along the time axis. For example, the convolutional layer of the 3D CNN 22 performs a convolution operation while the filter moves as if it scans an image like a two-dimensional convolution, and the convolution operation is performed by moving as much as the stride value on the time axis. can be done In addition, the pooling layer according to the 3D CNN 22 is a one-dimensional extension of the pooling layer 12 described with reference to FIG. 1 on the time axis, and all pixel values along the time axis can be used. The pre-combined layer according to the 3D CNN 22 obtains a weighted sum with parameters by vectorizing all pixels present in the last feature map, similarly to the pre-combining layer 13 according to FIG. 1, and a soap according to the 3D CNN 22 The max layer may operate like the softmax layer 14 according to FIG. 1 .

As such, the 3D CNN 22 may be advantageous in learning a temporally changing human motion because it performs learning in consideration of image data on the time axis. However, since image data on the time axis must be considered together, there is a problem in that more parameters and computations are required than in the 2D convolutional neural network.

Therefore, the present invention proposes a method capable of reducing the amount of computation according to 3D CNN and performing image recognition at high speed.

3A and 3B are histograms for explaining a score margin value according to an embodiment of the present invention.

In a general 3D CNN, a softmax value is calculated through the same 3D CNN for all video clips constituting an input image as shown in FIG. 2, and an image is recognized using the calculated softmax value. However, when calculating the softmax value through the 3D CNN for all video clips, there is a problem in that the calculation speed is lowered due to the large amount of calculation. In particular, since it is difficult to handle an excessive amount of computation in a small terminal capable of using only limited computational resources, a method of reducing the computational amount and recognizing an image at high speed is required.

In one embodiment of the present invention, as a means for solving this problem, the concept of a score margin may be defined. The score margin may be defined by Equation 1 below.

Referring to Equation 1, the score margin is the largest value (V _softmax1 ) and the second largest value (V softmax1 ) among the results of calculating the softmax function for each video clip through 3D CNN so far It can be defined as the difference value between V _{softmax2 ).} In this case, since the value calculated through the soft max function has a value between 0 and 1, the score margin may also have a value between 0 and 1.

In order to understand how much the score margin according to Equation 1 affects the success or failure of image recognition, graphs of the result of the score margin calculated for the UCF101 data set are shown in FIGS. 3A and 3B .

First, referring to FIG. 3A , the distribution of the score margin value (horizontal axis) of the image data (vertical axis) according to the case in which image recognition is successful can be confirmed. Data having a score margin value between 0.9 and 1 are significantly superior to each other. You can know a lot.

Also, referring to FIG. 3B , it can be seen that the distribution of the score margin value (horizontal axis) of the image data (vertical axis) according to the case in which image recognition fails. can

Therefore, combining FIGS. 3A and 3B , if the score margin is large enough, it can be determined that image recognition for the input image is successful only with the image clips analyzed through the current 3D CNN. Therefore, the need to perform image recognition is low. Hereinafter, when it is determined that image recognition is successful by evaluating the score margin calculated by inputting video clips to the 3D CNN so far, the process of inputting subsequent video clips into the 3D CNN is omitted, or the 3D CNN with low computational complexity. We propose a method for performing analysis on subsequent video clips using

4 is a flowchart of a high-speed image recognition method using a 3D CNN according to the first embodiment of the present invention.

Referring to FIG. 4 , in the high-speed image recognition method using the 3D CNN according to the first embodiment, first image clips among image clips constituting an input image are respectively input to a 3D CNN (3-dimension convolutional neural network). Step S100, obtaining result values of calculating a softmax function through the 3D CNN for each of the first video clips (S110), and using the obtained result values to obtain a score margin margin) calculating (S120), comparing the calculated score margin with a preset threshold (S130), and in response to the comparing, the first image among the video clips constituting the input image It may include the step of determining whether to input the remaining video clips excluding the clips to the 3D CNN (S140).

Here, the first video clips may mean one first video clip to be input to the 3D CNN, or may mean a plurality of video clips from the first video clip.

The score margin may be a difference value between a largest value and a second largest value among the result values. For example, the score margin may be defined according to Equation (1).

In the step of determining whether to input the remaining video clips to the 3D CNN (S140), if the score margin is greater than the threshold, the video clips after the first video clips are not input to the 3D CNN, The method may include performing image recognition on the input image using only the result values. Accordingly, an image recognition result may be finally derived only by the 3D CNN analysis of the first image clips, and the analysis of the image clips after the first image clips may be omitted.

The step of determining whether to input the remaining video clips to the 3D CNN (S140) may include: if the score margin is less than the threshold, inputting video clips after the first video clips to the 3D CNN may include.

Therefore, whenever video clips after the first video clips are input, the score margin is repeatedly obtained and the threshold value comparison is performed, thereby finally deriving the image recognition result at the current stage whether to input the next video clip and ending the image recognition You can decide whether to

The step of obtaining the result values ( S110 ) may further include accumulating and storing the result values obtained by calculating the softmax function in a memory. That is, by continuously accumulating and storing the result values, the score margin according to step S120 can be calculated by additionally including the result value of the softmax function calculated by inputting the next video clip to the 3D CNN in the stored result values. .

The threshold value may be determined according to at least one of a type of a terminal performing image recognition, arithmetic capability, a type of an input image, a resolution of the input image, and the number of frames constituting the input image.

Each of the image clips constituting the input image may be composed of a preset number of temporally consecutive frames from among a plurality of frames constituting the input image.

5 is a flowchart of a high-speed image recognition method dynamically using a 3D CNN according to a second embodiment of the present invention.

Referring to FIG. 5 , in the high-speed image recognition method dynamically using a 3D CNN (3-dimension convolutional neural network) according to the second embodiment, the first image clips among the image clips constituting the input image are respectively 3D CNN ( Step (S200) of inputting to a 3-dimension convolutional neural network), Step (S210) of obtaining results obtained by calculating a softmax function through the 3D CNN for each of the first video clips (S210), Obtaining In response to the steps of calculating a score margin using the obtained result values (S220), comparing the calculated score margin with a preset threshold value (S230), and in response to the comparing, the input image is It may include determining whether to input the next video clip of the first video clips among the video clips constituting the video clip to the same network as the 3D CNN (S240).

Here, the first video clips may mean one first video clip to be input to the 3D CNN, or may mean a plurality of video clips from the first video clip.

The score margin may be a difference value between a largest value and a second largest value among the result values. For example, the score margin may be defined according to Equation (1).

In the step of determining whether to input the next video clip to the same network as the 3D CNN (S240), if the score margin is greater than the threshold value, the next video clip of the first video clips is the same as the 3D CNN or It may include inputting into a network shallower than the 3D CNN. That is, if the score margin is greater than the threshold, the image recognition result inferred from the currently input video clips is highly likely to be correct. Therefore, the next video clip is calculated by inputting it to the same or shallower network as the 3D CNN used for the current inference. Speed can be improved. In this case, the shallow network may mean a network having a small number of convolutional layers or a low computational complexity.

The step of determining whether to input the next video clip to the same network as the 3D CNN (S240) is, if the score margin is less than the threshold value, the next video clip of the first video clips is more than the 3D CNN It may include entering into a deep network. That is, if the score margin is smaller than the threshold, the image recognition result inferred from the currently input video clips is highly likely to be wrong. Therefore, the next video clip is input into a deeper network than the 3D CNN used for the current inference to improve the computation speed. can do it In this case, a deep network may mean a network having a large number of convolutional layers or a high computational complexity.

The step of obtaining the result values ( S210 ) may further include accumulating and storing the result values obtained by calculating the softmax function in a memory. That is, the result values are continuously accumulated and stored, and the result value of the softmax function for the next video clip is additionally included in the previously stored result values, so that the score margin according to step S220 can be calculated.

The threshold value may be determined according to at least one of a type of a terminal performing image recognition, arithmetic capability, a type of an input image, a resolution of the input image, and the number of frames constituting the input image.

Each of the image clips constituting the input image may be composed of a preset number of temporally consecutive frames from among a plurality of frames constituting the input image.

If the network to which the next video clip is to be input is determined in step S240, the next video of the first video clips is input to the network determined in step S240, and the process from step S210 to step S240 is repeated, thereby all video clips constituting the input video can dynamically determine the network for

In addition, if the next video clip is the last video clip in step S240, the last video clip is input to the network determined in step S240 to calculate the softmax function, and finally image recognition by synthesizing the result values of the softmax function calculated during the operation By deriving a result, image recognition can be terminated.

Meanwhile, the first embodiment and the second embodiment according to FIGS. 4 and 5 may be implemented in combination with each other. Both the first and second embodiments may omit subsequent calculations based on the score margin defined in the present invention or use a different network to be applied. Accordingly, step S140 according to the first embodiment is applied by comparing the score margin calculated through steps S100 to S120 according to the first embodiment with a first threshold value, and the score margin calculated through steps S100 to S120 according to the first embodiment is applied. Step S240 according to the second embodiment may be applied by comparing with the threshold value 2 . Here, the first threshold value and the second threshold value may be set to different values, but setting the same value is not excluded.

6 is a block diagram of a high-speed image recognition apparatus using a 3D CNN according to a first embodiment of the present invention.

Referring to FIG. 6 , in the high-speed image recognition apparatus 100 using a 3D CNN according to the first embodiment, at least one processor 110 and the at least one processor 110 perform at least one step. It may include a memory 120 for storing instructions instructing to be performed.

Here, the at least one processor 110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. can Each of the memory 120 and the storage device 160 may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 120 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

In addition, the high-speed image recognition apparatus 100 using the 3D CNN may include a transceiver 130 that performs communication through a wireless network. In addition, the high-speed image recognition apparatus 100 using the 3D CNN may further include an input interface device 140 , an output interface device 150 , a storage device 160 , and the like. Each of the components included in the high-speed image recognition apparatus 100 using the 3D CNN may be connected by a bus 170 to communicate with each other.

The at least one step may include inputting first video clips from among video clips constituting an input video to a 3D CNN (3-dimension convolutional neural network), respectively, and using the 3D CNN for each of the first video clips. obtaining results obtained by calculating a softmax function through the steps of, calculating a score margin using the obtained result values, comparing the calculated score margin with a preset threshold value, and In response to the comparing, the method may include determining whether to input the remaining video clips excluding the first video clips among video clips constituting the input video to the 3D CNN.

The score margin may be a difference value between a largest value and a second largest value among the result values.

The determining whether to input the remaining video clips to the 3D CNN may include, if the score margin is greater than the threshold, do not input video clips after the first video clips into the 3D CNN, and the result value It may include performing image recognition on the input image using only

The step of determining whether to input the remaining video clips to the 3D CNN may include inputting video clips after the first video clips to the 3D CNN if the score margin is less than the threshold value. can

The obtaining of the result values may further include accumulating and storing the result values obtained by calculating the softmax function in a memory.

The threshold value may be determined according to at least one of a type of a terminal performing image recognition, arithmetic capability, a type of an input image, a resolution of the input image, and the number of frames constituting the input image.

Each of the image clips constituting the input image may be composed of a preset number of temporally consecutive frames from among a plurality of frames constituting the input image.

For example, a high-speed image recognition device 100 using a 3D CNN, a communicable desktop computer (desktop computer), a laptop computer (laptop computer), a notebook (notebook), a smart phone (smart phone), a tablet PC (tablet PC) , mobile phone, smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game console, navigation device, digital camera, DMB (digital multimedia broadcasting) player, digital audio recorder, digital audio player, digital video recorder, digital video player, PDA (Personal Digital Assistant) etc.

Meanwhile, the high-speed image recognition method dynamically using the 3D CNN according to the second embodiment according to FIG. 5 may also be performed in the device having the hardware configuration as shown in FIG. 6 . In this case, the method according to FIG. 5 may be implemented and performed as instructions executed by a processor similarly to the apparatus according to FIG. 6 , and detailed description will be omitted to prevent duplicate description.

The methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software.

Examples of computer-readable media may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as at least one software module to perform the operations of the present invention, and vice versa.

In addition, the above-described method or apparatus may be implemented by combining all or part of its configuration or function, or may be implemented separately.

Although the above has been described with reference to the preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be done.

Claims (14)

As a high-speed image recognition method using a 3D CNN (3-dimension convolutional neural network),
Two or more first video clips among video clips constituting the input video are input to a 3D CNN, and softmax functions are respectively calculated through the 3D CNN for each of the two or more first video clips to obtain a soft obtaining max values;
calculating one score margin indicating a probability of successful recognition by using the softmax values;
comparing the score margin with a preset threshold; and
In response to the comparing step, it is determined whether to input an additional video clip from among video clips constituting the input video to the 3D CNN, and image recognition for the input video using only the first video clips or performing image recognition on the input image using the first image clips and the additional image clips;
A high-speed image recognition method comprising a. In claim 1,
and the score margin is a difference value between a largest value and a second largest value among the softmax values. In claim 1,
The step of determining whether to input the additional video clips to the 3D CNN comprises:
If the score margin is greater than the threshold, performing image recognition on the input image using only the first image clips without inputting the image clips after the first image clips to the 3D CNN. Including, a high-speed image recognition method. In claim 1,
The step of determining whether to input the additional video clips to the 3D CNN comprises:
When the score margin is less than the threshold, video clips after the first video clips are input to the 3D CNN, and image recognition for the input video using the first video clips and the additional video clip A high-speed image recognition method comprising the step of performing a. In claim 1,
The step of obtaining the softmax values comprises:
and accumulating and storing the softmax values obtained by calculating the softmax function in a memory. In claim 1,
The threshold is
A high-speed image recognition method that is determined according to at least one of a type of a terminal performing image recognition, arithmetic capability, a type of an input image, a resolution of the input image, and the number of frames constituting the input image. In claim 1,
Each of the video clips constituting the input video,
and a preset number of temporally consecutive frames among a plurality of frames constituting the input image. As a high-speed image recognition device using a 3D CNN (3-dimension convolutional neural network),
at least one processor; and
a memory for storing instructions instructing the at least one processor to perform at least one step;
The at least one step is
Two or more first video clips among video clips constituting the input video are input to a 3D CNN, and softmax functions are respectively calculated through the 3D CNN for each of the two or more first video clips to obtain a soft obtaining max values;
calculating one score margin representing a probability of successful recognition by using the softmax values;
comparing the score margin with a preset threshold; and
In response to the comparing step, it is determined whether to input an additional video clip from among video clips constituting the input video to the 3D CNN, and image recognition for the input video using only the first video clips or performing image recognition on the input image using the first image clips and the additional image clip;
A high-speed image recognition device comprising a. In claim 8,
and the score margin is a difference value between a largest value and a second largest value among the softmax values. In claim 8,
The step of determining whether to input the additional video clips to the 3D CNN comprises:
If the score margin is greater than the threshold, performing image recognition on the input image using only the first image clips without inputting the image clips after the first image clips to the 3D CNN. Including, high-speed image recognition device. In claim 8,
The step of determining whether to input the additional video clips to the 3D CNN comprises:
When the score margin is less than the threshold, video clips after the first video clips are input to the 3D CNN, and image recognition for the input video using the first video clips and the additional video clip A high-speed image recognition device comprising the step of performing. In claim 8,
The step of obtaining the softmax values comprises:
and accumulating and storing the softmax values obtained by calculating the softmax function in a memory. In claim 8,
The threshold is
A high-speed image recognition apparatus, which is determined according to at least one of a type of a terminal performing image recognition, an arithmetic capability, a type of an input image, a resolution of the input image, and the number of frames constituting the input image. In claim 8,
Each of the video clips constituting the input video,
and a preset number of temporally consecutive frames among a plurality of frames constituting the input image.

Images