KR102138680B1 - Apparatus for Video Recognition and Method thereof - Google Patents

Abstract

The present invention, as an image recognition device, discloses an image recognition device capable of recognizing the behavior of an object in an image using a multi-product neural network capable of video analysis. The image recognition apparatus of the present invention includes a stream generating unit for generating an action stream including motion information related to the action for each object in an input image including at least one object; And the first recognizer receiving position information indicating the positional relationship of the action stream from the generated action stream or the input image, and outputting at least one class vector as an index for classifying the action of the object. Recognition unit for recognizing their behavior; It includes.

Description

Image recognition device and method {Apparatus for Video Recognition and Method thereof}

The present invention relates to an image recognition device. More specifically, the present invention relates to an image recognition device that recognizes an object's behavior in an image using a neural network.

Behavior recognition can be used to classify human behavior in video images that contain both temporal and spatial information. The relationship between the subject in the image and the people around it, the subject and the subject, and the relationship between the object and another object can be used as an element to classify the behavior of a specific object in the image.

The neural network is a statistical learning algorithm originating from a neural network of biology in machine learning and cognitive science. Recently, techniques for mechanically recognizing images in computer vision and the like are actively being studied. Convolutional Neural Network (CNN) is a neural network consisting of one or several convolutional layers and artificial neural network layers placed on it. It is an artificial neural network mainly used for two-dimensional input data, image and voice analysis.

Conventional, image recognition technology using an artificial neural network has been mainly performed only on a still image, and there is a limit to analyze a relationship between a subject and an object in the still image. In addition, the conventional image recognition technology has a problem that does not consider the passage of time in analyzing the relationship between objects, including people.

Accordingly, there is a need to develop a technology capable of recognizing an image in consideration of a contextual interrelation between an object in an image and an object based on an artificial neural network capable of performing video analysis on a video basis.

Korean Open Patent No. 10-2017-0034226 (announcement)

The present invention has been devised to solve the above problems, and discloses an image recognition device. In particular, an image recognition device capable of recognizing the behavior of an object in a video using a neural network is disclosed.

The present invention has been devised to achieve the above object, and the image recognition device of the present invention is a stream generating unit that generates an action stream including motion information about the action for each object in an input image including at least one object. ; And the first recognizer receiving position information indicating the positional relationship of the action stream from the generated action stream or the input image, and outputting at least one class vector as an index for classifying the action of the object. Recognition unit for recognizing their behavior; It includes.

In the present invention, the stream generator generates a plurality of frame images by dividing the input image over time, and a region of interest that detects a region of interest including at least a portion of the object for each object in the generated frame images Detection unit; Further comprising, it is possible to generate the action stream using the detected region of interest.

In the present invention, the recognition unit adds a class vector of action streams generated for each object according to a preset method using the first recognizer; Further comprising, it is possible to recognize the behavior of the objects using the summed class vector.

In the present invention, the stream generator calculates a connection score between the regions of interest detected in adjacent different frame images, and considers the calculated connection score to connect the regions of interest detected in the different frame images. Area connection; The method may further include generating the action stream for each object using the connected regions of interest.

In the present invention, the stream generating unit rearranges the region of interest detected in the frame images in units of frame images, and positions information regarding mutual positions of the action streams from the combined stream generated by connecting the rearranged regions of interest. A location information calculating unit for calculating; It may further include.

In the present invention, the region of interest connection unit is a similarity function that inputs feature information of the regions of interest detected in each of the adjacent frame images as input, and a cross ratio that outputs an overlap ratio of the regions of interest detected in each of the adjacent frame images as output. A connection score calculation unit calculating a connection score by considering at least one of a similarity between a function and class information of the regions of interest; In addition, the region of interest may be connected by using the calculated connection score.

In the present invention, when a frame image in which the region of interest is not detected exists, the region of interest connection unit may include the region of interest based on feature information of regions of interest detected in frame images adjacent to the frame image in which the region of interest is not detected. A feature information estimator for estimating feature information of a region of interest in the undetected frame image; In addition, the region of interest may be connected using the estimated feature information.

In the present invention, the region of interest detector comprises: a feature information calculator for calculating feature information indicating coordinates of the region of interest in the generated frame image; In addition, the region of interest may be detected using the calculated feature information.

In the present invention, the location information calculating unit is an area dividing unit for classifying and dividing the region of interest and the portion of the region of interest that are detected for each object in the frame images; And a rearrangement unit to rearrange the divided regions of interest by a predetermined combination method. Further comprising, it is possible to calculate the location information based on the combined stream generated by connecting the rearranged regions of interest.

In the present invention, the feature information calculating unit is a pre-processing unit for generating grid cells of a predetermined size by dividing each of the generated frame images; And a second recognizer using the generated frame images as an input and outputting a center coordinate of a boundary cell indicating a probability that the object exists with a center in the grid cell or a probability that the object exists in the boundary cell as an output. A calculation unit that calculates a center coordinate of each boundary cell and a probability that the object exists in the boundary cell using (Neural Network); In addition, the feature information may be calculated using a boundary cell in which the probability and the center coordinates are calculated.

In the present invention, the region-of-interest detection unit overlaps the object in consideration of whether a probability that the object exists in the boundary cell among the boundary cells including the object in the frame images is equal to or greater than a preset threshold. A boundary cell removal unit for removing a portion of the boundary cell; In addition, the region of interest may be detected using the removed and remaining boundary cells.

In the present invention, the class vector includes a list of actions to classify the behavior of the object and probability information indicating a probability that the actions of objects in the input image correspond to the list of actions, and the recognition unit includes the sum of the summed class vector. The behavior of the objects may be recognized using the probability information for each behavior list.

In addition, in order to achieve the above object, the image recognition method of the present invention includes generating an action stream including motion information on the action for each object in an input image including at least one object; And a first recognizer receiving at least one class vector as an index for classifying the behavior of the object by receiving position information indicating a positional relationship of the action stream from the generated action stream or the input image. Recognizing the behavior of the objects; It includes.

In the present invention, the generating step includes dividing the input image over time to generate a plurality of frame images, and detecting a region of interest including at least a portion of the object for each object in the generated frame images. ; Further comprising, it is possible to generate the action stream using the detected region of interest.

In the present invention, the step of recognizing includes summing a class vector of action streams generated for each object according to a preset method using the first recognizer; Further comprising, it is possible to recognize the behavior of the objects using the summed class vector.

In the present invention, the generating step calculates the connection points between the regions of interest detected in adjacent different frame images, and connects the regions of interest detected in the different frame images in consideration of the calculated connection points. step; The method may further include generating the action stream for each object using the connected regions of interest.

In the present invention, the generating step rearranges the region of interest detected in the frame images in units of frame images, and positions the mutually positioned locations of the action streams from the combined stream generated by connecting the rearranged regions of interest. Calculating information; It may further include.

In the present invention, the step of connecting is a similarity function using the feature information of the regions of interest detected in each of the adjacent frame images as an input, and an intersection ratio of the overlap ratio of the regions of interest detected in each of the adjacent frame images as the output. Calculating a connection score considering at least one of a similarity between a function and class information of the regions of interest; In addition, the region of interest may be connected by using the calculated connection score.

In the present invention, when the frame image in which the region of interest is not detected exists, the step of connecting may include the region of interest based on feature information of regions of interest detected in frame images adjacent to the frame image in which the region of interest is not detected. Estimating feature information of a region of interest in the undetected frame image; In addition, the region of interest may be connected using the estimated feature information.

In the present invention, the detecting may include calculating feature information indicating coordinates of the region of interest in the generated frame image; In addition, the region of interest may be detected using the calculated feature information.

In the present invention, calculating the location information may include distinguishing the region of interest detected by the object from the frame images and a portion of the region where the region of interest is not detected; And rearranging the divided regions of interest in a predetermined combination method. Further comprising, it is possible to calculate the location information based on the combined stream generated by connecting the rearranged regions of interest.

In addition, the present invention discloses a computer program stored in a computer-readable recording medium for executing the above-described image recognition method in a computer.

According to the present invention, the behavior of an object in an image can be recognized.

In particular, an image may be recognized using a convolutional neural network capable of video-based image analysis.

1 is a block diagram of an image recognition apparatus according to an embodiment of the present invention.
FIG. 2 is an enlarged block diagram of a stream generator in the embodiment of FIG. 1.
3 is an enlarged block diagram of a region of interest detector in the embodiment of FIG. 2.
4 is an enlarged block diagram of a feature information calculator in the embodiment of FIG. 3.
5 is a reference diagram illustrating a process of detecting a region of interest by the region of interest detector.
6 is an exemplary view showing regions of interest detected by the region of interest detector.
7 is an enlarged block diagram of a region of interest connection in the embodiment of FIG. 2.
8 is a reference diagram illustrating a process in which a region of interest connection unit connects regions of interest.
9 is an exemplary view illustrating a process in which the feature information estimator estimates feature information of a region of interest.
10 is an enlarged block diagram of a location information calculator in the embodiment of FIG. 2.
11 shows an image recognition process performed by the image recognition device of the present invention.
12 is an enlarged block diagram of a recognition unit in the embodiment of FIG. 1.
13 is a flowchart of an image recognition method according to an embodiment of the present invention.
14 is an enlarged flow chart of steps generated in the embodiment of FIG. 13.
15 is an enlarged flow chart of the detection step in the embodiment of FIG. 14.
FIG. 16 is an enlarged flow chart of steps of connecting in the embodiment of FIG. 14.
FIG. 17 is an enlarged flow chart of steps for calculating location information in the embodiment of FIG. 14.
FIG. 18 is an enlarged flow chart of steps recognized in the embodiment of FIG. 13.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

In the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numbers and redundant descriptions thereof will be omitted.

In addition, in describing the present invention, when it is determined that detailed descriptions of related well-known configurations or functions may obscure the subject matter of the present invention, detailed descriptions thereof may be omitted.

Terms used in the present application are only used to describe specific embodiments, and are not intended to limit the terms. Singular expressions include plural expressions unless the context clearly indicates otherwise. Each step described below may be provided with one or several software modules, or may be implemented with hardware in charge of each function, or a combination of software and hardware.

The specific meaning and examples of each term will be described below according to the order of each drawing.

Hereinafter, a detailed description of the configuration of the image recognition device 10 according to an embodiment of the present invention.

1 is a block diagram of an image recognition device 10 according to an embodiment of the present invention.

The image recognition device 10 includes a stream generation unit 100 and a recognition unit 200. For example, the image recognition apparatus 10 of the present invention receives an image including at least one object, generates an action stream from the input image, and uses an image generated by using the first recognizer as an input. I can recognize the behavior of my object.

In addition to the still image, the image recognized by the image recognition apparatus 10 of the present invention includes a video image (video) in which image information changes over time. Unlike the conventional image recognition technology, the image recognition apparatus 10 of the present invention may recognize an image by performing image analysis in a video unit. More specifically, the image recognition device 10 of the present invention can recognize an image by classifying the behavior of an object in the image.

The image recognition device 10 of the present invention can be used to control autonomous driving by being used in an autonomous driving device such as an autonomous driving robot, or used in a surveillance device such as CCTV, and used for automatic analysis of an input image. In addition, the image recognition device 10 of the present invention is provided as a control device of a home Internet of Things (IOT) and can be used to control the operation of a household appliance by automatically recognizing a person's behavior.

In the present invention, the first recognizer and the second recognizer used by the image recognition apparatus 10 may be provided as a neural network that is supervised learning based on a deep learning algorithm having a deep neural network structure having multiple layers. Can. Preferably, the first recognizer and the second recognizer used by the image recognition apparatus 10 of the present invention are a neural network structure learned based on supervised learning, and a convolutional neural network including at least one convolutional layer and a pooling layer ( Convolutional Neural Network). The artificial neural network designed based on the back propagation algorithm has been delayed for a while because the problems such as delay in learning time due to excessive depth are revealed when the number of neural network layers increases, but the overfitting problem is solved through a method such as dropout. As a result, the performance of the algorithm has improved dramatically.

Since the convolutional neural network, which is a representative deep learning structure used by the image recognition device 10, is designed to mimic a human visual processing process, it is evaluated as a deep learning algorithm suitable for image and image processing, and abstracts images. By extracting features that can be expressed, it shows high performance in the field of image recognition.

Specifically, the convolutional neural network may hierarchically extract features from the input image and form a feature map using the extracted features. Features distributed on the feature map include rearranged image information, and through this, the composite product neural network can effectively classify the image. The convolutional neural network can resize the feature values extracted from the convolutional layer by applying an activation function to the features extracted through the convolutional layer and repeatedly arranging the pooling layer.

The convolutional neural network used by the image recognition apparatus 10 of the present invention is basically provided as a convolutional neural network structure, and a structure in the neural network may be provided differently according to the shape, size, and pixel value information of the input image. That is, the number of input layers, hidden layers, and output layers of the convolutional neural network used by the image recognition apparatus 10, the number of nodes included in each layer, the weight applied to the edges of each node, and the learning rate The lights may be set differently depending on the purpose of image recognition.

In addition, as the convolutional neural network used by the image recognition apparatus 10, a neural network to which structures such as VGG-Net, GoogLeNet, and ResNet are applied may be used as the neural network.

In addition, the neural networks used by the image recognition apparatus 10 of the present invention are structures in which a fully-connected layer is connected to a CNN structure including a convolutional layer and a pooling layer, resulting in overfitting problems. If not, the deeper the layer, the higher the image recognition accuracy. This will be described with reference to FIG. 2.

The stream generator 100 includes a region of interest detection unit 120, a region of interest connection unit 140, and a location information calculating unit 160. For example, the stream generator 100 generates an action stream including motion information on the action for each object from an input image including at least one object. The action stream of the present invention may be generated by pre-processing an action tube generated for each object present in an image in a form for input to a neural network.

The action tube of the present invention may be generated for each object as a set of 3D (X, Y, Z) image data including at least a part of the object in the input image. The action tube of the present invention may include motion information regarding a change in an object's behavior, including a change in a pixel value in regions of interest that changes over time. The motion information of the present invention is a type of change in pixel values in a region of interest that changes with the passage of time, and may be provided as a differential value of a change in pixel value for each pixel in a unit time.

For example, the stream generator 100 may divide the input image to generate frame images having a predetermined time interval, detect the region of interest for each object from the generated frame images, and connect the detected regions of interest By creating an action tube, an action stream can be generated by preprocessing the generated action tube.

The action stream generated by pre-processing the action tube generated by connecting the region of interest detected in frame images having a predetermined time interval is generated by pre-processing the action tube including the pixel value change amount that changes over time. It can represent the behavioral change of the object according to the flow.

The action stream generated for each object in the stream generator 100 of the present invention may have a size of 56*56*3*16. The action stream generated by the stream generator 100 is generated by connecting regions of interest having a horizontal*vertical size of 56*56, and includes pixel values (pixel values) of an RGB color system, and one action stream includes 16 images. It may include frames. The size of the action stream generated by the stream generator 100 may vary according to a setting method of the first recognizer. This will be described with reference to FIG. 3.

The region of interest detection unit 120 includes a feature information calculation unit 122 and a boundary cell removal unit 126. For example, the region-of-interest detection unit 120 divides the input image over time to generate a plurality of frame images, and is divided by the object in the generated frame images to include the region of interest including at least a part of the object Can be detected. This will be described with reference to FIG. 4.

The feature information calculating unit 122 includes a pre-processing unit 123 and a calculation unit 124. For example, the feature information calculating unit 122 may calculate feature information indicating coordinates in which the region of interest is located in the generated frame image. In the present invention, the feature information is coordinates related to the location of the region of interest to be detected in the frame image, and may be provided as a center coordinate of the region of interest in the frame image or a coordinate located at a boundary of the region of interest. This will be described with reference to FIG. 5.

The pre-processing unit 123 may divide the generated frame images, respectively, to generate a grid cell of a predetermined size, as shown in FIG. 5. For example, the pre-processing unit 123 may generate grid cells of 9*6 or 7*7 size. The grid cells generated by the pre-processing unit 123 may include boundary cells that are dependent on the grid cells.

The calculation unit 124 uses the generated frame images as an input, and outputs a center coordinate of a boundary cell indicating a probability that the object exists with a center in the grid cell or a probability that the object exists in the boundary cell as an output. The second coordinate (Neural Network) is used to calculate the center coordinates of each boundary cell and the probability that the object exists in the boundary cell.

The second recognizer of the present invention is a neural network structure learned based on supervised learning, and may be provided as a convolutional neural network including at least one convolutional layer and a pooling layer. Preferably, the second recognizer used by the calculation unit 124 may be provided as Regions with Convolutional Neural Network (R-CNN) or Faster R-CNN.

The feature information calculating unit 122 may calculate feature information using boundary coordinates in which the center coordinates of each boundary cell and the probability that the object exists in the boundary cell are calculated. For example, the second recognizer used by the calculator 124 is connected to at least one convolutional layer for extracting a feature and one end of the convolutional layer so that the center coordinates of the boundary cell and the object in the boundary cell are It may include at least one pulley connected layer to calculate the probability of existence. In addition, the second recognizer used by the calculator 140 of the present invention may further include a pooling layer alternately arranged with the convolutional layer in addition to the convolutional layer and the fully connected layer.

The process of calculating the feature information by the feature information calculating unit 122 according to an embodiment of the present invention is as follows. The feature information calculating unit 122 generates a frame image by dividing an input image including at least one object, and generates grid cells at predetermined intervals in the frame image. The calculation unit 124 generates a random number of boundary cells 127, 128, 129, 132, and 133 that have a center coordinate inside the grid cell and uses the second recognizer, and is dependent on the grid cell, It is possible to calculate the probability that objects included in the boundary cell exist.

The boundary cell removing unit 126 overlaps the object in consideration of whether a probability that the object exists in the boundary cell is greater than or equal to a preset threshold among boundary cells that overlap the object in frame images. Remove part of the cell. For example, the boundary cell removing unit 126 is an object inside among the boundary cells 127, 128, 129, 132, 133, and 134, all of which have a different probability that an object exists inside the boundary cell generated by the calculation unit 124. The boundary cells 132 and 133 having the highest probability of being present may be detected as a region of interest.

To this end, the boundary cell removal unit 126 leaves only the boundary cells 132 and 133 having a probability that an object exists among predetermined boundary cells overlapping an arbitrary object present in the frame image, which is greater than or equal to a preset threshold, and the remaining boundary cells. (127, 128, 129, 134) can be removed. In addition, the boundary cell removal unit 126 uses only a single boundary cell using a Nom-maximal Suppression (NMS) algorithm to remove boundary cells having a probability that a lesion exists within a cell below a preset threshold. Can be left. This will be described with reference to FIG. 6.

The region of interest detector 120, as described above, uses the feature information calculated by the feature information calculator 122 to generate at least one object including a person or an object from frame images generated by dividing the input image. The region of interest including the image may be detected. This will be described with reference to FIG. 7.

The region of interest connection unit 140 includes a connection score calculation unit 142 and a feature government estimation unit 144. For example, the region of interest connection unit 140 calculates a connection score between the regions of interest detected in adjacent different frame images, and the region of interest detected in the different frame images in consideration of the calculated connection points. Connect. This will be described with reference to FIG. 8.

For example, the region of interest connection unit 140 may include regions of interest 331, 333, and 335 for the first object detected from the plurality of frame images 302, 304, 306, and 308 generated by dividing the input image. , 337, 339, 341, 343) and the regions of interest for the second object (332, 334, 336, 338, 340, 342, 344) by connecting each of the first action tube 361 for the first object A second action tube 362 for the second object may be generated.

Also, the region of interest connection unit 140 may pre-process the generated first action tube 361 and the second action tube 362 for the second object to generate action streams for the first object and the second object, respectively. have. That is, it is as described above that the action streams of the present invention may include motion information on actions for each object according to the passage of time. In the present invention, the region of interest connection unit 140 may connect the regions of interest generated in the frame images to generate action tubes by connecting objects, and pre-process the generated action tubes to generate action streams for each object. .

The connection score calculating unit 142 is a similarity function that inputs feature information of the regions of interest detected in each of the adjacent frame images as an input, and a cross ratio that outputs an overlap ratio of the regions of interest detected in each of the adjacent frame images as an output. The connection score is calculated by considering at least one of the similarity between the function and the class information of the regions of interest.

는 인접한 프레임 영상들에서 각각 검출된 관심 영역들의 행동 분류를 위한 클래스 정보의 유사도,

및

는 인접한 프레임 영상들의 인덱스 번호,

는 임의의 스칼라 값,

는

인덱스 번호의 프레임 영상에서 검출된 관심 영역의 특징 정보,

는

인덱스 번호의 프레임 영상에 검출된 관심 영역의 특징 정보를 나타낸다. 본 발명의 프레임 영상에서 검출된 관심 영역의 특징 정보는 관심 영역이 해당 프레임 영상에서 위치하는 좌표 정보를 포함함은 전술한 바와 같다. 상기 클래스 정보는 관심 영역 에서 검출된 객체의 행동을 분류하는 카테고리(목록)를 의미한다.Here, sim() is a similarity function having an output value between 0 and 1 by inputting feature information of the regions of interest detected in each of the frame images, and ov() is over the regions of interest detected in adjacent frame images. Cross-ratio function with rep ratio as output,

Is similarity of class information for classifying behaviors of regions of interest detected in adjacent frame images,

And

Is an index number of adjacent frame images,

Is an arbitrary scalar value,

The

Feature information of the region of interest detected in the frame image of the index number,

The

Represents feature information of a region of interest detected in a frame image of an index number. It is as described above that the feature information of the region of interest detected in the frame image of the present invention includes coordinate information in which the region of interest is located in the corresponding frame image. The class information means a category (list) that classifies the behavior of the object detected in the region of interest.

In addition, in Equation 1, the ov() function is an intersection ratio function that outputs an Intersection Over Union (IOU) value, and inputs an intersection region and a union region of regions of interest detected in adjacent frame images. Specifically, the cross ratio function may output a value obtained by dividing the intersection region by a union region.

The process of calculating the link score by the link score calculating unit 142 according to an embodiment of the present invention will be described in the video frames 302 and 304 as follows. The connection score calculating unit 142 includes first areas in the first region of interest 331 and the second image frame 304 of the first object among the regions of interest 331 and 332 detected in the first image frame 302. The connection score of the region of interest 333 is calculated.

In addition, the connection score calculator 142 calculates the connection scores of the first region of interest 331 in the first image frame 302 and the second region of interest 334 in the second image frame 304. That is, the connection score calculating unit 142 of the present invention calculates a link score for all cases in which a region of interest in one image frame can be connected to a region of interest in another adjacent image frame. Can. The region of interest connection unit 140 of the present invention may connect regions of interest in an image frame such that the connection score is highest. This will be described with reference to FIG. 9.

The feature information estimator 144 detects regions of interest 335 detected in frame images 375 adjacent to the frame image 372 where the region of interest is not detected when a frame image in which the region of interest is not detected exists. Based on the feature information of 336, 337, 338, 341, 342, 343, 344, feature information of the region of interest in the frame image 372 in which the region of interest is not detected is estimated.

For example, when the region of interest detector 120 of the present invention divides the input image and detects each region of interest in the frame image, it may occur that the region of interest may not be properly extracted due to problems such as image quality in the image. . That is, as illustrated in FIG. 9, a frame image 372 in which the region of interest is not detected may exist among the frame images 375.

In this case, the feature information estimator 144 has feature information of regions of interest in the frame images 371, 373, 379 adjacent to the frame images 374, 376, 378 where the region of interest is not detected as a function input A linear function may be generated, and feature information of a region of interest in the frame image 374, 376, and 378 in which the region of interest is not detected may be estimated using the generated linear function.

The feature information of the present invention means coordinates of a region of interest in a frame image, and the feature information estimator 144 includes frame images 371 adjacent to frame images 374, 376, and 378 in which the region of interest is not detected. 373, 379), the location of the region of interest detected in the frame images 274, 276, and 278 in which the region of interest is not detected may be estimated based on the location of the region of interest.

That is, the feature information estimator 144 performs interpolation based on feature information of adjacent frame images 371, 373, and 379 in which the region of interest is detected, thereby interested in frame images in which the region of interest is not detected. The feature information that is the coordinate information of the region is estimated. The ROI-connector 140 may generate an estimated ROI in the frame image in which the ROI is not detected using the estimated feature information, and connect the estimated ROIs to ROIs in an adjacent frame image.

The location information calculating unit 160 includes an area division unit 162 and a rearrangement unit 164. For example, the location information calculating unit 160 rearranges the region of interest detected from the frame images in units of frame images, and positions the action streams from each other from a combined stream generated by connecting the rearranged regions of interest. Calculate location information about.

For example, the location information calculator 160 rearranges regions of interest detected for each object in the frame images in a predetermined combination method, generates a combination stream using the rearranged regions of interest, and generates the combined stream. Use to calculate the location information about the mutual location of the action stream generated for each object. In the present invention, the location information refers to a relationship between mutual locations of regions of interest detected for each object in a frame image, not external information, and a detailed method of calculating location information will be described later.

For example, the location information calculator 160 intersects regions of interest related to the first object and regions of interest related to the second object in an input image including the first object (person) and the second object (object). By arranging or densely arranging, a combination stream can be generated, and the generated combination stream can be used to calculate position information regarding the mutual position of the action stream for the first object and the action stream for the second object. The method for the location information calculating unit 160 of the present invention to calculate the location relation of the action stream generated for each object is not limited to masking or binary processing, and other known techniques for inputting the location relation of regions of interest Includes. This will be described with reference to FIG. 11.

The region dividing unit 162 performs masking by classifying the region of interest and the portion of the region of interest, which are detected for each object, in frame images. A case in which the region of interest detection unit 120 detects regions of interest related to the first object and the second object from an input image including the first object (person) and the second object (object) will be described as an example.

In this case, in this embodiment, the region dividing unit 162 may include a masking unit (not shown) that classifies and masks a portion in which the region of interest is not detected.

For example, the masking unit (not shown) may mask the part in which the regions of interest related to the first object are detected from the frame images and the portion not of the region of interest related to the first object into two pixel value information. . In addition, a portion in which regions of interest related to the second object are detected and a portion other than the region of interest related to the second object may be divided into two pixel value information and masked.

In the present invention, the region dividing unit 162 masking a portion in which the regions of interest are detected in the frame images and a portion in which the regions of interest are not detected may be provided as binary processing of the frame image. The location information calculating unit 160 of the present invention may calculate location information indicating a location relationship between regions of interest detected in each frame image based on a masking process performed by the region division unit 162.

The rearrangement unit 164 rearranges the divided regions of interest in a predetermined combination method. For example, the rearrangement unit 164 may combine the masked regions of interest 277 for the first object and the masked regions of interest 279 for the second object in a predetermined combination method. The method of combining the regions of interest in which the rearrangement unit 164 is divided is a masked region of interest for the first object, a masked region of interest for the second object, and a masked region of interest for the first object in consideration of frame-specific information. It includes a method of cross-combining frame images for each object, such as a region.

In addition, in the method of combining the regions of interest in which the rearrangement unit 164 is divided, the masked regions of interest related to the first object are first arranged in consideration of the information of the first object and the second object, and then the And later arranging the masked regions of interest.

The location information calculating unit 160 generates a combination stream using regions of interest rearranged by a predetermined combination method, and calculates location information regarding mutual positions of action streams generated for each object using the generated combination stream. Can. In the present invention, the combined stream generated by the location information calculating unit 160 is a pairwise stream including a pair of streams in which masked regions of interest related to a first object are connected and a pair of streams in which masked regions of interest related to a second object are connected. Stream).

The pairwise stream of the present invention includes location information indicating the positional relationship of the action stream for each object, excluding appearance information included in the action stream for each object in the image. That is, the object location information calculating unit 160 may calculate location information indicating the positional relationship between the action stream related to the first object and the action stream related to the second object using the above-described combination stream, through which the present invention The image recognition device 10 may recognize the behavior of an object based on the positional relationship of each object in the image.

The recognition unit 200 includes a summation unit 220 and a comparison unit 240. For example, the recognizer 200 uses the generated action stream or location information of the action stream as an input, and uses a first recognizer that outputs at least one class vector as an index for classifying the action of the object. The behavior of the objects can be recognized. The class vector used by the recognition unit 200 of the present invention includes an action list for classifying the action of the object and probability information indicating a probability that an action of objects in the input image corresponds to the action list. The recognizer 200 may recognize the behavior of the objects using the probability information for each behavior list of the summed class vector.

The summing unit 220 adds the class vectors of the action streams classified for each object according to a preset method using a first recognizer. For example, the summing unit 220 acts on the output value of the first recognizer using as input a combination stream including the positional relationship of the action stream generated for each object and the action stream generated for each object in the stream generation unit 100. Each list can be summed according to a preset method.

For example, the summing unit 220 is a first class vector that is an index for classifying an action of a first object (person) output when an action stream related to the first object (person) is input to the first recognizer, the first class vector, the first The position of the action stream generated for each person or object in the second class vector and the first recognizer, which are indicators for classifying the behavior of the second object (object) that is output when the action stream related to the second object (object) is input to the recognizer. The combination stream including the relationship may be added by applying different weights to the third class vector output upon input, or may be added by applying the same weight as 1/3. When the summing unit 220 adds the first class vector, the second class vector, and the third class vector by applying the same weight 1/3, the sum of the first class vector, the second class vector, and the third class vector is averaged. It can be prepared as a process of seeking.

The comparator 240 compares the probability information for each action list that classifies the actions in the summed class vector with the probability information for each action list of the object stored in the previously set correct answer data set (UCF-101 Detection Data Set). For example, the comparison unit 240 compares whether the behavior of the object represented by the class vector summed in the summation unit 220 is matched by using a data set in which probability information for each behavior list of a known object is stored, and the image is compared. I can accurately recognize the behavior of my object.

FIG. 2 is an enlarged block diagram of a stream generator in the embodiment of FIG. 1.

The stream generator 100 includes a region of interest detection unit 120, a region of interest connection unit 140, and a location information calculating unit 160. For example, the stream generating unit 100 generates an action stream including motion information related to the action of an object included in the input image, for each object. The action stream is created by connecting regions of interest detected in each image frame over time.

Motion information about the behavior of an object may include both the amount of change in the position of the regions of interest and the amount of change in pixel values in the region of interest over time, but only the amount of change in the pixel values in the region of interest, excluding the amount of change in the region of interest in the frame image can do.

For example, since the positions of the regions of interest located in the image frame and the pixel value information vary with the passage of time, the amount of change in the position of the regions of interest changing in the image frame according to the passage of time, the region of interest changing with the passage of time The amount of change in the pixel value within may constitute motion information about the behavior change of the object by itself. However, preferably, the motion information of the present invention can include only the amount of change in a pixel value in a region of interest that changes over time.

However, since the image recognition apparatus 10 of the present invention uses a pair-wise stream, it will be described later that the behavior of objects in the image can be recognized in consideration of all the amount of position change of regions of interest in the frame image.

The region-of-interest detection unit 120 divides the input image over time to generate a plurality of frame images, and is divided for each object in the generated frame images to detect a region of interest including at least a part of the object. have. The region of interest detector 120 may divide the input images into units of 16 frames, and may detect regions of interest mainly including an image of an object in each divided frame image.

The region of interest connection unit 140 connects regions of interest detected in divided frame images over time to generate action tubes for each object. As described above, the region of interest connection unit 140 may generate an action stream by pre-processing the generated action tubes according to an input form of the first recognizer.

The location information calculating unit 160 rearranges the region of interest detected from the frame images in frame image units, and the location information calculating unit 160 connects the rearranged regions of interest to the action from the combined stream. Calculate position information about the positions of the streams. The image recognition device 10 of the present invention may recognize the action of an object in the image using the location relationship of the action stream generated for each object in the image using the location information calculated by the location information calculator 160.

3 is an enlarged block diagram of a region of interest detector in the embodiment of FIG. 2.

The region of interest detector 120 includes a feature information calculator 122 and a boundary cell remover 126. The feature information calculator 122 calculates feature information as coordinates of a location of a region of interest to be detected in a frame image. The feature information calculating unit 122 may use a vision recognition algorithm including YOLO (You Only Look Once), Fast R-CNN, etc. based on a convolutional neural network to detect a region of interest in an image.

The boundary cell removing unit 126 overlaps the object in consideration of whether a probability that the object exists in the boundary cell is greater than or equal to a preset threshold among boundary cells that overlap the object in frame images. Remove part of the cell. The method in which the boundary cell removal unit 126 removes the boundary cell and detects the region of interest using the remaining boundary cell is as described above.

4 is an enlarged block diagram of a feature information calculator in the embodiment of FIG. 3.

The feature information calculating unit 122 includes a pre-processing unit 123 and a calculation unit 124. The pre-processing unit 123 may generate grid cells of a predetermined size by dividing each of the generated frame images, and the generated grid cells include boundary cells that are dependent on the grid cells.

The calculation unit 124 uses the generated frame images as an input, and outputs a center coordinate of a boundary cell indicating a probability that the object exists with a center in the grid cell or a probability that the object exists in the boundary cell as an output. The second coordinate (Neural Network) is used to calculate the center coordinates of each boundary cell and the probability that the object exists in the boundary cell.

The second recognizer of the present invention may include at least one convolutional layer that extracts convolutional features and a fully connected layer that is connected to one end of the convolutional layer and calculates the center coordinates of the boundary cell and the probability that an object in the boundary cell exists. have.

5 is a reference diagram illustrating a process of detecting a region of interest by the region of interest detector.

The boundary cells 127, 128, 129, 132, 133, and 134, which are dependent on the grid cells generated by the pre-processing unit 123, are dependent on the grid cells and represent the probability that the object-related part is included therein. As described above, the region of interest detector 120 may remove all boundary cells whose probability of existence of an object among the generated boundary cells is not equal to or greater than a preset threshold, and detect the region of interest using the remaining boundary cells. same.

As illustrated in FIG. 5, the region of interest detector 120 considers the probability indicated by the boundary cells, and the second object having a probability of interest and a 67% probability of the first object (person) having a probability of 99% Object). The region of interest detector 120 includes probability information of regions of interest used to detect the region of interest, including the above-described Intersection Over Union (IOU).

6 is an exemplary view showing regions of interest detected by the region of interest detector.

The region of interest detector 120 may detect regions of interest for each object included in the image, as illustrated in FIG. 6. For example, when there are people in the input image, regions of interest regarding the first object (person) and the second object (person) can be detected, and when there are people and objects in the input image, the first object Regions of interest with respect to the (person) and the second object (object) can be detected.

7 is an enlarged block diagram of a region of interest connection in the embodiment of FIG. 2.

The region of interest connection unit 140 includes a connection score calculation unit 142 and a feature information estimation unit 144. For example, the connection score calculating unit 142 may calculate a similarity function that inputs feature information of the regions of interest detected in each of the adjacent frame images and an overlap ratio of the regions of interest detected in each of the adjacent frame images. A connection score may be calculated by considering at least one of a cross ratio function as an output and similarity of class information of the regions of interest. A detailed method of calculating the connection score using the equation 1 by the connection score calculation unit 142 of the present invention is the same as described above, and thus is omitted.

The feature information estimator 144 may determine that the region of interest is based on feature information of regions of interest detected from frame images adjacent to the frame image in which the region of interest is not detected when a frame image in which the region of interest is not detected exists. The feature information of the region of interest in the undetected frame image is estimated. A specific method of the feature information estimator 144 of the present invention using the interpolation method to estimate the location of the region of interest in the frame image in which the region of interest is not detected, and to connect the regions of interest using the estimated location of the region of interest As described above.

8 is a reference diagram illustrating a process in which a region of interest connection unit connects regions of interest.

For example, the region of interest connection unit 140 may include regions of interest 331, 333, and 335 for the first object detected from the plurality of frame images 302, 304, 306, and 308 generated by dividing the input image. , 337, 339, 341, 343) and the regions of interest for the second object (332, 334, 336, 338, 340, 342, 344) by connecting each of the first action tube 361 for the first object A second action tube 362 for the second object may be generated.

Also, the region of interest connection unit 140 may pre-process the generated first action tube 361 and the second action tube 362 for the second object to generate action streams for the first object and the second object, respectively. Yes, as described above.

9 is an exemplary view illustrating a process in which the feature information estimator estimates feature information of a region of interest.

The feature information estimator 144 may determine that the region of interest is based on feature information of regions of interest detected from frame images adjacent to the frame image in which the region of interest is not detected when a frame image in which the region of interest is not detected exists. The feature information of the region of interest in the undetected frame image is estimated. Since the feature information estimator 144 estimates the feature information in the frame image in which the region of interest is not detected using the interpolation process, it is omitted.

10 is an enlarged block diagram of a location information calculator in the embodiment of FIG. 2.

The location information calculating unit 160 includes an area division unit 162 and a rearrangement unit 164. For example, the location information calculating unit 160 rearranges the region of interest detected from the frame images in units of frame images, and positions the action streams from each other from a combined stream generated by connecting the rearranged regions of interest. Calculate location information about. The detailed process of calculating the positional relationship between the location information calculating unit 160 and the generated action stream for each object is the same as described above, and thus is omitted.

11 shows an image recognition process performed by the image recognition device of the present invention.

The image recognition apparatus 10 of the present invention generates action streams 272 and 274 for each object from an input image including at least one dynamic object, and simultaneously combines action streams generated for each object to generate a combination stream 279. To create. The combination stream of the present invention indicates the positional relationship of the generated action streams 272 and 274 for each object, as described above.

The image recognition device 10 inputs the generated action streams 272 and 274 for each object and the combination stream (pairwise stream 279) to the first recognizer, and sums the outputted class vectors in a preset manner. , Using the summed class vectors, the behavior of objects in the image can be recognized.

12 is an enlarged block diagram of a recognition unit in the embodiment of FIG. 1.

The recognition unit 200 includes a summation unit 220 and a comparison unit 240. For example, the recognizer 200 uses the generated action stream or location information of the action stream as an input, and uses a first recognizer that outputs at least one class vector as an index for classifying the action of the object. The behavior of the objects can be recognized.

13 is a flowchart of an image recognition method according to an embodiment of the present invention.

The image recognition method 10 performed by the image recognition apparatus 10 includes the following steps performed in time series.

In S100, the stream generating unit 100 generates an action stream including motion information on the action for each object from an input image including at least one object. When a plurality of objects are included in the input image, the stream generator 100 may generate an action stream for each object. In addition, the stream generator 100 generates a combination stream including position information indicating the positional relationship of the action stream generated for each object, and recognizes the action of the object in the image using the positional relationship of the action stream generated for each object. Can be as described above.

In S200, the recognizer 200 uses the generated action stream or the position information of the action stream as an input, and uses a first recognizer to output at least one class vector as an index for classifying the action of the object. The behavior of the objects can be recognized. For example, the recognition unit 200 recognizes the behavior of objects in the image by summing the outputs of the first recognizer, each of which has an input of an action stream generated for each object and a combination stream representing the positional relationship of the action stream generated for each object. The specific method is as described above.

14 is an enlarged flow chart of steps generated in the embodiment of FIG. 13.

In S120, the region-of-interest detection unit 120 divides the input image according to the passage of time to generate a plurality of frame images, and is divided by the object in the generated frame images to include a region of interest including at least a part of the object. To detect. The detailed method of detecting the region of interest from the plurality of frame images by the region of interest detector 120 is omitted as described above.

In S140, the region of interest connection unit 140 calculates the connection points between the regions of interest detected in different frame images, and connects the regions of interest detected in the different frame images in consideration of the calculated connection points. . The region of interest connection unit 140 may connect the region of interest in consideration of connection points between one region of interest detected in an arbitrary frame image and all regions of interest detected in another frame image.

In S160, the location information calculating unit 160 rearranges the region of interest for each object detected in the frame images in units of frame images, and connects the rearranged regions of interest to determine the positional relationship between the action streams from the combined stream. The position information to be displayed can be calculated. The method of rearranging the regions of interest detected in the frame images by the location information calculating unit 160 is the same as described above, and thus is omitted.

15 is an enlarged flow chart of the detection step in the embodiment of FIG. 14.

In S122, the feature information calculating unit 122 may calculate feature information indicating coordinates in which the region of interest is located in the generated frame image. In the present invention, the feature information is the coordinates of the location of the region of interest to be detected in the frame image, and it can be provided as the center coordinates of the region of interest in the frame image.

In S126, the boundary cell removing unit 126 duplicates the object in consideration of whether a probability that the object exists in the boundary cell among the boundary cells including the object in the frame images is greater than or equal to a preset threshold. A portion of the containing boundary cell is removed. The region-of-interest detection unit 120 removes the boundary-cell removal unit 126 and detects regions of interest using the remaining boundary cells.

16 is an enlarged flow chart of steps of connecting in the embodiment of FIG. 14.

In S142, the connection score calculating unit 142 outputs a similarity function using the feature information of the regions of interest detected in each of the adjacent frame images as input, and an overlap ratio of the regions of interest detected in each of the adjacent frame images. The connection score is calculated by considering at least one of the cross ratio function and the similarity of the class information of the regions of interest. A detailed method of calculating the connection scores of the regions of interest detected in each of the adjacent frame images is omitted because the connection score calculation unit 142 is described above.

In S144, the feature government estimator 144 detects a frame image adjacent to the frame image in which the region of interest is not detected, if a frame image in which the region of interest is not detected among frame images generated by dividing the input image. The location of the region of interest in the frame image in which the region of interest is not detected may be estimated using the coordinates of the regions of interest.

17 is an enlarged flow chart of steps for calculating location information in the embodiment of FIG. 14.

In S162, the region dividing unit 162 discriminates the region of interest detected for each object in the frame images and a portion in which the region of interest is not detected. For example, the region dividing unit 162 may mask a region of interest and a region other than the region of interest using two pixel value information in order to obtain location information of the regions of interest detected in the frame image.

In S164, the rearrangement unit 164 rearranges the divided regions of interest in a predetermined combination method. For example, the rearrangement unit 164 may combine the masked regions of interest 277 for the first object and the masked regions of interest 279 for the second object in a predetermined combination method. The method of combining the region of interest masked by the array unit 164 is a method of alternately combining the region of interest with respect to the first object and the region of interest with respect to the second object, pre-positioning regions of interest with respect to the first object, and the second It may include a method of rearranging regions of interest related to an object.

The image recognition apparatus 10 may recognize regions of interest detected by each object, and re-arrange the regions of interest to generate a combination stream, thereby recognizing the behavior of objects using a location relationship of objects in the image.

FIG. 18 is an enlarged flow chart of steps recognized in the embodiment of FIG. 13.

In S220, the summing unit 220 adds a class vector of each action stream classified for each object using a first recognizer for each action list according to a preset method. For example, the summing unit 220 may add and apply weights to class vectors, which are outputs of the first recognizer, using an input stream of an action stream for each object and a combination stream indicating a positional relationship between action streams generated for each object. .

In S240, the comparator 240, the comparator 240, by the action list of the object stored in the correct answer data set (UCF-101 Detection Data Set) provided in advance the probability information for each action list to classify the behavior in the summed class vector Compare with probability information.

All or part of the method of an embodiment of the present invention described above may be embodied in the form of a computer-executable recording medium (or computer program product), such as a program module executed by a computer. Here, the computer-readable medium may include a computer storage medium (eg, a memory, a hard disk, a magnetic/optical medium, or a solid-state drive (SSD)). Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media.

In addition, all or part of a method according to an embodiment of the present invention includes instructions executable by a computer, and the computer program includes programmable machine instructions processed by a processor, and is a high-level programming language. Language), Object-oriented Programming Language, assembly language or machine language.

Means or modules in this specification may mean hardware capable of performing functions and operations according to each name described in the specification, and computer program code capable of performing specific functions and operations It may mean, or an electronic recording medium on which computer program code capable of performing a specific function and operation is mounted, for example, a processor or a microprocessor. In other words, a means or module may mean a functional and/or structural combination of hardware for performing the technical idea of the present invention and/or software for driving the hardware.

Accordingly, a method according to an embodiment of the present invention may be implemented by executing a computer program as described above by a computing device. The computing device may include at least some of a processor, a memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to the low-speed bus and the storage device. Each of these components is connected to each other using various buses, and can be mounted on a common motherboard or mounted in other suitable ways.

The above description is merely illustrative of the technical spirit of the present invention, and those of ordinary skill in the art to which the present invention pertains can make various modifications, changes, and substitutions without departing from the essential characteristics of the present invention. will be. Therefore, the embodiments and the accompanying drawings disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments and the accompanying drawings. . The scope of protection of the present invention should be interpreted by the claims below, and all technical spirits within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

Claims (20)

Regarding the behavior of the first object as the acting subject in adjacent different frame images including the first object as the acting subject and at least one second object located around the other acting subject A stream generating unit that generates a first action stream including motion information, a second action stream according to changes in the second object, and combines the first action stream and the second action stream to generate a combination stream. ; And
A first recognizer receiving the first action stream, a second action stream, and a combination stream, and outputting each class vector as an index for classifying the action of an object that is the subject of the action; and output from the first recognizer It includes; a summation unit for adding the class vectors according to a preset method; including, a recognition unit for analyzing and recognizing the behavior of the objects using the summed class vector;
The combination stream is based on location information on mutual positions of regions of interest detected for each object of the first action stream and the second action stream generated for each of the first object and the second object,
The class vector includes a behavior list to classify the behavior of the first object and probability information indicating a probability that the behavior of the first object and the second objects in the input image corresponds to the behavior list, and the recognition unit sums the sum And recognizing the behavior of the objects in consideration of the association between the first object and the second object according to the probability information for each action list of a class vector. The method of claim 1, wherein the stream generating unit
The region of interest including a plurality of frame images by dividing the input image over time, and including at least a portion of the first object and the second object for each of the first and second objects in the generated frame images A region-of-interest detection unit that detects; Further comprising,
And generating the action stream using the detected region of interest. delete The method of claim 2, wherein the stream generator
A region of interest connection unit calculating a connection score between the regions of interest detected in the adjacent frame images, and connecting the regions of interest detected in the different frame images in consideration of the calculated connection points; Further comprising,
And generating the action stream for each of the first and second objects using the connected regions of interest. The method of claim 4, wherein the stream generating unit
Position information calculation for rearranging the regions of interest detected in the frame images in units of frame images, and calculating location information regarding mutual positions of the action streams from the combined stream generated by connecting the rearranged regions of interest part; Video recognition device further comprising a. The method of claim 4, wherein the region of interest connection
Similarity function that inputs feature information of the regions of interest detected in each of the adjacent frame images as input, cross ratio function that outputs the overlap ratio of the regions of interest detected in each of the adjacent frame images as output, and class information of the regions of interest A connection score calculation unit for calculating a connection score in consideration of at least one of the similarities of; Further comprising,
The apparatus for recognizing an image, wherein the region of interest is connected using the calculated connection score. The method of claim 6, wherein the region of interest connection
When a frame image in which the region of interest is not detected exists,
A feature information estimator estimating feature information of a region of interest in the frame image in which the region of interest is not detected based on feature information of regions of interest detected in frame images adjacent to the frame image in which the region of interest is not detected; Further comprising,
The apparatus for recognizing an image, wherein the region of interest is connected using the estimated feature information. The method of claim 2, wherein the region of interest detection unit
A feature information calculator that calculates feature information indicating coordinates of the region of interest in the generated frame image; Further comprising,
The apparatus for recognizing an image, wherein the region of interest is detected using the calculated feature information. The method of claim 5, wherein the location information calculation unit
A region dividing unit for classifying the region of interest and the portion of the region of interest not detected in the frame images for each of the first and second objects; And
A rearrangement unit for rearranging the divided regions of interest in a predetermined combination method; Further comprising,
The apparatus for recognizing an image, wherein the location information is calculated based on a combination stream generated by connecting the rearranged regions of interest. The method of claim 8, wherein the feature information calculation unit
A pre-processor for dividing each of the generated frame images into a grid cell of a predetermined size; And
The generated frame images are input, and a center coordinate of a boundary cell indicating a probability that the first or second object exists with a center in the grid cell or a probability that the object exists in the boundary cell is output. A calculation unit that calculates a center coordinate of each boundary cell and a probability that the object exists in the boundary cell using a second network (Neural Network); Further comprising,
The image recognition device, characterized in that for calculating the feature information using a boundary cell in which the probability and the center coordinates are calculated. The method of claim 10, wherein the region of interest detector
A boundary cell including the first object overlapping in consideration of whether a probability that the first object exists in the boundary cell among the boundary cells overlapping the first object in the frame images is equal to or greater than a preset threshold. A boundary cell removal unit to remove a portion of the; Further comprising,
An image recognition apparatus characterized by detecting the region of interest using the removed and remaining boundary cells. delete The first object that acts as the acting object in adjacent frame images including the first object acting as the stream generating unit and at least one second object positioned around another acting subject. A first action stream including motion information related to an action is generated, and a second action stream according to the change of the second object is generated, and a combination stream is generated by combining the first action stream and the second action stream. step; And
Analyzing and recognizing the behavior of objects using the class vector added by the recognition unit; It includes,
The step of analyzing and recognizing the behavior of objects using the summed class vector is:
The first recognizer receives the first action stream, the second action stream, and the combination stream, and outputs each class vector as an index for classifying the action of the object acting as the subject. 1, summing the class vectors output from the recognizer according to a preset method.
The combination stream is based on location information on mutual positions of regions of interest detected for each object of the first action stream and the second action stream generated for each of the first object and the second object,
The class vector includes a behavior list to classify the behavior of the first object and probability information indicating a probability that the behavior of the first object and the second objects in the input image corresponds to the behavior list, and the recognition unit sums the sum And recognizing the behavior of the objects in consideration of the association between the first object and the second object according to the probability information for each action list of a class vector. The method of claim 13, wherein the generating step
The region of interest including a plurality of frame images by dividing the input image over time, and including at least a portion of the first object and the second object for each of the first and second objects in the generated frame images Detecting; Further comprising,
And generating the action stream using the detected region of interest. The method of claim 14, wherein the step of generating
Calculating a connection score between the regions of interest detected in adjacent different frame images, and connecting the regions of interest detected in the different frame images in consideration of the calculated connection points; Further comprising,
And generating the action stream for each of the first and second objects using the connected regions of interest. The method of claim 15, wherein the step of generating
Rearranging the regions of interest detected in the frame images in units of frame images, and calculating location information regarding mutual positions of the action streams from a combination stream generated by connecting the rearranged regions of interest; The image recognition method further comprising a. The method of claim 15, wherein the connecting step
Similarity function that inputs feature information of the regions of interest detected in each of the adjacent frame images as input, cross ratio function that outputs the overlap ratio of the regions of interest detected in each of the adjacent frame images as output, and class information of the regions of interest Calculating a connection score in consideration of at least one of the similarities; Further comprising,
A method of recognizing an image, wherein the region of interest is connected using the calculated connection score. The method of claim 17, wherein the connecting step
When a frame image in which the region of interest is not detected exists,
Estimating feature information of a region of interest in a frame image in which the region of interest is not detected based on feature information of regions of interest detected in frame images adjacent to the frame image in which the region of interest is not detected; Further comprising,
A method of recognizing an image, wherein the region of interest is connected using the estimated feature information. The method of claim 16, wherein the step of calculating the location information
Distinguishing the region of interest and the portion of the region of interest not detected from the frame images for each of the first object and the second object; And
Rearranging the divided regions of interest in a predetermined combination method; Further comprising,
The method of recognizing an image, wherein the location information is calculated based on a combination stream generated by connecting the rearranged regions of interest. A program stored in a computer-readable recording medium that realizes the image recognition method according to any one of claims 13 to 19 through being executed by a processor.

Images