KR20190120489A - Apparatus for Video Recognition and Method thereof - Google Patents

Abstract

According to the present invention, disclosed is an image recognition apparatus, which can recognize behaviors of objects in an image using a convolutional neural network capable of performing video analysis. The image recognition apparatus comprises: a stream generation unit generating an action stream including motion information on behaviors of each object in an input image including one or more objects; and a recognition unit receiving position information indicating a positional relationship of an action stream in the generated action stream or the input image and using a first recognizer outputting one or more class vectors as an indicator for classifying behaviors of the objects to recognize the behaviors of the objects.

Description

Apparatus for Video Recognition and Method

The present invention relates to an image recognition device. More particularly, the present invention relates to an image recognition apparatus for recognizing the behavior of an object in an image using a neural network.

Behavioral awareness can be used to classify human behavior in video images that contain both temporal and spatial information. The subject in the image and its surroundings, the object that is the subject and the subject's action, and the relationship between the object and another object can be used as an element to classify the behavior of a specific object in the image.

Artificial Neural Network (Neural Network) is a statistical learning algorithm derived from the neural network of biology in machine learning and cognitive science. Recently, techniques for mechanically recognizing images in the field of computer vision have been actively studied. A convolutional neural network (CNN) is a neural network composed of one or several convolutional layers and artificial neural network layers mounted thereon. The convolutional neural network (CNN) is an artificial neural network mainly used for image and audio analysis, which is two-dimensional input data.

Conventional image recognition technology using an artificial neural network has been mainly made only in still images, and also there is a limit to analyze the relationship between the subject and the object in the still image. In addition, the conventional image recognition technology has a problem that does not consider the passage of time in analyzing the relationship between objects, including people and objects.

Therefore, based on an artificial neural network capable of performing image analysis on a moving image basis, development of a technology capable of recognizing an image in consideration of an object in the image and the contextual interrelationship of the objects is required.

Korean Unexamined Patent No. 10-2017-0034226 (Notice)

The present invention has been made to solve the above problems, and discloses an image recognition apparatus. In particular, an image recognition apparatus capable of recognizing the behavior of an object in a video using a neural network is disclosed.

The present invention has been made to achieve the above object, the image recognition apparatus of the present invention is a stream generation unit for generating an action stream including the motion information about the action for each object in the input image including at least one object ; And receiving the position information indicating the positional relationship of the action stream from the generated action stream or the input image, and outputting at least one class vector as an indicator for classifying the object's behavior. Recognition unit for recognizing their behavior; It includes.

In the present invention, the stream generation unit generates a plurality of frame images by dividing the input image over time, and detects an ROI including at least a portion of the object for each object in the generated frame images. Detection unit; The apparatus may further include: generating the action stream using the detected ROI.

In the present invention, the recognizer comprises: a summing unit for summing a class vector of an action stream generated for each object by a predetermined method using the first recognizer; Further comprising, it is possible to recognize the behavior of the objects using the summed class vector.

In the present invention, the stream generation unit calculates a connection score between the ROIs detected from adjacent different frame images, and connects the ROIs detected from the different frame images in consideration of the calculated connection points. Area connections; The apparatus may further include: generating the action stream for each object by using the connected ROIs.

In the present invention, the stream generation unit rearranges the ROIs detected in the frame images in frame image units, and position information about mutual positions of the action streams from the combined stream generated by concatenating the rearranged ROIs. A location information calculation unit for calculating a; It may further include.

In the present invention, the ROI connection unit outputs a similarity function for inputting feature information of the ROIs detected in adjacent frame images, and an intersection ratio for outputting an overlap ratio of the ROIs detected in the adjacent frame images. A connection score calculator configured to calculate a connection score in consideration of at least one of a similarity between a function and class information of the ROIs; In addition, the ROI may be connected using the calculated connection score.

In the present invention, when there is a frame image in which the ROI is not detected, the ROI connection unit based on the feature information of the ROIs detected in the frame images adjacent to the frame image in which the ROI is not detected. A feature information estimator for estimating feature information of the ROI in the undetected frame image; The apparatus may further include: connecting the ROI by using the estimated feature information.

The ROI detector of the present invention may include a feature information calculator configured to calculate feature information indicating coordinates at which the ROI is located in the generated frame image; In addition, the ROI may be detected by using the calculated feature information.

The location information calculator may include: a region separator configured to classify and distinguish portions of the ROI detected for each object from the frame images and portions from which the ROI is not detected; And a rearrangement unit for rearranging the divided regions of interest by a predetermined combination method. Further, the location information may be calculated based on the combined stream generated by concatenating the rearranged ROIs.

In the present invention, the feature information calculation unit comprises a pre-processing unit for generating a grid cell of a predetermined size by dividing the generated frame images, respectively; And a second recognizer configured to receive the generated frame images as inputs and output a center coordinate of a boundary cell indicating a probability that the object exists in a center of the grid cell or a probability that the object exists in the boundary cell. A calculation unit configured to calculate a center coordinate of each boundary cell and a probability that the object exists in the boundary cell using a neural network; Further, the feature information may be calculated using a boundary cell from which the probability and the center coordinates are calculated.

In the present invention, the ROI detector includes the object in consideration of whether a probability of the object exists in the boundary cell among the boundary cells including the object in the frame images is greater than or equal to a preset threshold. A boundary cell removal unit for removing a part of the boundary cell; In addition, the ROI may be detected by using the boundary cell remaining after the removal.

In the present invention, the class vector includes a behavior list classifying the behavior of the object and probability information indicating a probability that the behavior of the objects in the input image corresponds to the behavior list, and the recognition unit is configured to determine the sum of the summed class vector. The behavior of the objects may be recognized using the probability information for each behavior list.

In addition, in order to achieve the above object, the image recognition method of the present invention comprises the steps of: generating an action stream including motion information about the action for each object in an input image including at least one object; And a first recognizer that receives position information indicating a positional relationship of the action stream in the generated action stream or the input image, and outputs at least one class vector as an index for classifying the behavior of the object. Recognizing the behavior of the objects; It includes.

The generating may include generating a plurality of frame images by dividing the input image over time, and detecting an ROI including at least a portion of the object for each object in the generated frame images. ; The apparatus may further include: generating the action stream using the detected ROI.

In the present invention, the recognizing may include adding a class vector of an action stream generated for each object according to a preset method using the first recognizer; Further comprising, it is possible to recognize the behavior of the objects using the summed class vector.

The generating may include calculating a connection score between the ROIs detected in adjacent frame images, and connecting the ROIs detected in the different frame images in consideration of the calculated connection scores. step; The apparatus may further include: generating the action stream for each object by using the connected ROIs.

In the present invention, the generating may include rearranging the ROIs detected in the frame images in a frame image unit and linking the rearranged ROIs with respect to mutual positions of the action streams. Calculating information; It may further include.

In the present invention, the linking may include a similarity function as input of feature information of the ROIs detected in each of the adjacent frame images, and an intersection ratio as an output of an overlap ratio of the ROIs detected in the adjacent frame images. Calculating a connection score by considering at least one of a similarity between a function and class information of the ROIs; In addition, the ROI may be connected using the calculated connection score.

In the present invention, the step of connecting the ROI based on the feature information of the ROIs detected in the frame images adjacent to the frame image in which the ROI is not detected when there is a frame image in which the ROI is not detected. Estimating feature information of the ROI in the undetected frame image; The apparatus may further include: connecting the ROI by using the estimated feature information.

The detecting may include: calculating feature information indicating coordinates at which the ROI is located in the generated frame image; In addition, the ROI may be detected by using the calculated feature information.

The calculating of the position information may include: distinguishing the region of interest detected by each object from the frame images and the portion of which the region of interest is not detected; Rearranging the separated regions of interest by a predetermined combination method; Further, the location information may be calculated based on the combined stream generated by concatenating the rearranged ROIs.

The present invention also discloses a computer program stored in a computer-readable recording medium for executing the image recognition method on a computer.

According to the present invention, the behavior of the object in the image can be recognized.

In particular, an image may be recognized using a composite product neural network capable of analyzing an image in a video unit.

1 is a block diagram of an image recognition apparatus according to an exemplary embodiment.
FIG. 2 is an enlarged block diagram of a stream generator in the embodiment of FIG. 1.
3 is an enlarged block diagram of the ROI detector of the embodiment of FIG. 2.
4 is an enlarged block diagram of a feature information calculating unit in the embodiment of FIG. 3.
5 is a reference diagram illustrating a process of detecting an ROI by the ROI detector.
6 is an exemplary diagram illustrating the ROIs detected by the ROI detector;
FIG. 7 is an enlarged block diagram of a region of interest connector in the embodiment of FIG. 2.
8 is a reference diagram illustrating a process of connecting the ROI by the ROI.
9 is an exemplary diagram illustrating a process of estimating feature information of a region of interest by a feature information estimator.
FIG. 10 is an enlarged block diagram of a location information calculator in the embodiment of FIG. 2.
11 illustrates an image recognition process performed by the image recognition apparatus of the present invention.
12 is an enlarged block diagram of a recognizer in the embodiment of FIG. 1.
13 is a flowchart of an image recognition method according to an exemplary embodiment.
FIG. 14 is an enlarged flowchart of steps generated in the embodiment of FIG. 13. FIG.
FIG. 15 is an enlarged flowchart of a detecting step in the embodiment of FIG. 14.
FIG. 16 is an enlarged flowchart of connecting steps in the embodiment of FIG. 14;
17 is an enlarged flowchart of calculating position information in the embodiment of FIG. 14.
18 is an enlarged flowchart of a step of recognizing in the embodiment of FIG. 13.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

In the description with reference to the accompanying drawings, the same or corresponding components will be given the same reference numerals and redundant description thereof will be omitted.

In describing the present invention, when it is determined that detailed descriptions of related well-known structures or functions may obscure the gist of the present invention, the detailed description may be omitted.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of terms. Singular expressions include plural expressions unless the context clearly indicates otherwise. Each step described below may be provided by one or several software modules or may be implemented by hardware that is responsible for each function, or may be a combination of software and hardware.

Specific meanings and examples of each term will be described below in the order of the drawings.

Hereinafter, the drawings related to the configuration of the image recognition device 10 according to an embodiment of the present invention will be described in detail.

1 is a block diagram of an image recognition device 10 according to an exemplary embodiment.

The image recognition apparatus 10 includes a stream generator 100 and a recognizer 200. For example, the image recognition apparatus 10 of the present invention receives an image including at least one object, generates an action stream from the input image, and uses the first recognizer to input the generated action stream. I can recognize the behavior of my objects.

The image recognized by the image recognition apparatus 10 of the present invention includes a video image (video) in which image information changes over time as well as a still image. Unlike conventional image recognition technology, the image recognition apparatus 10 of the present invention may recognize an image by performing image analysis on a video unit basis. More specifically, the image recognition apparatus 10 of the present invention may recognize an image by classifying the behavior of an object in the image.

The image recognition device 10 of the present invention may be used in an autonomous driving device such as an autonomous driving robot to control autonomous driving, and may be used in a monitoring device such as CCTV to be used for automatic analysis of an input image. In addition, the image recognition apparatus 10 of the present invention may be used as a control device of a home Internet of Things (IOT) and may be used to control the operation of a home appliance by automatically recognizing a human action.

In the present invention, the first recognizer and the second recognizer used by the image recognition apparatus 10 may be provided as a neural network that is supervised learning based on a deep learning algorithm having a deep neural network structure having multiple layers. Can be. Preferably, the first recognizer and the second recognizer used by the image recognition apparatus 10 of the present invention are a neural network structure that is learned based on supervised learning, and a composite product neural network including at least one convolutional layer and a pooling layer ( Convolutional Neural Network). The artificial neural network designed based on the back propagation algorithm has been delayed for some time because the problems such as delay of learning time due to excessive depth are revealed when the neural network becomes large. However, the overfit problem is solved through the method of dropout. As a result, the performance of the algorithm has improved dramatically.

Since the convolutional neural network, a representative deep learning structure used by the image recognition apparatus 10, is designed to mimic a human visual processing process, it is evaluated as a deep learning algorithm suitable for image and image processing, and abstracts an image. By extracting a feature that can be represented by a high performance in the field of image recognition.

In detail, the convolutional neural network may hierarchically extract features from an input image and form a feature map using the extracted features. The features distributed on the feature map include relocated image information, through which the composite product neural network can effectively classify the images. The multiplicative neural network can resize the feature values extracted from the convolutional layer by applying an activation function to the features extracted through the convolutional layer and repeatedly placing the pooling layer.

The multiplicative neural network used by the image recognition apparatus 10 according to the present invention basically has a multiplicative neural network structure, and a structure within the neural network may be provided differently according to the shape, size, and pixel value information of the input image. That is, the number of input layers, hidden layers, and output layers of the composite product neural network used by the image recognition apparatus 10, the number of nodes included in each layer, a weight applied to the edges of the nodes, and a learning rate. Etc. may be set differently according to the purpose of image recognition.

In addition, the composite product neural network used by the image recognition apparatus 10 may use a neural network to which a structure such as VGG-Net, GoogLeNet, and ResNet is applied.

In addition, the neural networks used by the image recognition apparatus 10 of the present invention have a structure in which a fully-connected layer is connected to a CNN structure including a convolutional layer and a pooling layer, and an overfit problem occurs. Otherwise, the deeper the layer, the higher the accuracy of image recognition. It demonstrates with reference to FIG.

The stream generator 100 includes an ROI detector 120, an ROI 140, and a location information calculator 160. For example, the stream generator 100 generates an action stream including motion information regarding the action of each object in an input image including at least one object. The action stream of the present invention may be generated by preprocessing an action tube generated for each object existing in an image into a neural network.

The action tube of the present invention may be generated for each object as a set of three-dimensional (X, Y, Z) image data including at least a part of the object in the input image. The action tube of the present invention may include motion information regarding a change in behavior of an object, including an amount of change in pixel values in regions of interest that change over time. The motion information of the present invention is a kind of change amount of pixel values in the ROI that change with time, and may be provided as a derivative value of the pixel value change amount for each pixel in unit time.

For example, the stream generator 100 may generate frame images having a predetermined time interval by dividing an input image, detect an ROI for each object from the generated frame images, and connect the detected ROIs. By doing so, an action tube can be generated, and the generated action tube can be preprocessed to generate an action stream.

Since the action stream generated by preprocessing the action tube generated by concatenating the regions of interest detected in the frame images having a predetermined time interval is generated by preprocessing the action tube including the amount of change in pixel value according to time. It can represent the behavior change of the object according to the flow.

The action stream generated for each object in the stream generator 100 of the present invention may have a size of 56 * 56 * 3 * 16. The action stream generated by the stream generator 100 is generated by concatenating regions of interest that are 56 * 56 in width and length, and includes pixel values (pixel values) of an RGB color system, and one action stream includes 16 images. It may include frames. The size of the action stream generated by the stream generator 100 may vary depending on the setting method of the first recognizer. It demonstrates with reference to FIG.

The ROI detector 120 includes a feature information calculator 122 and a boundary cell remover 126. For example, the ROI detector 120 generates a plurality of frame images by dividing an input image over time, and includes an ROI that includes at least a portion of the objects divided by the objects in the generated frame images. Can be detected. It demonstrates with reference to FIG.

The feature information calculator 122 includes a preprocessor 123 and a calculator 124. For example, the feature information calculator 122 may calculate feature information indicating coordinates at which the ROI is located in the generated frame image. In the present invention, the feature information may be provided as coordinates regarding a position of the ROI to be detected in the frame image, and may be provided as a center coordinate of the ROI in the frame image or a coordinate located at a boundary of the ROI. It demonstrates with reference to FIG.

The preprocessor 123 may generate the grid cells having a predetermined size as shown in FIG. 5 by dividing the generated frame images. For example, the preprocessor 123 may generate grid cells having a size of 9 * 6 or 7 * 7. The grid cells generated by the preprocessor 123 may include boundary cells depending on the grid cells.

The calculation unit 124 receives the generated frame images as inputs and outputs the center coordinates of a boundary cell indicating a probability that the object exists in the grid cell or the probability that the object exists in the boundary cell. The center coordinate of each boundary cell and a probability that the object exists in the boundary cell are calculated using a second network.

The second recognizer of the present invention is a neural network structure that is learned based on supervised learning, and may be provided as a convolutional neural network including at least one convolutional layer and a pooling layer. Preferably, the second recognizer used by the calculator 124 may be provided as Regions with Convolutional Neural Network (R-CNN) or Faster R-CNN.

The feature information calculator 122 may calculate feature information by using the center cells of each boundary cell and boundary cells having a probability that the object exists in the boundary cell. For example, the second recognizer used by the calculation unit 124 is connected to at least one convolutional layer for extracting a feature and one end of the convolutional layer so that the center coordinates of the boundary cell and the object in the boundary cell are connected. It may include at least one pulley connected layer for calculating the existence probability. In addition, the second recognizer used by the calculator 140 of the present invention may further include a pooling layer that is alternately arranged with the convolutional layer in addition to the convolutional layer and the pulley connected layer.

Referring to the process of calculating the feature information by the feature information calculation unit 122 according to an embodiment of the present invention as follows. The feature information calculator 122 generates a frame image by dividing an input image including at least one object, and generates grid cells having a predetermined interval from the frame image. The calculation unit 124 has a center coordinate inside the grid cell by using the second recognizer, and generates an arbitrary number of boundary cells 127, 128, 129, 132, and 133 that are dependent on the grid cell. The probability that there are objects included in the boundary cell can be calculated.

The boundary cell remover 126 includes a boundary cell that overlaps the object in consideration of whether a probability that the object exists in the boundary cell among the boundary cells including the object in the frame images is greater than or equal to a preset threshold. Remove part of the cell. For example, the boundary cell remover 126 may select an object from among boundary cells 127, 128, 129, 132, 133, and 134 which have different probability that the object exists in the boundary cell generated by the calculator 124. The boundary cells 132 and 133 having the highest probability of are detected as the ROI.

To this end, the boundary cell remover 126 may leave only the boundary cells 132 and 133 having a probability that an object exists more than a preset threshold among boundary cells including an arbitrary object existing in the frame image. (127, 128, 129, 134) can be removed. In addition, the boundary cell removal unit 126 uses only one boundary cell by using a Nom-maximal Suppression Non-Maximum Suppression (NMS) algorithm in order to remove a boundary cell having a probability that a lesion is present within a cell. I can leave it. It demonstrates with reference to FIG.

As described above, the ROI detector 120 may detect at least one object including a person or an object from the frame images generated by splitting the input image using the feature information calculated by the feature information calculator 122. The region of interest including the image may be detected. It demonstrates with reference to FIG.

The ROI 140 includes a connection score calculator 142 and a feature government estimator 144. For example, the ROI 140 may calculate a connection score between the ROIs detected from adjacent different frame images, and the ROI detected from the different frame images in consideration of the calculated connection score. Connect It demonstrates with reference to FIG.

For example, the ROI 140 may include ROIs 331, 333, 335 of the first object detected in the plurality of frame images 302, 304, 306, and 308 generated by splitting an input image. , 337, 339, 341, 343, and regions of interest 332, 334, 336, 338, 340, 342, and 344 for the second object, respectively, to connect the first action tube 361 to the first object. A second action tube 362 may be generated for the second object.

In addition, the ROI 140 may preprocess the generated first action tube 361 and the second action tube 362 for the second object to generate an action stream for the first object and the second object, respectively. have. In other words, as described above, the action streams of the present invention may include motion information about an action for each object over time. In the present invention, the region of interest connector 140 may connect the regions of interest generated in the frame images, generate action tubes by connecting the objects, and generate the action stream for each object by preprocessing the generated action tubes. .

The connection score calculator 142 outputs a similarity function as an input of feature information of the ROIs detected in the adjacent frame images, and an overlap ratio for outputting an overlap ratio of the ROIs detected in the adjacent frame images. The connection score is calculated by considering at least one of a similarity between a function and class information of the ROIs.

는 인접한 프레임 영상들에서 각각 검출된 관심 영역들의 행동 분류를 위한 클래스 정보의 유사도,

및

는 인접한 프레임 영상들의 인덱스 번호,

는 임의의 스칼라 값,

는

인덱스 번호의 프레임 영상에서 검출된 관심 영역의 특징 정보,

는

인덱스 번호의 프레임 영상에 검출된 관심 영역의 특징 정보를 나타낸다. 본 발명의 프레임 영상에서 검출된 관심 영역의 특징 정보는 관심 영역이 해당 프레임 영상에서 위치하는 좌표 정보를 포함함은 전술한 바와 같다. 상기 클래스 정보는 관심 영역 에서 검출된 객체의 행동을 분류하는 카테고리(목록)를 의미한다.Here, sim () is a similarity function having an output value between 0 and 1 by inputting feature information of the ROIs detected in each of the frame images, and ov () is over of the ROIs detected in adjacent frame images, respectively. An odds ratio function with the output

Is similarity of class information for behavior classification of the ROIs detected in adjacent frame images,

And

Is the index number of adjacent frame images,

Is any scalar value,

Is

Feature information of the ROI detected from the frame image of the index number,

Is

The feature information of the ROI detected in the frame image of the index number is shown. As described above, the feature information of the ROI detected from the frame image of the present invention includes coordinate information located in the ROI of the frame image. The class information refers to a category (list) that classifies the behavior of the detected object in the ROI.

In addition, in Equation 1, the ov () function is an intersection ratio function that outputs an IOU (Intersection Over Union) value, and inputs an intersection region and a union region of the ROIs detected in adjacent frame images, respectively. Specifically, the intersection ratio function may output a value obtained by dividing the intersection region by the union region.

The process of calculating the link score by the link score calculator 142 according to an embodiment of the present invention will be described below with reference to the image frames 302 and 304. The connection score calculator 142 may include the first ROI 331 of the first object of the ROIs 331 and 332 detected in the first image frame 302 and the first within the second image frame 304. The connection score of the region of interest 333 is calculated.

In addition, the connection score calculator 142 calculates a connection score between the first ROI 331 in the first image frame 302 and the second ROI 334 in the second image frame 304. That is, the link score calculator 142 of the present invention calculates a link score for all the links in which the ROI in one image frame can be connected to the ROI in another adjacent image frame. Can be. The region of interest connector 140 of the present invention may connect regions of interest in the image frame such that the connection score is the highest. It demonstrates with reference to FIG.

When there is a frame image in which the ROI is not detected, the feature information estimator 144 detects the ROIs 335 detected in the frame images 375 adjacent to the frame image 372 in which the ROI is not detected. Based on the feature information of 336, 337, 338, 341, 342, 343, and 344, feature information of the ROI in the frame image 372 in which the ROI is not detected is estimated.

For example, when the ROI detection unit 120 of the present invention divides an input image and detects ROIs from a frame image, the ROI may not be properly extracted due to problems such as image quality. . That is, as shown in FIG. 9, there may exist a frame image 372 in which the ROI is not detected among the frame images 375.

In this case, the feature information estimator 144 has, as a function input, feature information of the ROIs in the frame images 371, 373, 379 adjacent to the frame images 374, 376, 378 where the ROI is not detected. A linear function may be generated, and the generated linear function may be used to estimate characteristic information of the ROI in the frame images 374, 376, and 378 in which the ROI is not detected.

The feature information of the present invention means the coordinates of the ROI in the frame image, and the feature information estimator 144 can determine the frame images 371, which are adjacent to the frame images 374, 376, 378 for which the ROI is not detected. Based on the positions of the regions of interest in 373 and 379, the positions of the regions of interest detected in the frame images 274, 276, and 278 in which the regions of interest are not detected may be estimated.

That is, the feature information estimator 144 performs an interpolation process based on the feature information of adjacent frame images 371, 373, and 379 from which the ROI is detected, so that the ROI of the image of the ROI is not detected. The feature information which is the coordinate information of the area is estimated. The region of interest connector 140 may generate an estimated region of interest in the frame image in which the region of interest is not detected using the estimated feature information, and may connect the estimated regions of interest with regions of interest in the adjacent frame image.

The location information calculator 160 includes an area separator 162 and a rearrangement unit 164. For example, the location information calculator 160 rearranges the ROIs detected in the frame images in frame image units, and positions the action streams from the combined streams generated by concatenating the rearranged ROIs. Compute positional information about.

For example, the location information calculator 160 rearranges the ROIs detected for each object in the frame images by a predetermined combination method, generates a combination stream using the rearranged ROIs, and generates the generated combination stream. Using the to calculate the position information about the mutual position of the action stream generated for each object. In the present invention, the position information refers to a relationship of mutual positions of the ROIs detected for each object in the frame image, not the appearance information, and a detailed method of calculating the position information will be described later.

For example, the location information calculator 160 intersects the ROIs of the first object and the ROIs of the second object in an input image including the first object (person) and the second object (thing). By arranging or densely arranging, a combination stream may be generated, and the generated combination stream may be used to calculate position information regarding positions of the action streams related to the first object and the action streams related to the second object. The method for calculating the positional relationship of the action stream generated for each object by the positional information calculating unit 160 of the present invention is not limited to masking or binary processing, and other well-known techniques for inputting the positional relationship of the ROIs are described. Include. It demonstrates with reference to FIG.

The area divider 162 classifies and masks a portion of the ROI detected for each object and a portion of the frame images in which the ROI is not detected. A case in which the ROI detection unit 120 detects ROIs related to the first object and the second object from an input image including the first object (person) and the second object (thing) will be described as an example.

In this case, in the present exemplary embodiment, the area separator 162 may include a masking part (not shown) for classifying and masking a portion in which the ROI is not detected.

For example, the masking unit (not shown) may divide and mask a portion, in which the regions of interest of the first object are detected, and a portion which is not the region of interest in the first object, with two pixel value information in the frame images. . In addition, a portion in which the regions of interest of the second object are detected and a portion which is not the region of interest of the second object may be divided and masked by two pieces of pixel value information.

In the present invention, masking a portion in which the regions of interest are detected and portions in which the regions of interest are not detected by the region separator 162 may be provided by performing binary processing on the frame image. The location information calculator 160 may calculate location information indicating a positional relationship between the ROIs detected in the frame images, based on a masking process performed by the area divider 162.

The rearrangement unit 164 rearranges the divided ROIs in a predetermined combination method. For example, the rearrangement unit 164 may combine the masked ROIs 277 related to the first object and the masked ROIs 279 related to the second object in a predetermined combination method. The rearrangement unit 164 combines the divided ROIs in consideration of the frame-by-frame information so that the masked ROI of the first object-masked ROI of the second object-masked ROI of the first object. It includes a method of cross-combining the frame images of each object like the region.

In addition, the method in which the rearrangement unit 164 combines the divided ROIs may be performed by first arranging masked ROIs of the first object in consideration of first and second object-specific information. A method of post-arraying the masked regions of interest.

The location information calculation unit 160 generates a combination stream using the rearranged ROIs by a predetermined combination method, and calculates location information regarding mutual positions of action streams generated for each object using the generated combination stream. Can be. In the present invention, the combined stream generated by the location information calculator 160 includes a pair of streams in which masked regions of interest regarding the first object are connected and a pair of streams in which masked regions of interest regarding the second object are connected. Stream).

The pairwise stream of the present invention includes location information indicating the positional relationship of the action stream for each object except the appearance information included in the action stream for each object in the image. That is, the object position information calculator 160 may calculate position information indicating the positional relationship between the action stream regarding the first object and the action stream regarding the second object by using the combination stream described above. The image recognition apparatus 10 may recognize the behavior of an object based on the positional relationship of each object in the image.

The recognition unit 200 includes an adder 220 and a comparator 240. For example, the recognizer 200 uses the generated action stream or position information of the action stream as an input and uses a first recognizer that outputs at least one class vector as an indicator for classifying the behavior of the object. The behavior of the objects can be recognized. The class vector used by the recognition unit 200 of the present invention includes a behavior list classifying the behavior of the object and probability information indicating a probability that the behavior of the objects in the input image corresponds to the behavior list. The recognition unit 200 may recognize the actions of the objects by using the probability information for each action list of the summed class vector.

The adder 220 adds the class vectors of the action streams classified for each object according to a preset method using a first recognizer. For example, the adder 220 acts on an output value of a first recognizer that uses a combination stream including positional relationships between an action stream generated for each object and an action stream generated for each object in the stream generator 100. Each list can be added according to a preset method.

For example, the adder 220 may classify the behavior of the first object (person) that is output when the action stream regarding the first object (person) is input to the first recognizer. The second class vector, which is an indicator for classifying the behavior of the second object (object) output when the action stream regarding the second object (object) is input to the recognizer, and the location of the action stream generated for each person or object in the first recognizer The combined stream including the relationship may be summed by applying different weights to the third class vector output at the time of input, or may be summed by applying the same weight as 1/3. The summing unit 220 adds equal weights 1/3 to the first class vector, the second class vector, and the third class vector to obtain an average of the first class vector, the second class vector, and the third class vector. Can be prepared by the process of seeking.

The comparison unit 240 compares the probability information for each action list classifying the behaviors in the summed class vector with the probability information for each action list of the object stored in the preliminary answer data set (UCF-101 Detection Data Set). For example, the comparison unit 240 compares whether or not the behaviors of the objects represented by the class vector summed by the adder 220 match using a data set in which probability information for each action list of known objects is stored. I can accurately recognize the behavior of my objects.

FIG. 2 is an enlarged block diagram of a stream generator in the embodiment of FIG. 1.

The stream generator 100 includes an ROI detector 120, an ROI 140, and a location information calculator 160. For example, the stream generator 100 generates an action stream including motion information regarding the behavior of the object included in the input image for each object. The action stream is generated by concatenating the regions of interest detected in each image frame over time.

The motion information about the behavior of the object may include both the position change amount of the ROIs and the pixel value change amount of the ROIs over time, but includes only the pixel value change amount of the ROI excluding the position change of the ROIs in the frame image can do.

For example, since the position and pixel value information of the ROIs located in the image frame vary with time, the position change amount of the ROIs that change in the image frame with time, and the ROI change with time. The amount of change in the pixel value and the like in itself can constitute motion information regarding the behavior change of the object. However, as described above, preferably, the motion information of the present invention may include only the pixel value change amount in the ROI that changes with time.

However, since the image recognition apparatus 10 of the present invention uses the pair-wise stream, it will be described later that the behavior of the objects in the image can be recognized in consideration of the positional variation of the regions of interest in the frame image.

The ROI detector 120 generates a plurality of frame images by dividing an input image over time, and detects an ROI including at least a portion of the objects divided by the objects from the generated frame images. have. The ROI detector 120 may divide the input images into 16 frame units, and detect ROIs mainly including an image of an object from each of the divided frame images.

The region of interest connector 140 generates action tubes for each object by connecting the regions of interest detected in the divided frame images with time. As described above, the ROI 140 may generate the action stream by preprocessing the generated action tubes according to the input form of the first recognizer.

The location information calculator 160 rearranges the ROIs detected from the frame images in frame image units, and connects the rearranged ROIs from the combined stream generated. Compute positional information about mutual positions of streams. The image recognition apparatus 10 of the present invention may recognize the behavior of the object in the image using the positional relationship of the action stream generated for each object in the image by using the position information calculated by the position information calculator 160.

3 is an enlarged block diagram of the ROI detector of the embodiment of FIG. 2.

The ROI detector 120 includes a feature information calculator 122 and a boundary cell remover 126. The feature information calculator 122 calculates feature information as coordinates of a location of the ROI to be detected in the frame image. The feature information calculator 122 may use a vision recognition algorithm including YOLO based on a multiplicative neural network, Fast R-CNN, etc. to detect a region of interest in an image.

The boundary cell remover 126 includes a boundary cell that overlaps the object in consideration of whether a probability that the object exists in the boundary cell among the boundary cells including the object in the frame images is greater than or equal to a preset threshold. Remove part of the cell. As described above, the boundary cell removal unit 126 removes the boundary cell and detects the ROI using the remaining boundary cell.

4 is an enlarged block diagram of a feature information calculating unit in the embodiment of FIG. 3.

The feature information calculator 122 includes a preprocessor 123 and a calculator 124. The preprocessing unit 123 may generate the grid cells having a predetermined size by dividing the generated frame images, and include the boundary cells that are dependent on the grid cells.

The calculation unit 124 receives the generated frame images as inputs and outputs the center coordinates of a boundary cell indicating a probability that the object exists in the grid cell or the probability that the object exists in the boundary cell. The center coordinate of each boundary cell and a probability that the object exists in the boundary cell are calculated using a second network.

The second recognizer of the present invention may include at least one convolutional layer extracting the convolutional feature and a pulley connected layer connected to one end of the convolutional layer to calculate the center coordinates of the boundary cell and the probability that an object in the boundary cell exists. have.

5 is a reference diagram illustrating a process of detecting an ROI by the ROI detector.

Boundary cells 127, 128, 129, 132, 133, and 134, which depend on the grid cells generated by the preprocessor 123, depend on the grid cells to indicate the probability that an object part is included therein. As described above, the ROI detector 120 may remove all boundary cells whose probability that the object exists among the generated boundary cells does not exceed a predetermined threshold value, and detect the ROI using the remaining boundary cells. same.

As illustrated in FIG. 5, the ROI detector 120 may consider a ROI of a first object (person) having a 99% probability and a second object having a 67% probability based on a probability indicated by boundary cells. The ROI may be detected. The ROI detection unit 120 includes probability information of the ROIs used to detect the ROI as described above.

6 is an exemplary diagram illustrating the ROIs detected by the ROI detector;

As shown in FIG. 6, the ROI detector 120 may detect ROIs for each object included in the image. For example, if there are people in the input image, the ROIs may be detected with respect to the first object (person) and the second object (person). If there are people and objects in the input image, the first object may be detected. The regions of interest relating to the (person) and the second object (thing) can be detected.

FIG. 7 is an enlarged block diagram of a region of interest connector in the embodiment of FIG. 2.

The ROI 140 includes a connection score calculator 142 and a feature information estimator 144. For example, the connection score calculator 142 may calculate a similarity function using the feature information of the ROIs detected in the adjacent frame images, and the overlap ratio of the ROIs detected in the adjacent frame images. The connection score may be calculated in consideration of at least one of a similarity between the cross ratio function as an output and the class information of the ROIs. Since the connection point calculation unit 142 of the present invention calculates the connection point using Equation 1 as described above, it will be omitted.

When there is a frame image in which the ROI is not detected, the feature information estimator 144 may determine that the ROI is based on the feature information of the ROIs detected in the frame images adjacent to the frame image in which the ROI is not detected. The feature information of the ROI in the undetected frame image is estimated. According to a specific method of the present invention, the feature information estimator 144 estimates the location of the ROI in the frame image in which the ROI is not detected by using an interpolation method, and connects the ROIs using the estimated location of the ROI. As described above.

8 is a reference diagram illustrating a process of connecting the ROI by the ROI.

For example, the ROI 140 may include ROIs 331, 333, 335 of the first object detected in the plurality of frame images 302, 304, 306, and 308 generated by splitting an input image. , 337, 339, 341, 343, and regions of interest 332, 334, 336, 338, 340, 342, and 344 for the second object, respectively, to connect the first action tube 361 to the first object. A second action tube 362 may be generated for the second object.

In addition, the ROI 140 may preprocess the generated first action tube 361 and the second action tube 362 for the second object to generate an action stream for the first object and the second object, respectively. Yes is as described above.

9 is an exemplary diagram illustrating a process of estimating feature information of a region of interest by a feature information estimator.

When there is a frame image in which the ROI is not detected, the feature information estimator 144 may determine that the ROI is based on the feature information of the ROIs detected in the frame images adjacent to the frame image in which the ROI is not detected. The feature information of the ROI in the undetected frame image is estimated. Since the feature information estimator 144 estimates the feature information in the frame image in which the ROI is not detected using the interpolation process is omitted as described above.

FIG. 10 is an enlarged block diagram of a location information calculator in the embodiment of FIG. 2.

The location information calculator 160 includes an area separator 162 and a rearrangement unit 164. For example, the location information calculator 160 rearranges the ROIs detected in the frame images in frame image units, and positions the action streams from the combined streams generated by concatenating the rearranged ROIs. Compute positional information about. A detailed process of calculating the positional relationship between the location information calculator 160 and the action stream generated for each object is the same as described above, and thus will be omitted.

11 illustrates an image recognition process performed by the image recognition apparatus of the present invention.

The image recognition apparatus 10 of the present invention generates action streams 272 and 274 for each object in an input image including at least one dynamic object, and simultaneously generates a combination stream 279 by combining the action streams generated for each object. Create As described above, the combined stream of the present invention indicates the positional relationship of the action streams 272 and 274 generated for each object.

The image recognition apparatus 10 inputs the action streams 272 and 274 and the combination stream (pairwise stream 279) generated for each object to the first recognizer, and adds the output class vectors in a predetermined manner. In addition, the sum of the class vectors can be used to recognize the behavior of the objects in the image.

12 is an enlarged block diagram of a recognizer in the embodiment of FIG. 1.

The recognition unit 200 includes an adder 220 and a comparator 240. For example, the recognizer 200 uses the generated action stream or position information of the action stream as an input and uses a first recognizer that outputs at least one class vector as an indicator for classifying the behavior of the object. The behavior of the objects can be recognized.

13 is a flowchart of an image recognition method according to an exemplary embodiment.

The image recognition method 10 performed by the image recognition apparatus 10 includes the following steps performed in time series.

In S100, the stream generator 100 generates an action stream including motion information regarding the action for each object in an input image including at least one object. The stream generator 100 may generate an action stream for each object when a plurality of objects are included in the input image. In addition, the stream generation unit 100 generates a combination stream including position information indicating the positional relationship of the action stream generated for each object, thereby recognizing the behavior of the object in the image using the positional relationship of the action stream generated for each object. May be as described above.

In S200, the recognition unit 200 uses the generated action stream or position information of the action stream as an input, and uses a first recognizer that outputs at least one class vector as an indicator for classifying the behavior of the object. The behavior of the objects can be recognized. For example, the recognizer 200 recognizes the behavior of the objects in the image by summing the outputs of the first recognizer that respectively inputs a combination stream indicating the positional relationship between the action stream generated for each object and the action stream generated for each object. The specific method is as described above.

FIG. 14 is an enlarged flowchart of steps generated in the embodiment of FIG. 13. FIG.

In S120, the ROI detector 120 generates a plurality of frame images by dividing an input image over time, and generates an ROI including at least a portion of the objects divided by the objects in the generated frame images. Detect. The method of detecting the ROI from the plurality of frame images by the ROI 120 is the same as described above, and thus will be omitted.

In S140, the ROI 140 may calculate a connection score between the ROIs detected in different frame images, and connect the ROIs detected in the different frame images in consideration of the calculated connection points. . The ROI 140 may connect the ROI in consideration of a connection score between one ROI detected in an arbitrary frame image and all ROIs detected in another frame image.

In S160, the location information calculator 160 rearranges the ROIs for each object detected in the frame images in frame image units, and calculates the positional relationship between the action streams from the combined stream generated by concatenating the rearranged ROIs. The positional information shown can be calculated. The method of rearranging the ROIs detected from the frame images by the location information calculator 160 is omitted as described above.

FIG. 15 is an enlarged flowchart of a detecting step in the embodiment of FIG. 14.

In S122, the feature information calculator 122 may calculate feature information indicating coordinates at which the ROI is located in the generated frame image. In the present invention, the feature information is coordinates regarding the position of the ROI to be detected in the frame image, and may be provided as the center coordinates of the ROI in the frame image.

In S126, the boundary cell removing unit 126 overlaps the object in consideration of whether a probability that the object exists in the boundary cell among the boundary cells including the object in the frame images is greater than or equal to a preset threshold. Remove part of the boundary cell it contains. The region of interest detector 120 detects regions of interest using boundary cells remaining after the boundary cell remover 126 is removed.

FIG. 16 is an enlarged flowchart of connecting steps in the embodiment of FIG. 14;

In operation S142, the connection score calculator 142 outputs a similarity function that inputs feature information of the ROIs detected in each of the adjacent frame images, and an overlap ratio of the ROIs detected in each of the adjacent frame images. The connection score is calculated by considering at least one of a similarity ratio between the intersection ratio function and the class information of the ROIs. Since the connection score calculator 142 calculates the connection scores of the ROIs detected from each of the adjacent frame images, the connection score calculation unit 142 is omitted as described above.

In S144, when the frame image in which the region of interest is not detected is present among the frame images generated by splitting the input image, the feature determining unit 144 detects a frame image adjacent to the frame image in which the region of interest is not detected. The location of the ROI in the frame image in which the ROI is not detected may be estimated using the coordinates of the ROIs.

17 is an enlarged flowchart of calculating position information in the embodiment of FIG. 14.

In operation S162, the area separator 162 may distinguish between the ROI detected for each object and the portion in which the ROI is not detected from the frame images. For example, the area divider 162 may distinguish and mask a region of interest and a non-interest region by using two pieces of pixel information in order to know the location information of the detected regions of interest in the frame image.

In S164, the rearrangement unit 164 rearranges the divided ROI in a predetermined combination method. For example, the rearrangement unit 164 may combine the masked regions of interest 277 related to the first object and the masked regions of interest 279 related to the second object in a predetermined combination method. The method of combining the ROI masked by the array unit 164 may include a method of alternately combining an ROI of the first object and an ROI of the second object, arranging the ROIs of the first object, and arranging the second ROI. It may include a method of rearranging regions of interest relating to an object.

The image recognition apparatus 10 may recognize the behavior of the objects by using the positional relationship of the objects in the image by distinguishing the ROIs detected for each object and rearranging the ROIs.

18 is an enlarged flowchart of a step of recognizing in the embodiment of FIG. 13.

In operation S220, the adder 220 adds the class vectors of the action streams classified for each object by action list using a first recognizer according to a preset method. For example, the adder 220 may add and add weights to class vectors that are outputs of a first recognizer that takes a combination stream indicating a positional relationship between an action stream for each object and an action stream generated for each object. .

In operation S240, the comparison unit 240 compares the probability information for each action list for classifying the behaviors in the summed class vector for each action list of the object stored in the preliminary answer data set (UCF-101 Detection Data Set). Compare with probability information.

All or part of the method of one embodiment of the present invention described above may be implemented in the form of a computer-executable recording medium (or a computer program product), such as a program module executed by a computer. Here, the computer readable medium may include a computer storage medium (eg, memory, hard disk, magnetic / optical media or solid-state drive, etc.). Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media.

In addition, all or part of the method according to an embodiment of the present invention includes instructions executable by a computer, the computer program including programmable machine instructions processed by a processor, and a high-level programming language. Language, an object-oriented programming language, an assembly language, or a machine language.

Means or modules in the present specification may mean hardware capable of performing functions and operations according to each name described in the present specification, and computer program code capable of performing specific functions and operations. It may mean an electronic recording medium, for example, a processor or a microprocessor, on which computer program code capable of performing specific functions and operations may be implemented. In other words, a means or module may mean a functional and / or structural combination of hardware for performing the technical idea of the present invention and / or software for driving the hardware.

Thus, a method according to an embodiment of the present invention may be implemented by executing a computer program as described above by a computing device. The computing device may include at least a portion of a processor, a memory, a storage device, a high speed interface connected to the memory and a high speed expansion port, and a low speed interface connected to the low speed bus and the storage device. Each of these components are connected to each other using a variety of buses and may be mounted on a common motherboard or otherwise mounted in a suitable manner.

The above description is merely illustrative of the technical idea of the present invention, and various modifications, changes, and substitutions may be made by those skilled in the art without departing from the essential characteristics of the present invention. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by the embodiments and the accompanying drawings. . The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (20)

A stream generator configured to generate an action stream including motion information regarding the action of each object in an input image including at least one object; And
Receiving the position information indicating the positional relationship of the action stream in the generated action stream or the input image, by using a first recognizer for outputting at least one class vector as an indicator for classifying the object's behavior Recognition unit for recognizing the behavior; Image recognition device comprising a. The method of claim 1, wherein the stream generating unit
A region of interest detector configured to generate a plurality of frame images by dividing the input image over time, and to detect a region of interest including at least a portion of the object for each object in the generated frame images; More,
And generating the action stream by using the detected region of interest. The method of claim 1, wherein the recognition unit
An adder configured to add a class vector of an action stream generated for each object by using the first recognizer according to a preset method; More,
And recognizing the behavior of the objects using the summed class vector. The method of claim 2, wherein the stream generating unit
A region of interest connection unit configured to calculate a connection score between the ROIs detected from adjacent different frame images, and to connect the ROIs detected from the different frame images in consideration of the calculated connection scores; More,
And generating the action stream for each object using the connected ROIs. The method of claim 4, wherein the stream generating unit
A position information calculator configured to rearrange the ROIs detected from the frame images in units of frame images, and calculate position information regarding mutual positions of the action streams from the combined stream generated by concatenating the rearranged ROIs; ; Image recognition apparatus further comprises a. The method of claim 4, wherein the region of interest connection unit
A similarity function as an input of feature information of the ROIs detected in each adjacent frame image, an intersection ratio function outputting an overlap ratio of the ROIs detected in each of the adjacent frame images, and class information of the ROIs A connection score calculator configured to calculate a connection score in consideration of at least one of similarities of? More,
And connect the ROI using the calculated connection score. The method of claim 6, wherein the region of interest connector is
If there is a frame image in which the ROI is not detected,
A feature information estimator estimating feature information of the ROI in the frame image in which the ROI is not detected, based on feature information of the ROIs detected in the frame images adjacent to the frame image in which the ROI is not detected; More,
And the region of interest is connected using the estimated feature information. The apparatus of claim 2, wherein the ROI detector comprises:
A feature information calculator configured to calculate feature information representing coordinates at which the ROI is located in the generated frame image; More,
And the ROI is detected by using the calculated feature information. The method of claim 5, wherein the location information calculation unit
An area separator configured to distinguish between the ROI detected for each object from the frame images and the portion where the ROI is not detected; And
A rearrangement unit for rearranging the divided ROIs in a predetermined combination method; More,
And calculating the location information based on the combined stream generated by concatenating the rearranged ROIs. The method of claim 8, wherein the feature information calculation unit
A preprocessor for dividing the generated frame images to generate a grid cell having a predetermined size; And
A second recognizer configured to output the generated frame images as inputs and output a center coordinate of a boundary cell indicating a probability that the object exists in the grid cell or a probability that the object exists in the boundary cell; A calculation unit for calculating a center coordinate of each boundary cell and a probability that the object exists in the boundary cell using a neural network; More,
And calculating the feature information using a boundary cell from which the probability and the center coordinates are calculated. The method of claim 10, wherein the region of interest detector is
A boundary cell that removes a part of boundary cells that overlap the object in consideration of whether or not a probability that the object exists in the boundary cell among the boundary cells that overlap the object in the frame images is greater than or equal to a preset threshold. Cell removal unit; More,
And detecting the region of interest using the removed boundary cells. The method of claim 3,
The class vector includes a behavior list classifying the behavior of the object and probability information indicating a probability that the behavior of the objects in the input image corresponds to the behavior list.
And the recognition unit recognizes the actions of the objects by using the probability information for each action list of the summed class vector. Generating an action stream including motion information about the action of each object in an input image including at least one object; And
By using the first recognizer that receives position information indicating the positional relationship of the action stream in the generated action stream or the input image, and outputs at least one class vector as an indicator for classifying the behavior of the object; Recognizing the behavior of objects; Image recognition method comprising a. The method of claim 13, wherein the generating step
Dividing the input image over time to generate a plurality of frame images, and detecting an ROI including at least a part of the object for each object in the generated frame images; More,
And generating the action stream using the detected ROI. 15. The method of claim 14, wherein said generating is
Calculating a connection score between the ROIs detected from adjacent different frame images, and concatenating the ROIs detected from the different frame images in consideration of the calculated connection scores; More,
And generating the action stream for each object using the connected ROIs. The method of claim 15, wherein the generating step
Rearranging the ROIs detected in the frame images in frame image units, and calculating position information regarding mutual positions of the action streams from a combined stream generated by concatenating the rearranged ROIs; Image recognition method further comprises. The method of claim 15, wherein the connecting step
A similarity function as an input of feature information of the ROIs detected in each adjacent frame image, an intersection ratio function outputting an overlap ratio of the ROIs detected in each of the adjacent frame images, and class information of the ROIs Calculating a connection score in consideration of at least one of the similarities of; More,
And connecting the ROI using the calculated connection score. The method of claim 17, wherein the connecting step
If there is a frame image in which the ROI is not detected,
Estimating feature information of the ROI in the frame image in which the ROI is not detected, based on the feature information of the ROIs detected in the frame images adjacent to the frame image in which the ROI is not detected; More,
And connecting the region of interest by using the estimated feature information. The method of claim 16, wherein the calculating of the location information
Dividing the region of interest detected by each object from the frame images and a portion of which the region of interest is not detected; And
Rearranging the separated regions of interest by a predetermined combination method; More,
And calculating the location information based on the combined stream generated by concatenating the rearranged ROIs. A program stored in a computer-readable recording medium which realizes the image recognition method according to any one of claims 13 to 19 through execution by a processor.

Images