KR102261669B1 - Artificial Neural Network Based Object Region Detection Method, Device and Computer Program Thereof - Google Patents

Abstract

The present invention relates to a method and apparatus for detecting an object region based on an artificial neural network, and a computer program therefor, and more particularly, reviewing an object region preliminarily detected based on a parameter value derived from an image decoding process using an artificial neural network. And, by referring this to image analysis, it relates to an artificial neural network-based object region detection method, apparatus, and computer program therefor that can further speed up analysis such as object recognition and tracking.

Description

Artificial Neural Network Based Object Region Detection Method, Device and Computer Program Thereof

The present invention relates to a method and apparatus for detecting an object region based on an artificial neural network, and a computer program therefor, and more particularly, reviewing an object region preliminarily detected based on a parameter value derived from an image decoding process using an artificial neural network. And, by referring this to image analysis, it relates to an artificial neural network-based object region detection method, apparatus, and computer program therefor that can further speed up analysis such as object recognition and tracking.

Recently, image data collected from smart phones, CCTVs, black boxes, high-definition cameras, and the like are rapidly increasing. Accordingly, the requirements for recognizing a person or object based on atypical image data to extract meaningful information and to visually analyze and utilize the content are increasing.

The image data analysis technology refers to various technologies for performing learning and analysis on these various images to search for a desired image or to recognize a situation such as an event occurrence.

However, since the technology for recognizing, analyzing, and tracking image data is an algorithm that requires a considerable amount of computation, that is, the complexity is high, and the larger the size of the image data, the greater the load on the computing device. Accordingly, it takes a longer time to analyze the image data having an increased size. Accordingly, there is a steady demand for a method capable of reducing the time for analyzing image information.

On the other hand, the image security market is continuously growing as awareness of security has been strengthened due to terrorism, etc. in recent years, and accordingly, the demand for intelligent image processing is also increasing.

Recently, a technology capable of efficiently compressing, transmitting, and verifying a high-resolution image based on a block-based video codec according to protocols such as H.264 and H.265 has spread. Such a high-resolution image is also applied to monitoring images such as CCTV, but in the analysis and tracking of such high-resolution images, as the resolution of the image increases, the conventional object detection method requires a higher amount of computation, and therefore, a real-time image There was a point that the analysis was not carried out smoothly.

Meanwhile, in Prior Document 1 (Korean Patent Registration No. 10-1409826, registered on June 13, 2014), a motion vector of a reference frame is calculated based on a histogram of the motion vectors of blocks in a reference frame, and reference is made based on the global motion vector. A technique for determining the type of area of a block is disclosed.

However, in the case of Prior Document 1, since it is necessary to calculate a motion vector for the entire region and calculate histogram data for the entire region, it is difficult to obtain a speed capable of real-time processing in the current high-resolution image, and, Since motion vectors must be considered for all blocks, there is a problem that operations must be performed once for unnecessary blocks.

In addition, when only a motion vector is considered as a factor to be considered as in Prior Document 1, it may be difficult to accurately detect an object region. Accordingly, in the case of Prior Document 1, in order to accurately determine the object region, it is necessary to re-calculate the feature quantities inside the image, so that it is difficult to analyze the high-resolution image quickly and accurately.

Prior Document 1: Korean Patent Registration No. 10-1409826, 'Motion prediction method using an adaptive search range', registered on June 13, 2014)

It is an object of the present invention to receive a bitstream of a compressed image received from a camera, extract data that can be used for object region detection, object recognition, and tracking, and extract a pre-detected object region based on the extracted parameters. It is to provide an object region detection method, apparatus, and computer program therefor that can further speed up analysis such as object recognition and tracking by examining using an artificial neural network and referencing it to image analysis.

In order to solve the above problems, the present invention provides a method for detecting an object region performed in a computing system including one or more processors and a main memory for storing instructions executable by the processor, comprising: a variable-length decoding step for image data; an image decoding step of decoding an image by performing an inverse quantization step, an inverse transform step, and an addition step; size information of data for a macroblock included in the image data; and a first object region detecting step of deriving first object region information of an image based on one or more image decoding parameters extracted in the image decoding step; and a second object region detection step of deriving second object region information by examining the first object region information derived in the first object region detection step through an artificial neural network; It provides an object area detection method including

In the present invention, in the second object region detection step, it is determined whether an object exists in the first object region information derived in the first object region detection step based on a learned recurrent neural network (RNN) model. 2 Object area information can be derived.

In the present invention, the step of detecting the second object region includes: A group object area generation step of deriving group object area information for a group object area generated by grouping macroblocks or subblocks constituting macroblocks in contact with each other among the first object area information detected in the first object area detection step ; a vector sequence generation step of generating sequence data according to time of an image by expressing the group object region information as a real vector; a group object region review step of deriving a probability that an object exists in the group object region by using the sequence data as an input of the recurrent neural network (RNN); a third discrimination step of detecting a second object region based on whether the derived probability that an object exists in the group object region satisfies a preset criterion; may include.

In the present invention, the step of generating the vector sequence comprises: an identification code designation step of designating an identification code in the group object area; a parameter deriving step of deriving an image decoding parameter of a macroblock constituting the group object area or a subblock constituting the macroblock; a group clustering step of clustering identification codes of a group object region determined to include the same object in successive frames of an image based on the image decoding parameter; and a group vector information generation step of expressing the clustered information of the group object area as a vector; may include.

In the present invention, the image decoding parameter includes motion vector information of a macroblock or a subblock constituting the macroblock; and prediction error information of a macroblock or subblocks constituting the macroblock; It may include one or more of

In the present invention, the group clustering step includes a group including the same object based on at least one of location information, motion vector information, and prediction error information of a macroblock constituting the group object area or subblock constituting a macroblock. The object area can be identified.

In the present invention, the vector expressing the information of the group object area includes: location information of the group object area; and parameter information of the group object area; may include.

In the present invention, the parameter information of the group object area includes: a direction histogram for motion vector information of a macroblock constituting the group object area or subblock constituting the macroblock; and a size histogram for motion vector information of a macroblock constituting the group object area or a subblock constituting the macroblock. may include

In order to solve the above problems, the present invention provides an apparatus for detecting an object region performed in a computing system including one or more processors and a main memory for storing instructions executable by the processor, comprising: a variable length decoding step for image data; an image decoding unit for decoding an image by performing an inverse quantization step, an inverse transform step, and an addition step; size information of data for a macroblock included in the image data; and a first object region detection unit for deriving first object region information of an image based on one or more image decoding parameters extracted in the image decoding step. and a second object region detection unit for deriving second object region information by examining the first object region information derived in the first object region detection step through an artificial neural network. It provides an object area detection device comprising a.

In order to solve the above problems, the present invention provides a computer program stored in a computer-readable medium including a plurality of instructions executed by one or more processors, the computer program comprising: a variable-length decoding step for image data; an image decoding step of decoding an image by performing an inverse quantization step, an inverse transform step, and an addition step; size information of data for a macroblock included in the image data; and a first object region detecting step of deriving first object region information of an image based on one or more image decoding parameters extracted in the image decoding step; and a second object region detection step of deriving second object region information by examining the first object region information derived in the first object region detection step through an artificial neural network; It provides a computer program comprising a.

According to an embodiment of the present invention, when performing analysis such as recognizing, tracking, tracking, or identifying an object from encoded image data, since the image analysis is performed only on the detected object area by detecting the object area in advance, It is possible to exert the effect of processing the image at a faster speed.

According to an embodiment of the present invention, even in detecting an object region before image analysis, by selectively removing unnecessary macroblocks, the amount of computation for object region detection is reduced, thereby increasing the computational speed of the system as a whole. can perform

According to an embodiment of the present invention, it is possible to exhibit the effect of detecting an object region determined to exist in which an object may exist with a fairly high level of accuracy despite the minimum amount of computation in object region detection.

According to an embodiment of the present invention, without changing the configuration of the decoder unit for decoding the corresponding image data, by using the parameters generated in the decoding process in the corresponding decoder unit, H.264 and H.265, etc. In the case of a codec method using blocks according to the present invention, even if the codec is changed, an effect that can be easily applied can be exhibited.

According to an embodiment of the present invention, it is possible to exhibit the effect of detecting the object region with a high level of accuracy by first detecting the object region and then examining the reliability of the detected object region based on the artificial neural network.

1 is a diagram schematically illustrating the overall structure of an object region detection system according to an embodiment of the present invention.
2 is a diagram schematically illustrating a detailed configuration of an object region detection system according to an embodiment of the present invention.
3 is a diagram schematically illustrating a structure before decoding of a data stream of image data according to an embodiment of a video codec using blocks according to protocols such as H.264 and H.265.
4 is a diagram schematically illustrating a data field structure of a macroblock of image data according to an embodiment of a video codec using a variable block.
5 is a diagram schematically illustrating a detailed configuration of a first object area detection unit according to an embodiment of the present invention.
6 is a diagram illustrating some examples of macroblocks.
7 is a diagram illustrating an example of a macroblock including subblocks.
8 is a block diagram illustrating a process of a first object region detection step according to an embodiment of the present invention.
9 is a diagram illustrating block division information for an image screen according to an embodiment of the present invention.
10 is a diagram illustrating motion vector information for a video screen according to an example of the present invention.
11 is a diagram illustrating prediction error information for a video screen according to an example of the present invention.
12 schematically shows an encoder system for generating an encoded image according to an embodiment of the present invention.
13 is a diagram schematically illustrating examples of frames of image data.
14 is a diagram schematically illustrating a detailed configuration of a second object area detection unit according to an embodiment of the present invention.
15 is a diagram illustrating a result of a group object area creation step according to an embodiment of the present invention.
16 is a diagram schematically illustrating a detailed configuration of a vector sequence generator according to an embodiment of the present invention.
17 is a diagram illustrating a process of a group clustering step according to an embodiment of the present invention.
18 is a diagram illustrating a configuration of group vector information according to an embodiment of the present invention.
19 is a diagram schematically illustrating an operation of a group object area review unit according to an embodiment of the present invention.
20 is a diagram schematically illustrating an operation of a group object area review unit according to an embodiment of the present invention.
21 is a diagram schematically illustrating an operation of a group object area review unit according to an embodiment of the present invention.
22 exemplarily illustrates an internal configuration of a computing device according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS Various embodiments are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In this specification, various descriptions are presented to provide an understanding of the present invention. However, it is apparent that these embodiments may be practiced without these specific descriptions. In other instances, well-known structures and devices are provided in block diagram form in order to facilitate describing the embodiments.

As used herein, the terms “component,” “module,” “system,” “part,” and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and/or thread of execution, and a component may reside within one computer.

It may be localized within, or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored therein. The components may contain, for example, a signal having one or more data packets (eg, data from one component interacting with another component in a local system, a distributed system, and/or data via a network such as the Internet with another system via a signal). ) may communicate via local and/or remote processes.

Also, the terms "comprises" and/or "comprising" mean that the feature and/or element is present, but excludes the presence or addition of one or more other features, elements, and/or groups thereof. should be understood as not

Also, terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, are generally understood by those of ordinary skill in the art to which the present invention belongs. have the same meaning as Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the embodiment of the present invention, an ideal or excessively formal meaning is not interpreted as

1 is a diagram schematically illustrating the overall structure of an object region detection system according to an embodiment of the present invention.

The object region detection system in FIG. 1 includes a first object region detection unit 2000, a second object region detection unit 4000, an image decoding unit 1000, and an image analysis unit 3000 that process received image data in a broad sense. includes The video data is video data encoded by a standard codec, and preferably corresponds to data encoded using a variable size block, preferably according to a protocol including H.264 and H.265 codec methods. Corresponds to the encoded video data. More preferably, it corresponds to video data encoded according to a protocol using a variable size block.

The image data may correspond to images stored in the object area detection system or image data received by another image collecting device (eg, CCTV or monitoring device) in real time.

The image decoding unit 1000 corresponds to an apparatus for decoding or decoding an image encoded according to a protocol including the H.264 and H.265 codec methods, and is configured according to the decoding method of the corresponding codec.

The image analysis unit 3000 performs analysis such as object recognition, tracking, and identification on the decoded and received image, and the image analysis unit 3000 includes a preprocessor, a feature information extractor, and a feature information analysis according to the purpose of image analysis. It may include various components such as a part.

In the present invention, in order to further improve the image processing speed of the image analysis unit 3000 , the first object area detection unit 2000 and the second object area detection unit 2000 are introduced. The first object region detection unit 2000 includes information extracted from the image data itself and information and decoding of undecoded image data, not the decoded image itself, such as image decoding parameters extracted in the image decoding process of the image decoding unit 1000 . The first object region is detected through the parameter information extracted in the process, and the first object region information is transmitted to the second object region detection unit 4000, and the second object region detection unit includes the first object region information. is reviewed through an artificial neural network, and the second object region information is transmitted to the image analysis unit 3000 . At this time, the second object region detection unit 4000 determines whether an object exists in the first object region information derived in the first object region detection step based on the learned recurrent neural network (RNN) model. The second object area information can be derived.

Therefore, the image analysis unit 3000 does not perform object recognition, feature point extraction, pre-processing, etc. on the entire image, but only on the region according to the second object region information received from the second object region detection unit 4000 . analysis can be performed.

In an embodiment of the present invention, the image analysis unit 3000 sets an object area to be analyzed based on the second object area information received from the second object area detection unit 4000 , and thus object recognition and feature point extraction , preprocessing, etc. However, the present invention is not limited thereto, and in another embodiment of the present invention, the image analysis unit 3000 is a preliminary object area to be analyzed based on the second object area information received from the second object area detection unit 4000 . is set, and the object area is set through a process such as additional object area detection or object area detection according to preliminary object area verification for the preliminary object area, and object recognition, feature point extraction, pre-processing, etc. can be performed accordingly. .

That is, in embodiments of the present invention, the object region is not determined by the second object region detection unit, but an additional object using the results of the first object region detection unit and/or the second object region detection unit in the image analysis unit. Derivation or verification of the area may be performed.

The first object region detection unit 2000, the second object region detection unit 4000, the image decoding unit 1000, and the image analysis unit 3000 may be configured as a single computing device, but are configured by two or more computing devices. could be

That is, the object region detection system according to the present invention may be implemented by a computing system including one or more processors and a main memory for storing instructions executable by the processor.

2 is a diagram schematically illustrating a detailed configuration of an object region detection system according to an embodiment of the present invention.

The object region detection apparatus according to an embodiment of the present invention is implemented by a computing system including one or more processors and a main memory for storing instructions executable by the processor.

As shown in FIG. 2 , an image decoding unit 1000 including a variable length decoding unit 1100 , an inverse quantization unit 1200 , an inverse transform unit 1300 , an adder 1400 , and a prediction unit 1500 . Decodes the encoded image data by the image encoding method.

The configuration of the image decoding unit 1000 may be configured according to the configuration of an encoding unit for encoding image data. For example, the configuration of the image decoding unit 1000 shown in FIG. 2 corresponds to a configuration for decoding an image encoded according to FIG. 12 , and in an embodiment of the present invention, the H.264 or H.265 codec is used. 2 and 12 are shown.

The variable-length decoding unit 1100 variable-length decodes (decodes) input image data. Through this, the variable length decoding unit 111 may separate the image data into motion vectors, quantization values, and DCT coefficients or extract motion vectors, quantization values, and DCT coefficients from the image data.

The inverse quantization unit 1200 inversely quantizes the DCT coefficient output from the variable length decoding unit 1100 according to the extracted quantization value.

The inverse transform unit 1300 performs inverse transform (IDCT) on the DCT coefficient inverse quantized by the inverse quantizer 1200 to obtain a differential image.

The prediction unit 1500 performs prediction according to whether the corresponding frame is intra mode or inter mode. The motion compensator 1530 of the prediction unit 1500 compensates for the motion of the current image data using the motion vector and previous image data. Through this, the motion compensator generates a predicted image.

The configuration of the variable-length decoding unit 1100, the inverse quantization unit 1200, the inverse transform unit 1300, the adder 1400, and the prediction unit 1500 of the image decoding unit 1000 is the encoding method of the image data or It may be changed according to the codec, which a person skilled in the art may implement according to the codec of the corresponding image data. The present invention has the advantage that it can be configured by adding the first object region detection unit 2000 to the existing decoding unit for decoding an image according to the protocol such as H.264 and H.265.

The first object area detection unit 2000 may include size information of data for a macroblock included in the image data; and deriving first object region information of an image based on one or more image decoding parameters extracted in the image decoding step. The derived first object region information is reviewed through the second object region detection unit 4000, and the second object region detection unit 4000 derives second object region information based on the result of the review and analyzes the image. It is transmitted to the unit 3000, and the image analysis unit 3000 performs image processing on only the second object region information, thereby significantly reducing the amount of computation required for image analysis.

3 is a diagram schematically illustrating a structure before decoding of a data stream of image data according to an embodiment of a video codec according to protocols such as H.264 and H.265.

The data stream shown in FIG. 3 corresponds to image data that is stored or transmitted without performing any decoding input to the variable length decoding unit 1100 in FIG. 2 . Such image data or data stream consists of a Network Abstraction Layer (NAL), and each NAL consists of a Nal Unit and a Raw Byte Sequence Payload (RBSP) as a payload. The NAL may correspond to a unit in which parameter information such as SPS and PPS is described or a unit in which slice data corresponding to a video coding layer (VCL) is described.

The SLICE NAL corresponding to the VCL consists of a header and data, where the data consists of a plurality of macroblock fields and a delimiter field. In the present invention, the encoding method of image data for performing object region detection is a method of encoding data in the NAL state into macroblocks having a constant block size. In the data field divided into MB in FIG. 3, data for a block of a certain size is encoded.

The first discriminating unit 2100 of the first object region detecting unit 2000, which will be described later, uses the data size of each macroblock, that is, the portion indicated by MB in FIG. 3 from the image data that has not been decoded. According to this method, it is possible to preliminarily derive the object region candidate region using the original data before the primary decoding procedure such as the variable length decoder 1100 .

In the case of video data encoded with a general H.264 or H.265 codec, the MB data field of FIG. 3 encodes and stores data for a macroblock having a size of 16 pixels x 16 pixels, and a variable length decoding unit 1100 ), the detailed block information of the macroblock can be checked in a partially encoded form.

4 is a diagram schematically illustrating a data field structure of a macroblock of image data according to an embodiment of a video codec using a variable block.

The data field of the macroblock shown in FIG. 4 is decoded by the variable length decoding unit 1100 . Basically, the data field of the macroblock includes a Type field including information such as the size of the block; a prediction type field including information on whether encoding is performed in intra mode or inter mode, and reference frame information and motion vector information in case of inter mode; a Coded Picture Buffer (CPB) field including information for maintaining a bitstream of a previous picture input during decoding; a Quantization Parameter (QP) field including information on a quantization parameter; and a DATA field including information on DCT coefficients for the color of the corresponding block.

When a macroblock includes a plurality of subblocks, a plurality of data units of type-prediction type-CPB-QP-DATA shown in the second column of FIG. 4 are connected.

The second determining unit 2200, which will be described later, includes a block size and a block size that can be known from the type field of the entire macroblock (when there is no subblock) or subblocks constituting the macroblock decoded by the variable length decoding unit 1100. An object region is detected using motion vector information known from the prediction type field.

On the other hand, the color information of the DATA field includes information on a plurality of colors (including color information in the YCbCr system in FIG. 4) in an encoded form. The color information of the DATA field is decoded by the inverse quantization unit 1200 and the inverse transform unit 1300, and the color value in the original image data and the image decoding unit 1000 in the adder 1400. Prediction error information corresponding to the difference between the color values predicted by the prediction unit 1500 may be derived.

As described above, the prediction error information that can be derived from the adder 1400 may be used to detect the object region by the second determiner 2200 .

5 is a diagram schematically illustrating a detailed configuration of the first object area detection unit 2000 according to an embodiment of the present invention.

extracting, by the first object region detection unit 2000, size information of macroblock data from the image data before the variable length decoding step is performed; and determining whether size information of data for the macroblock meets a preset criterion. a first determining unit 2100 that performs and a second method for detecting a first object region based on one or more image decoding parameters extracted in the image decoding step with respect to a macroblock whose size information of data satisfies a preset criterion by the first determining unit 2100 a determining unit 2200; and an object region output unit 2300 for outputting information on the first object region detected by the second determining unit 2200 to the second object region detecting unit 4000 .

In the first determining step, as shown in FIG. 2 , size information of data for each macroblock is derived from the NAL format data stream of image data not decoded by the variable length decoding unit 1100 . That is, based on the data size of each data field denoted by MB in FIG. 3 , it is determined whether each macroblock is a target for determination by the second determining unit 2200 to be described later.

In the image encoding method using variable size blocks, in the case of a macroblock in which a complex image is located, the macroblock (16x16) is divided into a plurality of subblocks (8x8, 4x4, etc.) or because it contains various information, the corresponding macroblock is it gets bigger In addition, in video encoding methods such as H.264 and H.265, when there are frequently occurring values and non-frequently occurring values among macroblock data, a short code is assigned to a frequently occurring value, and a long code is assigned to a value that does not occur frequently. is encoded in a way to reduce the total data amount by allocating

The first determining unit 2100 derives the data size of each macroblock from the image data before the decoding procedure of the variable-length decoding unit 1100 is performed by using this characteristic of the encoding method using the variable size block. and, when the corresponding data size satisfies a preset criterion, that is, when it is greater than or equal to a preset value, it is classified as a target for discrimination by the secondary discrimination unit.

In this way, in the present invention, without changing the image decoding unit 1000 , valid data capable of detecting an object region is extracted from the image data input to the image decoding unit 1000 , and the corresponding macroblock is processed with a simple operation. Determines whether there is an object area.

In another embodiment of the present invention, the determination result by the first determination unit 2100 and the determination result by the second determination unit 2200 are comprehensively determined to determine whether each macroblock or subblock is an object area. can also be judged.

Meanwhile, when the macroblock does not include a subblock, the second determining unit 2200 detects an object region based on one or more image decoding parameters for the entire macroblock, and the macroblock is a subblock. , the object region is detected based on one or more image decoding parameters for each of the sub-blocks.

Here, the second determining unit 2200 derives object region information based on one or more of block size information for the target block, motion vector information, and prediction error information, and preferably, the block for the target block. First object region information is derived based on size information, and at least two pieces of information among motion vector information and prediction error information, and most preferably, the first object region information is derived based on block size information, motion vector information, and prediction error information. Derive area information.

Specifically, the second determining unit 2200 derives first object region information based on the evaluation result of at least one of block size information for the target block, motion vector information, and prediction error information, and preferably , derives the first object area information by synthesizing each evaluation result for two or more pieces of block size information on the target block, motion vector information, and prediction error information, and most preferably, block size information; And by synthesizing each evaluation result of motion vector information and prediction error information, the first object region information is derived. In one embodiment of the present invention, scoring is performed according to a predetermined criterion for each of block size information, motion vector information, and prediction error information, and a weight is applied to the evaluation score of each element to derive a comprehensive score. After this, the first object region information may be derived according to whether the derived score satisfies a preset condition or value.

In an embodiment of the present invention, the one or more image decoding parameters include block size information of a macroblock or a subblock constituting the macroblock. The block size information relates to whether the corresponding block is, for example, 16x16 in size, 8x8 in size, or 4x4 in size.

In the case of the area where the object exists, it is highly likely to have a more complex shape than the background area. On the other hand, in the image encoding method using a variable block, a macroblock having a complex shape is divided into a plurality of sub-blocks and encoded, and in an embodiment of the present invention, the second discrimination unit 2200 uses such encoding characteristics. Perform object area detection in That is, the second determining unit 2200 determines that the smaller the size of the macroblock or subblock for determining whether it is an object area, the higher the probability that it corresponds to the object area, or the smaller the size of the score for determining whether it is an object area. Gives a higher score than the block.

Here, as shown in FIG. 5, the block size information is derived from information decoded in the variable length decoding step. In this way, it is possible to derive a parameter capable of detecting the object region while maintaining the configuration of the image decoding unit 1000 without generating additional feature information for deriving the object region.

In an embodiment of the present invention, the one or more image decoding parameters include motion vector information of a macroblock or a subblock constituting the macroblock. include Each macroblock or subblock of the intermode frame includes motion vector information (direction and magnitude), and the second determining unit 2200 uses the motion vector magnitude information among the motion vector information to use the corresponding macroblock or subblock. Determines whether a block can correspond to an object area or a value related to the determination.

In the case of a region where an object exists, there is a higher possibility of movement than the background region. Meanwhile, each image block of a reference frame (eg, P frame) in an image encoding method using a variable block includes size information of a motion vector with respect to the reference frame. In an embodiment of the present invention, the object region detection is performed by the second determining unit 2200 by using such an encoding characteristic. That is, the second determining unit 2200 determines that the larger the size of the motion vector of the macroblock or subblock to determine whether it is an object region, the higher the probability that it corresponds to the object region, or the motion in the score for determining whether the object region is an object region. A block with a smaller vector size is given a higher score than a block.

Here, as shown in FIG. 5 , the motion vector information is derived from information decoded by the variable length decoding unit 1100 . In this way, it is possible to derive a parameter capable of detecting the object region while maintaining the configuration of the image decoding unit 1000 without generating additional feature information for deriving the object region.

In an embodiment of the present invention, the one or more image decoding parameters include prediction error information of a macroblock or a subblock constituting the macroblock.

The prediction error information is derived in the adding step, and the prediction error information is color information based on the decoded image data for the macroblock or subblock constituting the macroblock, and the prediction step performed in the adding step. It is derived based on the difference in color information of the corresponding macroblock or subblocks constituting the macroblock.

In the case of a region in which an object exists, there is a high possibility that there is a color change or a morphological change of the image, and therefore the size of the prediction error information is high.

Preferably, in the present invention, in the YCrCb color space, the object region is detected by the second determining unit 2200 based on the prediction error information on the Y color information corresponding to the luminance value (LUMA) region.

That is, the second determining unit 2200 determines that the prediction error information for Y color information of a macroblock or subblock to determine whether it is an object region is higher, determines that it is more likely to correspond to the object region, or determines whether the object region is an object region or not. In the score to be determined, a higher score is given to a block having a smaller prediction error information.

As described above, the second determining unit 2200 determines whether the corresponding macroblock or subblock corresponds to the object region based on one or more values among block size information, motion vector information, and prediction error information. Determination of whether or not the final object area is the final object area by the second determining unit 2200 is determined as an object area when each piece of information exceeds a predetermined reference value, or a predetermined number when each piece of information exceeds a predetermined reference value In the case of abnormality, it may be determined as an object area, or a score may be given for each piece of information according to a predetermined rule, and if the overall score for each score exceeds a predetermined reference value, it may be determined as the final object area.

On the other hand, the discrimination criteria or reference values of the first discriminating unit 2100 and the second discriminating unit 2200 are determined by statistical values derived from the corresponding image or statistical values derived through image processing in the system or input values. can be implemented.

In a preferred embodiment of the present invention, the first discriminating unit 2100 filters the discrimination target primarily by the size of macroblock data, and the second discriminating unit 2200 includes block size information, motion vector information, and prediction error information. by comprehensively determining the block corresponding to the object area. Accordingly, it is possible to detect the object region more accurately than in the case where only motion vectors are used under minimal computational load.

In another embodiment of the present invention, the first object area detection unit 2000 may derive the object area by synthesizing the determination results of the first determination unit 2100 and the second determination unit 2200 . Specifically, the score for the size information of the macroblock in the first determining unit 2100, the block size information in the second determining unit 2200, the motion vector information, and the prediction error information in the discrimination score for one or more Based on the whole macroblock or subblocks constituting the macroblock, the object region may be detected.

In another embodiment of the present invention, the first object area detecting unit 2000 may derive the object area based on the determination result of the first determining unit 2100 .

In another embodiment of the present invention, the first object area detection unit 2000 may derive the object area based on the determination result of the second determination unit 2200 .

6 is a diagram illustrating some examples of macroblocks.

FIG. 6(A) shows a case in which a macroblock consists of one block, FIG. 6(B) shows a case in which it consists of 4 blocks, and FIG. 6(C) shows a case in which it consists of 7 blocks and FIG. 6(D) shows a case of 16 blocks.

As described above, the first determining unit 2100 determines data size information for each macroblock. In the case of a plurality of sub-blocks as shown in (D) of FIG. 6 , there is a high possibility that the data size of the macroblock is generally higher than that of FIG. The first determining unit 2100 determines the macroblock as shown in FIG. 6(D) in a direction that is more likely to correspond to the object area than the macroblock as shown in FIG. 6(A).

On the other hand, the block size of the sub-block in Fig. 6 (D) is smaller than the block size of the entire sub-block in Fig. 6 (B) or the macroblock in Fig. 6 (A), so the second discrimination unit ( 2200) performs the determination in a direction in which the sub-block in FIG. 6 (D) is more likely to correspond to the object area than the sub-block in FIG. 6 (B) or the entire macroblock of FIG. 6 (A). .

7 is a diagram illustrating an example of a macroblock including subblocks.

In encoding image data using variable size blocks, subblocks in the same macroblock may have different block sizes and motion vectors as shown in FIG. 7 .

Blocks #4, #5, #6, and #7 of FIG. 7 have smaller block sizes and larger motion vectors than blocks #1, #2, and #3. , #5, #6, and #7 may be given a higher score than blocks #1, #2, and #3 in the score of whether the block corresponds to the object area.

8 is a block diagram illustrating a process of a first object region detection step according to an embodiment of the present invention.

8A shows 16 macroblocks.

8B shows the steps of extracting size information of macroblock data from the image data before the variable length decoding step is performed; and determining whether the data size information for the macroblock satisfies a preset criterion. A portion indicated by a dotted line in FIG. 8B corresponds to macroblocks in which the size of data for the macroblock is less than a preset value, and a portion indicated by a solid line in FIG. 8B is the size of data for the macroblock. Corresponds to macroblocks greater than or equal to a preset value.

FIG. 8(C) shows one or more images extracted in the image decoding step with respect to a macroblock (solid line area in FIG. 8(B)) in which data size information meets a preset criterion by the first discrimination step. A state in which the second discrimination step of detecting the object region is performed based on the decoding parameter is shown.

A portion indicated by a dotted line in FIG. 8(C) corresponds to a block that does not satisfy a preset criterion in the second discrimination step, and a part indicated by a solid line in FIG. It corresponds to a satisfied subblock or macroblock.

In a preferred embodiment of the present invention, some of the macroblocks are excluded from the discrimination target of the second determining unit 2200 by the first determining unit 2100, and macroblocks satisfying the criteria of the first determining unit 2100 By performing the discrimination of the second discriminator 2200 only with respect to the images, the amount of computation in the object region detection step can be further reduced, and as a result, the overall image processing speed can be improved.

9 is a diagram illustrating block division information for an image screen according to an embodiment of the present invention.

As shown in FIG. 9 , in the case of an object area, it can be confirmed that each macroblock is divided into more detailed blocks, and by reflecting such characteristics, the second determining unit 2200 determines the corresponding detailed blocks or macros. Determines whether a block is an object area.

10 is a diagram illustrating motion vector information for a video screen according to an example of the present invention.

As shown in FIG. 10 , in the case of an object area, it can be seen that the size of the motion vector of each macroblock or subblock increases. Reflecting this characteristic, the second determining unit 2200 determines the corresponding It is determined whether a subblock or macroblock is an object area.

11 is a diagram illustrating prediction error information for a video screen according to an example of the present invention.

As shown in FIG. 11 , in the case of an object area, prediction error information of each macroblock or subblock (in FIG. 11, prediction error information about a Y value corresponding to a brightness value in the YCrCb color space among prediction error information) It can be seen that the value of light (expressed as a large number) and dark (expressed as a small value) increases depending on the numerical value, and reflecting these characteristics, the second discrimination unit determines whether the corresponding subblock or macroblock Determines whether it is an object area.

12 schematically shows an encoder system for generating an encoded image according to an embodiment of the present invention.

The first object region detecting unit and the image decoding unit according to an embodiment of the present invention described above may be applied to image data encoded using a variable size block as shown in FIG. 6 . As a representative example, it may be applied to image data encoded by the H.264 or H.265 codec.

The encoder 10 shown in FIG. 12 includes a DCT unit (Discrete Cosine Transform) 11, a quantization unit 12, and an inverse quantization unit to generate data shown as image data in FIGS. 1 and 2 . (Inverse Quantization; IQ) 13, Inverse Discrete Cosine Transform (IDCT) 14, Frame Memory 15, Motion Estimation and Compensation (ME/MC) 16 and Variable It may include a Variable Length Coding (VLC) 17 . Likewise, it is preferable that the image decoding unit is configured to correspond to the configuration of the encoding unit.

To briefly explain this, the DCT unit 11 performs a DCT operation on image data input in units of pixel blocks of a preset size (eg, 4×4) in order to remove spatial correlation.

Thereafter, the quantization unit 12 quantizes the DCT coefficients obtained by the DCT unit 11 and expresses them as several representative values, thereby performing high-efficiency lossy compression.

Also, the inverse quantization unit 13 inversely quantizes the image data quantized by the quantization unit 12 .

The inverse transform unit 14 performs IDCT transformation on the image data inverse quantized by the inverse quantizer 13 .

The frame memory 15 stores the IDCT-transformed image data in the inverse transform unit 14 in units of frames.

On the other hand, the motion estimation and compensation unit 16 estimates a motion vector (MV) per macro block using the input image data of the current frame and the image data of the previous frame stored in the frame memory unit 15 to block the block. Calculate the sum of absolute difference (SAD) corresponding to the blockmatching error.

The variable length coding unit 17 removes statistical redundancy from the DCT and quantized data according to the motion vector estimated by the motion estimation and compensation unit 16 .

13 is a diagram schematically illustrating examples of frames of image data.

The video portion of a typical moving picture includes I frames (frames denoted by “I” in Fig. 13), P frames (frames denoted as “P” in Fig. 13), and B frames (frames denoted by “B” in Fig. 13). ) is composed of

I-frame is a key frame that includes the entire image, can function as an access point in a video file, corresponds to an independently encoded frame, and has a low compression rate.

On the other hand, in the case of a P frame, as a frame generated by forward prediction with reference to a previous I frame or P frame, it does not correspond to an independently encoded frame. Such a P frame has a higher compression ratio than the I frame. Here, the "previous" frame means one of a plurality of frames existing before the frame as well as the immediately preceding frame, and the "following" frame means not only the next frame but also one of a plurality of frames existing after the corresponding frame. means one

On the other hand, in the case of a B frame, it is a frame created by forward and backward prediction with reference to a previous frame and a subsequent frame, and does not correspond to an independently encoded frame. Such a B frame has a higher compression ratio than the I and P frames. Accordingly, the independently encoded frame may correspond to an I frame, and the non-independently encoded frame may correspond to the remaining B frames or P frames.

The B and P frames correspond to reference frames, and preferably, the first object area detection unit detects the object area with respect to the reference frames.

14 is a diagram schematically illustrating a detailed configuration of the second object area detection unit 4000 according to an embodiment of the present invention.

The second object area detection unit 4000 is a group object area generated by grouping abutting macroblocks or subblocks constituting macroblocks among the first object area information detected by the first object area detecting unit 2000 . a group object area generating unit 4100 deriving group object area information; a vector sequence generator 4200 for generating sequence data according to time of an image by expressing the group object region information as a real vector; a group object area review unit 4300 for deriving a probability that an object exists in the group object area by using the sequence data as an input of the recurrent neural network (RNN); a third determining unit 4400 configured to detect a second object region based on whether the derived probability that an object exists in the group object region satisfies a preset criterion; includes

The group object area generating unit 4100 receives the first object area information detected by the first object area detecting unit 2000, and divides adjacent macroblocks or subblocks constituting the macroblock among the first object area information with each other. group to create a group.

When an object exists in image data, the object may exist within a macroblock of the image data, or may exist across a plurality of macroblocks or subblocks constituting the plurality of macroblocks. As described above, when there is an object occupying a plurality of macroblocks or subblocks constituting the plurality of macroblocks, the macroblocks or subblocks occupied by the object may be grouped and analyzed. In this way, when macroblocks or subblocks are grouped and analyzed as a group, the existence of an object in a plurality of blocks can be simultaneously determined, and the existence of an object can be organically identified in successive frames to make an accurate determination. be able to

The vector sequence generator 4200 generates sequence data according to time by expressing the characteristic information on the group object region as a vector. In the present invention, the second object region detection unit 4000 may determine whether an object exists in the first object region information based on a recurrent neural network (RNN) model. To this end, the vector sequence generator 4200 creates vector sequence data according to time that can be inputted to the recurrent neural network (RNN), so that the information on the group object region can be analyzed through the recurrent neural network (RNN). do.

The sequence data according to time may include information on the group object area. At this time, when a plurality of group object regions exist, the vector sequence generator 4200 adds an identification code to the group object region so that the vector sequence generator 4200 can analyze each group object region. Sequence data can be generated by designating, deriving parameters of each group object area, classifying the group object area determined to contain the same object in successive frames, and expressing it as a vector.

At this time, the vector sequence generator 4200 calls and utilizes motion vector information or prediction error information from the variable length coding unit 1100 or the adder 1400, respectively, in order to derive the parameters of the group object region. can The motion vector information and the prediction error information may be used as information for generating sequence data, or may be used as information for determining whether the same object is included in the group object area to generate the sequence data.

The group object area review unit 4300 analyzes the group object area based on a recurrent neural network (RNN). A recurrent neural network (RNN) is an artificial neural network suitable for analyzing time series data. In the present invention, the sequence data generated by the vector sequence generator 4200 is input to the recurrent neural network (RNN) to analyze information of the group object area. can do.

In an embodiment of the present invention, the group object area review unit 4300 performs learning on the recurrent neural network (RNN) in advance, and analyzes the group object area based on the learned recurrent neural network (RNN). can be done

In an embodiment of the present invention, the group object area review unit 4300 may receive the sequence data including information on the group object area, and derive a probability that an object exists in the group object area. In the present invention, the third determining unit 4400 can analyze the group object area based on such a probability.

The third determining unit 4400 analyzes the result derived from the recurrent neural network (RNN) in the group object area review unit 4300 to determine whether an object exists in the group object area. In an embodiment of the present invention, the group object area review unit 4300 derives the probability that an object exists in the group object area, and the third determination unit 4400 determines whether the probability meets a preset criterion. based on the reliability of the group object area. In an embodiment of the present invention, the group object area review unit 4300 derives the group object area from the first object area detection unit 2000 when the probability that an object exists in the group object area is less than a preset reference value. The second object region information is derived and transmitted to the image analysis unit 3000 by deleting it from the first object region information and leaving only the case where the probability that an object exists in the group object region is greater than or equal to a preset reference value. The image analysis unit 3000 can analyze the image at a faster speed and with high accuracy by analyzing the image based on the reviewed second object region information.

In an embodiment of the present invention, the second object area information may include information on the group object area. As described above, since the information on the group object region generated by the group object region generating unit 4100 is included in the second object region information, when the image analysis unit 3000 analyzes the image, the grouped macroblock or By detecting the object based on the sub-block, the effect of more easily detecting the object can be exhibited.

15 is a diagram illustrating a result of a group object area creation step according to an embodiment of the present invention.

FIG. 15 shows a state in which some macroblocks among a plurality of macroblocks are discriminated as a first object area by the first object area detection unit 2000 . Macroblocks determined as the first object area by the first object area detection unit 2000 are indicated by solid lines, and other macroblocks are indicated by dotted lines. In the embodiment of the present invention, the object area may be set in units of subblocks constituting macroblocks as well as macroblocks. However, for convenience of explanation, FIG. 15 shows an embodiment of image information composed of only macroblocks without subblocks. do.

In Figure 15, blocks #33, #34, #3a, #3b, #43, #44, #4a, #4b, #4c, #53, #5a, #5b, #5c, #6a, and #6b are It is detected as 1 object area.

Among them, blocks #33 and #34 are in contact with each other, and the group object area generating unit 4100 groups them into a group G1 to create a group object area. Also, the group G1 composed of blocks #33 and #34 is in contact with blocks #43 and #44, and the group object area generator 4100 adds the blocks #43 and #44 to the group G1 to group them. Also, the group G1 composed of blocks #33, #34, #43, and #44 is in contact with the block #53, and the group object area generator 4100 adds the block #53 to the group G1 to group them.

In this way, the group object area generating unit 4100 groups blocks #3a, #3b, #4a, #4b, #4c, #5a, #5b, #5c, #6a and #6b into a group G2. You can create a group object area.

In another embodiment of the present invention, the group object area generating unit 4100 may create a group object area by grouping not only the blocks in contact with each other but also macroblocks or subblocks constituting the macroblocks within a preset distance.

For example, when the preset distance is twice the width of the macroblock and block #64 is detected as the first object area in the embodiment of FIG. 15, it is located within a preset distance from block #53 of the group G1. Group G1 can be grouped by adding the block #64.

16 is a diagram schematically illustrating a detailed configuration of a vector sequence generator 4200 according to an embodiment of the present invention.

The vector sequence generation unit 4200 includes: an identification code designation unit 4210 for designating an identification code in the group object area; a parameter deriving unit 4220 for deriving an image decoding parameter of a macroblock constituting the group object area or a subblock constituting the macroblock; a group clustering unit 4230 for clustering identification codes of group object regions determined to include the same object in consecutive frames of an image based on the image decoding parameter; and a group vector information generating unit 4240 for expressing the clustered information of the group object area as a vector; includes

The image decoding parameter may include motion vector information of a macroblock or a subblock constituting the macroblock; and prediction error information of a macroblock or subblocks constituting the macroblock; It may include one or more of

The identification code designation unit 4210 designates an identification code in the group object area. The identification code is information for distinguishing the group object area generated by the group object area generating unit 4100. Preferably, for each frame of image data, an identification code is applied to each of all group object areas existing in the frame. can be specified.

The parameter derivation unit 4220 derives image decoding parameters of macroblocks constituting the group object area or subblocks constituting macroblocks. The image decoding parameter of each macroblock or subblock constituting the group object region may be obtained from the variable length coding unit 1100 or the adder 1400 of the image decoding unit 1000 . The parameter derivation unit 4220 may be directly connected to the variable length coding unit 1100 and the adder 1400 to obtain image decoding parameters such as motion vector information or prediction error information, or the image decoding parameter It may be obtained through the first object image detection unit 2000 that has obtained .

The group clustering unit 4230 is a group object region including the same object based on at least one of location information, motion vector information, and prediction error information of a macroblock constituting the group object region or subblock constituting the macroblock. can be identified. The group clustering unit 4230 classifies a plurality of group object regions existing in each frame of image data into group object regions determined to contain the same object, so that the group object region can be analyzed over time. For example, the group clustering unit 4230 classifies a group object area having similar location information of the group object area in an adjacent frame. At this time, when an object included in a specific group object area moves in a specific frame, the movement will appear as a motion vector in the macroblock of the group object area, and in the frame immediately after, the movement is performed according to the motion vector. Since there is a high probability that the object will be located, if there is a group object area existing at a position moved according to the motion vector, it may be determined that the same object is included.

The group vector information generating unit 4240 expresses information that can represent the characteristics of the group object region as a vector so that it can be input to the RNN of the group object region review unit 4300 . To this end, the group vector information generating unit 4240 may obtain information on the macroblock constituting the group object area or the subblock constituting the macroblock through the parameter deriving unit 4220 or the like.

In an embodiment of the present invention, the vector expressing the information of the group object area includes location information of the group object area; and parameter information of the group object area; may include. As described above, it is possible to determine whether an object exists in the group object area by indicating the characteristics of the group object area through location information and parameter information, and examining it through a recurrent neural network (RNN).

17 is a diagram illustrating a process of a group clustering step according to an embodiment of the present invention.

Frame 1 and Frame 2 shown in FIG. 17 are continuous frames included in the image data. In detail, frame 2 is a frame immediately after frame 1.

In frame 1, two group object areas to which identification codes are designated by the identification code designation unit 4210 exist. In an embodiment of the present invention, the identification code designation unit 4210 is assigned to the two group object areas, respectively, G1-1 as group object area 1 of frame 1 and G1- as meaning of group object area 2 of frame 1, respectively. The identification code of 2 has been assigned. Similarly, the identification code designation unit 4210 assigns identification codes of G2-1 and G2-2 to the two group object regions existing in frame 2, respectively.

In the embodiment of FIG. 17, the parameter derivation unit 4220 directly or indirectly obtains motion vector information of a macroblock constituting the group object area or a subblock constituting a macroblock from the variable length coding unit 1100 ( through the first object area detection unit, etc.). In Fig. 17, motion vectors of macroblocks constituting the group object area are shown in each macroblock.

Each macroblock of the group object area G1-1 has a motion vector in the upper right direction in FIG. 17, and each macroblock in the group object area G1-2 has a motion vector in the lower left direction.

Motion vector information of frame 2 is omitted.

The group clustering unit 4230 clusters the group object area determined to include the same object among the group object areas. To this end, the group clustering unit 4230 configures a group including the same object based on at least one of location information, motion vector information, and prediction error information of a macroblock constituting the group object area or subblock constituting a macroblock. The object area can be identified.

In FIG. 17, in the case of the group object area G2-1, the location information of the constituent macroblocks is similar to that obtained by adding the motion vector of the group object area G1-1 to the location information of the group object area G1-1. That is, in the group clustering unit 4230, the object included in the group object area G2-1 of the frame 2 immediately after the frame 1 moves along the motion vector of the object included in the group object area G1-1 of the frame 1 can be judged to have been When it is determined that the same object is included in this way, the group clustering unit 4230 clusters identification codes G1-1 and G2-1 of the two group object areas and classifies them into the same classification.

In the same way, it can be determined that the group object area G1-2 and the group object area G2-2 also contain the same object, and the identification codes G1-2 and G2-2 of the two group object areas are clustered and classified into the same classification. do.

The identification code divided into the same classification as described above is used to create a vector sequence of group object area information.

18 is a diagram illustrating a configuration of group vector information according to an embodiment of the present invention.

18A shows the group object area grouped by the group object area generating unit 4100. As shown in FIG. The group object area shown in FIG. 18A is a part of a frame of image data, and other macroblocks that are not determined to be object areas are omitted.

The group object area shown in FIG. 18A has a width of 5 and a height of 3, and the coordinates of the macroblock located at the top right of the group object area are (24,425). Motion vector information derived from the variable length coding unit 1100 is displayed in each macroblock.

FIG. 18B shows motion vector information of the group object area shown in FIG. 18A. Most of the motion vectors of each macroblock are directed upward in FIG. 18, but the detailed directions and magnitudes are different. The direction of the motion vector may be classified based on a preset direction criterion in order to indicate the characteristics of the group object region through each motion vector as described above. In (B) of FIG. 18 , the directions were classified into d1, d2 to d8, respectively, according to the criterion of dividing the direction into eight angles. When classified according to the above criteria, each motion vector of the group object region is classified according to the direction criteria of d1, d2, d7, and d8. In the present invention, it is possible to generate a histogram of the direction and magnitude of the motion vector of the group object region by such classification. Such a histogram can be used as the characteristic information of the group object area by using the histogram as a parameter.

18C shows the group object area information expressed as a vector. The group object area information may include an identification code for classifying the group object area, location information indicating a location of the group object area, and parameter information indicating characteristics of the group object area.

In an embodiment of the present invention, the parameter information of the group object area includes: a direction histogram for motion vector information of a macroblock constituting the group object area or a subblock constituting the macroblock; and a size histogram for motion vector information of a macroblock constituting the group object area or a subblock constituting the macroblock. may include.

As shown in (C) of FIG. 18, the group object area information includes the X coordinate, Y coordinate, width and height indicating the location of the group object area of FIG. Similarly, it may be configured in a manner in which information of a direction histogram and a magnitude histogram for a motion vector of each macroblock is listed in order.

19 to 21 are diagrams schematically illustrating the operation of the group object area review unit according to an embodiment of the present invention.

19 and 20 show three consecutive frames of image data according to an embodiment of the present invention. In each frame, the object region derived by the first object region detector 2000 is indicated by a solid line, and the macroblock of the object region is the group object region generator 4100 of the second object region detector 4000 . ), and are clustered by the group clustering unit 4230 of the vector sequence generating unit 4200 . That is, the group object area G1 of frame 1, the group object area G1 of frame 2, and the group object area G1 of frame 3 generated by the group object area generation unit 4100 are the same objects generated by the group clustering unit 4230. It is considered to be included and clustered.

The information of the clustered group object area is expressed as a vector, and sequence data is generated according to the temporal order of the frame. At this time, in the embodiment of FIG. 19 , information on each group object region is expressed as one vector for each frame in the sequence data, and is input to a learned recurrent neural network (RNN).

In this way, sequence data according to time is sequentially input to the recurrent neural network (RNN), so that it is possible to determine whether an object is included in the group object region.

In an embodiment of the present invention, the information of the group object area is received through the recurrent neural network (RNN) to derive a probability that an object is included in the group object area, and the third determining unit 4400 determines the probability. By determining whether the predetermined criterion is satisfied, the second object area information can be derived by deleting or retaining the corresponding group object area from the object area.

To this end, since information of a plurality of group object regions is simultaneously input to the recurrent neural network (RNN), the respective probabilities can be derived by classifying them.

Meanwhile, in another embodiment of the present invention as shown in FIG. 20, information of each group object region is expressed as a vector, and such a vector may be input to a plurality of learned recurrent neural networks RNN1 and RNN2, respectively. .

In such an embodiment, the information of each group object area is input to a separate cyclic neural network, and the probability that the object is included in each group object area is derived, respectively, and the third discrimination unit 4400 is derived from the It may be determined whether the probability satisfies a preset criterion.

21 shows another embodiment of the present invention.

In the embodiment of FIG. 21, three frame groups in which a plurality of consecutive frames are grouped are shown. In the embodiment of FIG. 21, each frame group is grouped into n frames, the first frame group contains frames 1 to n, the second frame group contains frames 2 through n+1, and the third frame group contains 3 frames. Frames from to n+2 are included. In the embodiment of FIG. 21 , the frame group includes n frames equally, but in another embodiment of the present invention, each frame group may include a different number of frames. In addition, in the embodiment of FIG. 21 , one frame is overlapped and included in a plurality of frame groups (when n is 4, frame 2 is in the first and second frame groups, and frame 3 is in the first and second frame groups). included in both the third and third framegroups), in another embodiment of the present invention, one frame may be included in only one framegroup.

In each of the three frame groups shown in FIG. 21, accumulated object areas derived based on each object area derived by the first object area detection unit 2000 from a plurality of frames included in the frame group are shown. . The accumulated object area may be derived based on object areas of frames included in the frame group. In one embodiment of the present invention, the accumulated object area may be an area in which all object areas of the frames are combined, and in another exemplary embodiment, the accumulated object area may be an area derived as an object area more than a preset number of times, and another In an embodiment, the accumulated object area may be an area derived by averaging the object areas of frames. The macroblocks of the cumulative object area are grouped by the group object area generation unit 4100 of the second object area detection unit 4000, and are clustered by the group clustering unit 4230 of the vector sequence generation unit 4200. have. That is, the group object area G1 of the three group frames generated by the group object area generation unit 4100 is determined to include the same object by the group clustering unit 4230 and is clustered.

The information of the clustered group object area is expressed as a vector, and sequence data is generated according to the time order of the group frame. At this time, in the embodiment of FIG. 21 , information on each group object region is expressed as one vector for each group frame in the sequence data and is input to a learned recurrent neural network (RNN).

In this way, sequence data according to time is sequentially input to the recurrent neural network (RNN), so that it is possible to determine whether an object is included in the group object region.

22 exemplarily illustrates an internal configuration of a computing device according to an embodiment of the present invention. All or some of the components illustrated in FIG. 22 may include components of a computing device to be described later.

22, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input/output subsystem ( It may include at least an I/O subsystem) 11400 , a power circuit 11500 , and a communication circuit 11600 .

The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. have. The memory 11200 may include a software module, an instruction set, or other various data necessary for the operation of the computing device 11000 .

In this case, access to the memory 11200 from other components such as the processor 11100 or the peripheral interface 11300 may be controlled by the processor 11100 . The processor 11100 may be configured as a single or plural number, and may include a GPU and a TPU type processor to improve operation processing speed.

Peripheral interface 11300 may couple input and/or output peripherals of computing device 11000 to processor 11100 and memory 11200 . The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

The input/output subsystem 11400 may couple various input/output peripherals to the peripheral interface 11300 . For example, the input/output subsystem 11400 may include a controller for coupling peripheral devices such as a monitor, keyboard, mouse, printer, or a touch screen or sensor as needed to the peripheral interface 11300 . According to another aspect, input/output peripherals may be coupled to peripheral interface 11300 without going through input/output subsystem 11400 .

The power circuit 11500 may supply power to all or some of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or a power source. It may include any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, if necessary, the communication circuit 11600 may transmit and receive an RF signal, also known as an electromagnetic signal, including an RF circuit, thereby enabling communication with other computing devices.

22 is only an example of the computing device 11000, and the computing device 11000 omits some components shown in FIG. 22, or further includes additional components not shown in FIG. 22, or 2 It may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 22 , and various communication methods (WiFi, 3G, LTE) are provided in the communication circuit 1160 . , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented as hardware, software, or a combination of both hardware and software including an integrated circuit specialized for one or more signal processing or applications.

Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in computer-readable media. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the user terminal through a file provided by the file distribution system. As an example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of the user terminal.

According to an embodiment of the present invention, when performing analysis such as recognizing, tracking, or identifying an object from encoded image data, the image analysis is performed only on the detected object area by detecting the object area in advance, so that faster It can have the effect of processing images at high speed.

According to an embodiment of the present invention, even in detecting an object region before image analysis, by selectively removing unnecessary macroblocks, the amount of computation for object region detection is reduced, thereby increasing the computational speed of the system as a whole. can perform

According to an embodiment of the present invention, it is possible to exhibit the effect of detecting an object region determined to exist in which an object may exist with a fairly high level of accuracy despite the minimum amount of computation in object region detection.

According to an embodiment of the present invention, the codec is changed in the case of a codec method using variable size blocks by using parameters generated in the decoding process in the corresponding decoder unit without changing the configuration of the decoder unit for decoding the corresponding image data. Even if it is, an effect that can be easily applied can be exhibited.

According to an embodiment of the present invention, it is possible to exhibit the effect of detecting the object region with a high level of accuracy by first detecting the object region and then examining the reliability of the detected object region based on the artificial neural network.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computing devices, and may be stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

delete delete An object region detection method performed in a computing system including one or more processors and a main memory for storing instructions executable by the processor, the method comprising:
an image decoding step of decoding an image by performing a variable length decoding step, an inverse quantization step, an inverse transform step, and an addition step on the video data;
size information of data for a macroblock included in the image data; and a first object region detecting step of deriving first object region information of an image based on one or more image decoding parameters extracted in the image decoding step; and
a second object region detection step of deriving second object region information by examining the first object region information derived in the first object region detection step through an artificial neural network; including,
The second object region detection step includes:
A group object area generation step of deriving group object area information for a group object area generated by grouping macroblocks or subblocks constituting macroblocks in contact with each other among the first object area information detected in the first object area detection step ;
a vector sequence generation step of generating sequence data according to time of an image by expressing the group object region information as a real vector;
a group object region review step of deriving a probability that an object exists in the group object region by using the sequence data as an input of a recurrent neural network (RNN);
a third discrimination step of detecting a second object region based on whether the derived probability that an object exists in the group object region satisfies a preset criterion; Including, an object area detection method.
4. The method according to claim 3,
The vector sequence generation step is,
an identification code designation step of designating an identification code in the group object area;
a parameter deriving step of deriving an image decoding parameter of a macroblock constituting the group object area or a subblock constituting the macroblock;
a group clustering step of clustering identification codes of a group object region determined to include the same object in successive frames of an image based on the image decoding parameter; and
a group vector information generation step of expressing the clustered information of the group object area as a vector; An object area detection method comprising a.
5. The method according to claim 4,
The video decoding parameters are
motion vector information of a macroblock or subblocks constituting the macroblock; and
prediction error information of a macroblock or subblocks constituting the macroblock; An object area detection method including at least one of
5. The method according to claim 4,
The group clustering step is
An object region detection method for identifying a group object region including the same object based on at least one of position information, motion vector information, and prediction error information of a macroblock constituting the group object region or subblock constituting the macroblock.
4. The method according to claim 3,
The sequence data is
location information of the group object area; and parameter information of the group object area; An object area detection method comprising a.
8. The method of claim 7,
The parameter information of the group object area is,
a direction histogram for motion vector information of a macroblock constituting the group object area or a subblock constituting the macroblock; and
a size histogram for motion vector information of a macroblock constituting the group object area or a subblock constituting a macroblock; An object area detection method comprising a.
An object region detection apparatus comprising one or more processors and a main memory for storing instructions executable by the processor, the apparatus comprising:
an image decoding unit for decoding an image by performing a variable length decoding step, an inverse quantization step, an inverse transform step, and an addition step on the video data;
size information of data for a macroblock included in the image data; and a first object region detection unit for deriving first object region information of an image based on one or more image decoding parameters extracted from the image decoding unit. and
a second object region detection unit for deriving second object region information by examining the first object region information derived from the first object region detection unit through an artificial neural network; including,
The second object area detection unit,
a group object area generator for deriving group object area information for a group object area generated by grouping macroblocks or subblocks constituting macroblocks in contact with each other among the first object area information detected by the first object area detection unit;
a vector sequence generator for generating sequence data according to time of an image by expressing the group object region information as a real vector;
a group object area review unit for deriving a probability that an object exists in the group object area by using the sequence data as an input of a recurrent neural network (RNN);
a third determining unit configured to detect a second object region based on whether the derived probability that an object exists in the group object region satisfies a preset criterion; Including, object area detection device.
A computer program stored on a computer-readable medium comprising a plurality of instructions executed by one or more processors, the computer program comprising:
The computer program is
an image decoding step of decoding an image by performing a variable length decoding step, an inverse quantization step, an inverse transform step, and an addition step on the video data;
size information of data for a macroblock included in the image data; and a first object region detecting step of deriving first object region information of an image based on one or more image decoding parameters extracted in the image decoding step; and
a second object region detection step of deriving second object region information by examining the first object region information derived in the first object region detection step through an artificial neural network; including,
The second object region detection step includes:
A group object area generation step of deriving group object area information for a group object area generated by grouping macroblocks or subblocks constituting macroblocks that are in contact with each other among the first object area information detected in the first object area detection step ;
a vector sequence generating step of generating sequence data according to time of an image by expressing the group object region information as a real vector;
a group object area review step of deriving a probability that an object exists in the group object area by using the sequence data as an input of a recurrent neural network (RNN);
a third discrimination step of detecting a second object region based on whether the derived probability that an object exists in the group object region satisfies a preset criterion; comprising, a computer program.

Images