KR102345258B1 - Object Region Detection Method, Device and Computer Program Thereof - Google Patents

Abstract

The present invention relates to a method, apparatus, and computer program for detecting an object area, and more particularly, by detecting an object area in advance based on a parameter value derived from an image decoding process and referring it to image analysis, The present invention relates to a method and apparatus for detecting an object area capable of further speeding up analysis such as object recognition and tracking, and a computer program therefor.

Description

Object Region Detection Method, Device and Computer Program Thereof

The present invention relates to a method, apparatus, and computer program for detecting an object area, and more particularly, by detecting an object area in advance based on a parameter value derived from an image decoding process and referring it to image analysis, The present invention relates to a method and apparatus for detecting an object area capable of further speeding up analysis such as object recognition and tracking, and a computer program therefor.

Recently, image data collected from smart phones, CCTVs, black boxes, high-definition cameras, and the like are rapidly increasing. Accordingly, the requirements for recognizing a person or object based on atypical image data to extract meaningful information and to visually analyze and utilize the content are increasing.

The image data analysis technology refers to various technologies for performing learning and analysis on these various images to search for a desired image or to recognize a situation such as an event occurrence.

However, since the technique of recognizing, analyzing, and tracking image data is an algorithm that requires a considerable amount of computation, that is, the complexity is high, and the larger the size of the image data, the greater the load on the computing device. Accordingly, it takes a longer time to analyze the image data having an increased size. Therefore, there is a steady demand for a method for reducing the time for analyzing image information.

On the other hand, the image security market is continuously growing as awareness of security has been strengthened due to terrorism, etc. in recent years, and accordingly, the demand for intelligent image processing is also increasing.

Recently, a technology capable of efficiently compressing, transmitting, and verifying a high-resolution image based on a block-based video codec according to a protocol such as H.264 has spread. Such high-resolution images are also applied to monitoring images such as CCTV, but in the analysis and tracking of such high-resolution images, as the resolution of the image increases, the conventional object detection method requires a higher amount of computation, and thus real-time images There was a point that the analysis was not carried out smoothly.

Meanwhile, in Prior Document 1 (Korean Patent Registration No. 10-1409826, registered on June 13, 2014), a motion vector of a reference frame is calculated based on a histogram of the motion vectors of blocks in the reference frame, and reference is made based on the global motion vector. Disclosed is a technique for determining the type of area of a block.

However, in the case of Prior Document 1, since it is necessary to calculate a motion vector for the entire region and calculate histogram data for the entire region, it is difficult to obtain a speed capable of real-time processing in the current high-resolution image, and, Since motion vectors must be considered for all blocks, there is a problem in that an operation must be performed once for unnecessary blocks.

In addition, when only the motion vector is considered as a factor to be considered as in Prior Document 1, it may be difficult to accurately detect the object region. Accordingly, in the case of Prior Document 1, in order to accurately determine the object region, it is necessary to re-calculate the feature quantities inside the image, so there is a problem in that it is practically difficult to quickly and accurately analyze a high-resolution image.

Prior Document 1: Korean Patent Registration No. 10-1409826, 'Motion prediction method using an adaptive search range', registered on June 13, 2014)

It is an object of the present invention to detect an object region in advance based on a parameter value derived from an image decoding process and refer to it for image analysis, thereby further speeding up the analysis of object recognition and tracking. , to provide a device and a computer program therefor.

In order to solve the above problems, in one embodiment of the present invention, there is provided an object region detection method performed in a computing system including one or more processors and a main memory for storing instructions executable by the processor. an image decoding step of decoding an image by performing a length decoding step, an inverse quantization step, an inverse transform step, and an addition step; and size information of data for a block included in the image data. and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step;

In some embodiments of the present invention, the step of detecting the object region includes: extracting size information of block data from the bitstream of the image data before the variable-length decoding step is performed; and determining whether the size information of the data for the block satisfies a preset criterion; deriving the first object area information by; and deriving second object region information based on one or more image decoding parameters extracted in the image decoding step, wherein the object region information is based on the first object region information and the second object region information. can be decided.

In some embodiments of the present invention, in the detecting of the object region, when the image data is encoded into a plurality of macroblocks, and some of the macroblocks include subblocks, the image data contains a macro block as identifier information of a bitstream of the image data. classifying blocks or subblocks and extracting size information of data of blocks corresponding to macroblocks or subblocks therefrom; and determining whether size information of data for a macroblock or subblock not having a subblock satisfies a preset criterion; deriving the first object area information by;

In some embodiments of the present invention, when the block corresponds to a macroblock and the macroblock does not include a subblock, the object region detecting step is based on one or more image decoding parameters for the entire macroblock. to derive second object region information, and when the macroblock includes sub-blocks, the second object region information may be derived based on one or more image decoding parameters for each of the sub-blocks.

In some embodiments of the present invention, in the object region detecting step, first object region information is derived based on size information of bitstream data for a block included in the image data, and in the image decoding step, the variable length The decoding unit detects derivation of a second object region based on motion vector information derived from the decoded information, and the object region information includes information on a region corresponding to the first object region information or the second object region information. can do.

In some embodiments of the present invention, the step of detecting the object region comprises: based on size information of bitstream data for blocks in a plurality of frames in a preset number and motion vector information in a plurality of frames in a preset number Area information can be derived.

In some embodiments of the present invention, the step of detecting the object region includes summing the size information of the bitstream data of each block for each of a plurality of frames of a preset number, and calculating a comprehensive block data size determination value for each block. deriving; and deriving first object area information on the basis of information on blocks in which the comprehensive block data size determination value meets a preset criterion, wherein the object area information includes information including the first object area information. can be determined based on

In some embodiments of the present invention, the size information of the bitstream data of the block may be a normalized value based on total information on the data size in each frame.

In some embodiments of the present invention, the detecting of the object region comprises determining the motion vector of each block when the size of the motion vector of each block for each of the plurality of frames of a predetermined number meets a predetermined criterion. assigning the value as a first numerical value and assigning the motion vector determination value of each block as a second numerical value when the magnitude of the motion vector of each block does not meet a preset criterion; deriving a comprehensive motion vector judgment value for each block based on the motion vector judgment values for each block of the predetermined number of each of the plurality of frames; and deriving second object region information based on information on blocks in which the comprehensive motion vector determination value meets a preset criterion, wherein the object region information includes: can be determined based on

In some embodiments of the present invention, the step of deriving the second object region information includes whether the comprehensive motion vector judgment value of each block meets a preset criterion and is derived according to the direction of the motion vector of each block. The second object region information may be derived based on the grouping information.

In some embodiments of the present invention, the step of deriving and detecting the object region comprises: information on the size of data for blocks in a plurality of frames of a preset number, magnitudes of motion vectors in a plurality of frames in a preset number of frames, and motion vectors. Object area information can be derived based on the direction.

In order to solve the above problems, in one embodiment of the present invention, there is provided an object region detection apparatus including one or more processors and a main memory for storing instructions executable by the processor, comprising: a variable length decoding step for image data; an image decoding unit for decoding an image by performing an inverse quantization step, an inverse transform step, and an addition step; and size information of data for a block included in the image data. and an object region detection unit deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step;

In order to solve the above problems, in one embodiment of the present invention, as a computer program stored in a computer-readable medium comprising a plurality of instructions executed by one or more processors, the computer program, the image data an image decoding step of decoding an image by performing a variable length decoding step, an inverse quantization step, an inverse transform step, and an addition step; and size information of data for a block included in the image data. and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step.

According to an embodiment of the present invention, when performing analysis such as recognizing, tracking, tracking, or identifying an object from encoded image data, since the image analysis is performed only on the detected object area by detecting the object area in advance, The effect of processing the image at a faster speed can be exhibited.

According to an embodiment of the present invention, even in detecting the object area before image analysis, it is possible to reduce the amount of calculation for object area detection, thereby increasing the operation speed of the system as a whole.

According to an embodiment of the present invention, it is possible to exhibit the effect of detecting the object area determined that the object may exist with a fairly high level of accuracy despite the minimum amount of computation in the object area detection.

According to an embodiment of the present invention, H.264, H.265 (HEVC), H.264, H.265 (HEVC), In the case of a codec method using blocks according to a protocol such as H.266 (VVC), even if the codec is changed, an effect that can be easily applied can be exhibited.

According to an embodiment of the present invention, it can be utilized for real-time image analysis, reading, and detection such as high-resolution CCTV image detection by detecting an object area with high accuracy and high speed regardless of environmental factors such as the characteristics of a shooting device or a shooting location. can have an effect.

1 is a diagram schematically showing the overall structure of an object region detection system according to an embodiment of the present invention.
2 is a diagram schematically illustrating a detailed configuration of an object region detection system according to an embodiment of the present invention.
3 is a diagram schematically illustrating a structure before decoding of a data stream of image data according to an embodiment of a video codec using blocks according to a protocol such as H.264.
4 is a diagram schematically illustrating a data field structure of a macroblock of image data according to an embodiment of a video codec using a variable block.
5 is a diagram schematically illustrating a detailed configuration of an object region detection unit according to an embodiment of the present invention.
6 is a diagram illustrating some examples of macroblocks.
7 is a diagram illustrating an example of a macroblock including subblocks.
8 is a block diagram illustrating a process of an object region detection step according to an embodiment of the present invention.
9 is a diagram exemplarily illustrating the operation of the first object area information derivation unit on a block basis according to an embodiment of the present invention.
10 is a diagram exemplarily illustrating the operation of the first object area information derivation unit on a block basis according to an embodiment of the present invention.
11 is a block diagram exemplarily illustrating the operation of the second object area information derivation unit according to an embodiment of the present invention.
12 is a diagram illustrating an example of an image screen and a detailed data processing map according to an operation of the first object area information derivation unit according to an embodiment of the present invention.
13 is a diagram illustrating an example of an image screen and a detailed data processing map according to an operation of a second object region information derivation unit according to an embodiment of the present invention.
14 is a diagram illustrating an example of object area detection according to the object area detection method according to an embodiment of the present invention.
15 schematically shows an encoder system for generating an encoded image according to an embodiment of the present invention.
16 is a diagram schematically illustrating examples of frames of image data.
17 exemplarily shows an internal configuration of a computing device according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS Various embodiments are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In this specification, various descriptions are presented to provide an understanding of the present invention. However, it is apparent that these embodiments may be practiced without these specific descriptions. In other instances, well-known structures and devices are presented in block diagram form in order to facilitate describing the embodiments.

The terms “component,” “module,” “system,” “unit,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and/or thread of execution, and a component may reside within one computer.

It may be localized within, or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored therein. The components may contain, for example, a signal having one or more data packets (eg, data from one component interacting with another component in a local system, a distributed system and/or data via a network such as the Internet with another system via a signal). ) may communicate via local and/or remote processes.

Also, the terms "comprises" and/or "comprising" mean that the feature and/or element is present, but excludes the presence or addition of one or more other features, elements, and/or groups thereof. should be understood as not

Also, terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, are generally understood by those of ordinary skill in the art to which the present invention belongs. have the same meaning as Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in an embodiment of the present invention, an ideal or excessively formal meaning is not interpreted as

1 is a diagram schematically showing the overall structure of an object region detection system according to an embodiment of the present invention.

The object region detection system in FIG. 1 includes an object region detection unit 2000 that processes received image data in a broad sense, an image decoding unit 1000 , and an image analysis unit 3000 . Video data is video data encoded by a standard codec, and preferably corresponds to data encoded by video using a variable size block, and preferably conforms to a protocol including H.264, H.265 codec, etc. Corresponds to the encoded image data. More preferably, it corresponds to video data encoded according to the convention of using a variable size block.

The image data may correspond to an image stored in the object area detection system or image data received by another image collecting device (eg, CCTV or monitoring device) in real time.

The image decoding unit 1000 corresponds to an apparatus for decoding or decoding an image encoded according to a protocol including the H.264 and H.265 codec methods, and is configured according to the decoding method of the corresponding codec.

The image analysis unit 3000 performs analysis such as object recognition, tracking, and identification on the decoded and received image, and the image analysis unit 3000 includes a preprocessor, a feature information extractor, and a feature information analysis according to the purpose of image analysis. It may include various components such as a part.

In the present invention, in order to further improve the image processing speed of the image analysis unit 3000, the object region detection unit 2000 is introduced. The object region detection unit 2000 extracts information extracted from the bitstream of the image data and the undecoded image data bitstream, not the decoded image itself, such as the image decoding parameters extracted in the image decoding process of the image decoding unit 1000 The object region information is detected through the information used and the parameter information extracted in the decoding process, and the object region information is transmitted to the image analysis unit 3000 . The object area information may correspond to coordinate information of a rectangular area or may correspond to atypical area information formed of a plurality of blocks.

Therefore, the image analysis unit 3000 does not perform object recognition, feature point extraction, pre-processing, etc. on the entire image, but performs image analysis only on the area according to the object area information received from the object area detection unit 2000. can

The object region detection unit 2000 , the image decoding unit 1000 , and the image analysis unit 3000 may be configured as a single computing device, or may be configured by two or more computing devices.

That is, the object region detection system according to the present invention may be implemented by a computing system including one or more processors and a main memory for storing instructions executable by the processor.

2 is a diagram schematically illustrating a detailed configuration of an object region detection system according to an embodiment of the present invention.

The object region detection apparatus according to an embodiment of the present invention is implemented by a computing system including one or more processors and a main memory for storing instructions executable by the processor.

As shown in FIG. 2 , an image decoding unit 1000 including a variable length decoding unit 1100 , an inverse quantization unit 1200 , an inverse transform unit 1300 , an adder 1400 , and a prediction unit 1500 . Decodes the encoded image data by the image encoding method.

The configuration of the image decoding unit 1000 may be configured according to the configuration of an encoding unit for encoding image data. For example, the configuration of the image decoding unit 1000 shown in FIG. 2 is in accordance with the configuration for decoding the H.264 codec in the embodiment, but the present invention is not limited thereto, and block-based decoding/encoding is performed. If it is a codec method, it can be applied. For example, even in the case of the H.265 codec, the present invention can be applied.

The variable-length decoding unit 1100 variable-length decodes (decodes) input image data. Through this, the variable length decoding unit 111 may separate the image data into motion vectors, quantization values, and DCT coefficients or extract a motion vector, quantization values, and DCT coefficients from the image data.

The inverse quantization unit 1200 inversely quantizes the DCT coefficient output from the variable length decoding unit 1100 according to the extracted quantization value.

The inverse transform unit 1300 performs inverse transform (IDCT) on the DCT coefficient inverse quantized by the inverse quantizer 1200 to obtain a differential image.

The prediction unit 1500 performs prediction according to whether the corresponding frame is intra mode or inter mode. The motion compensator 1530 of the prediction unit 1500 compensates for the motion of the current image data by using the motion vector and previous image data. Through this, the motion compensator generates a predicted image.

The configuration of the variable-length decoding unit 1100, the inverse quantization unit 1200, the inverse transform unit 1300, the adder 1400, and the prediction unit 1500 of the image decoding unit 1000 is the encoding method of the image data or It may be changed depending on the codec, which a person skilled in the art may implement according to the codec of the corresponding image data. The present invention has the advantage that it can be configured by adding the object region detection unit 2000 to the existing decoding unit for decoding an image according to H.264, H.265, and the like.

The object region detection unit 2000 may include size information of data for a block included in the image data; and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step.

Here, the block may correspond to a block having a variable size or, in the case of using the macroblock method, may correspond to a macroblock or a subblock included in the macroblock.

The thus derived object region information is transmitted to the image analysis unit 3000 , and the image analysis unit 3000 performs image processing on only the object region information, thereby significantly reducing the amount of computation required for image analysis.

3 is a diagram schematically illustrating a structure before decoding of a data stream of image data according to an embodiment of a video codec according to a protocol such as H.264.

The data stream shown in FIG. 3 corresponds to image data that is stored or transmitted without performing any decoding input to the variable length decoding unit 1100 in FIG. 2 . Such image data or data stream consists of a Network Abstraction Layer (NAL), and each NAL consists of a Nal Unit and a Raw Byte Sequence Payload (RBSP) as a payload. The NAL may correspond to a unit in which parameter information such as SPS and PPS is described or a unit in which slice data corresponding to a video coding layer (VCL) is described.

The SLICE NAL corresponding to the VCL consists of a header and data, where the data consists of a plurality of macroblock fields and a delimiter field. In the present invention, the encoding method of image data for performing object region detection is a method of encoding data in the NAL state into macroblocks having a constant block size. In the data field divided into MBs in FIG. 3, data for blocks of a certain size are encoded.

The first object region information derivation unit 2100 of the object region detection unit 2000, which will be described later, may use the data size of each macroblock, that is, the portion indicated by MB in FIG. 3 from image data that has not been decoded.

In the case of video data encoded by the general H.264 codec, in the MB data field of FIG. 3, data for a macroblock having a size of 16 pixels x 16 pixels is encoded and stored. The detailed block information of the macroblock can be checked in partially encoded form.

4 is a diagram schematically illustrating a data field structure of a macroblock of image data according to an embodiment of a video codec using a variable block.

The data field of the macroblock shown in FIG. 4 is decoded by the variable length decoding unit 1100 . Basically, the data field of the macroblock includes a Type field including information such as the size of the block; a prediction type field including information on whether encoded in intra mode or inter mode, and reference frame information and motion vector information in case of inter mode; a Coded Picture Buffer (CPB) field including information for maintaining a bit stream of a previous picture input during decoding; a Quantization Parameter (QP) field including information on a quantization parameter; and a DATA field including information on DCT coefficients for the color of the corresponding block.

When a macroblock includes a plurality of subblocks, a plurality of data units of type-prediction type-CPB-QP-DATA shown in the second column of FIG. 4 are connected. The first object region information derivation unit 2100, which will be described later, may derive the first object region information based on the data size of the bitstream of each subblock. The data size of the bitstream of the subblock may be extracted based on the identifier before variable-length decoding, or may be extracted during or after variable-length decoding is completed.

The second determining unit 2200, which will be described later, determines the block size and the An object region is detected using motion vector information known from the prediction type field.

On the other hand, the color information of the DATA field includes information on a plurality of colors (including color information in the YCbCr system in FIG. 4) in an encoded form. The color information of the DATA field is decoded by the inverse quantization unit 1200 and the inverse transform unit 1300 , and the color value in the original image data and the image decoding unit 1000 in the adder 1400 . Prediction error information corresponding to the difference between the color values predicted by the prediction unit 1500 may be derived.

5 is a diagram schematically illustrating a detailed configuration of an object region detection unit 2000 according to an embodiment of the present invention.

The object region detection unit 2000 may include size information of data for blocks included in the image data; and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step.

extracting, by the object region detection unit 2000, size information of block data from the bitstream of the image data before the variable length decoding step is performed; and a first object area information derivation unit 2100 for deriving first object area information by determining whether the size information of the data for the block satisfies a preset criterion; and a second object region information derivation unit 2200 for deriving second object region information based on one or more image decoding parameters extracted in the image decoding step. and an object region output unit 2300 that determines object region information based on the first object region information and the second object region information, and transmits the object region information to the image analysis unit.

As shown in FIG. 2, the first object region information derivation unit 2100 determines the size of data for each macroblock from the NAL format data stream of image data not decoded by the variable length decoding unit 1100. information can be derived. It is determined whether each macroblock corresponds to the object area based on the data size of each data field indicated by MB in FIG. 3 .

In the image encoding method using a variable size block, in the case of a macroblock in which a complex image is located, the macroblock (16x16) is divided into a plurality of subblocks (8x8, 4x4, etc.) or because it contains various information, the corresponding macroblock is it gets bigger In addition, in the video encoding method such as H.264, when there are frequently occurring values and non-frequently occurring values among macroblock data, a short code is assigned to a frequently occurring value and a long code is assigned to a value that is not. Encoding in a way that reduces the amount of data.

The first object region information derivation unit 2100 uses this characteristic of the encoding method using a variable size block, and the data of each macroblock from the image data before the decoding procedure of the variable length decoding unit 1100 is completed. The size may be derived, and the first object area information may be derived based on this.

In this way, in the present invention, without changing the image decoding unit 1000 , valid data capable of detecting an object region is extracted from the image data input to the image decoding unit 1000 , and the corresponding macroblock is processed by a simple operation. Determines whether there is an object area.

In an embodiment of the present invention, even when not encoded in the form of a separate macroblock/subblock, the first object region information derivation unit 2100 is Object area information can be derived.

When the image data is encoded into a plurality of macroblocks and some of the macroblocks include subblocks, the first object region information deriving unit 2100 performs only the bitstream data size of the macroblocks as described above. However, in one embodiment of the present invention, the first object region information derivation unit 2100 classifies a macroblock or subblock with identifier information of the bitstream of the image data, and corresponds to the macroblock or subblock from this. extracting size information of the bitstream data of the block; and determining whether size information of bitstream data for a macroblock or subblock not having a subblock satisfies a preset criterion;

In an embodiment of the present invention, when the macroblock does not include subblocks, the second object region information derivation unit 2200 determines the object region based on one or more image decoding parameters for the entire macroblock. and, when the macroblock includes subblocks, the object region is detected based on one or more image decoding parameters for each of the subblocks.

The inventor of the present invention has implemented the second object region information derivation unit 2200 using various information such as block size information, motion vector size information, grouping information based on motion vector direction angle, and prediction error information through various simulations. As a result, when detecting the object region in association with the bitstream data size of the block, only motion vector size information is used, or grouping information (or motion vector direction angle information) based on motion vector size information and motion vector direction angle is used. In this case, it was confirmed that the form satisfies both accuracy and operation speed.

In a preferred embodiment of the present invention, the one or more image decoding parameters include motion vector information of a macroblock or a subblock constituting the macroblock. include Each macroblock or subblock of the intermode frame includes motion vector information (direction and magnitude), and preferably, the second object region information derivation unit 2200 provides information on the magnitude of the motion vector among the motion vector information. is used to determine whether the corresponding macroblock or subblock can correspond to the object area or a value related to the determination.

In the case of a region where an object exists, there is a higher possibility of movement than the background region. Meanwhile, each image block of a reference frame (eg, P frame) in an image encoding method using a variable block includes size information of a motion vector with respect to the reference frame. In an embodiment of the present invention, the object region detection is performed by the second object region information derivation unit 2200 using such encoding characteristics. That is, the second object region information derivation unit 2200 determines whether the motion vector of a macroblock or subblock for determining whether an object region is an object region is greater, determines that it is more likely to correspond to an object region, or determines whether an object region is an object region or not. In this case, a higher score is given to a block with a smaller motion vector size.

Here, as shown in FIG. 5 , the motion vector information is derived from information decoded by the variable length decoding unit 1100 . In this way, it is possible to derive a parameter capable of detecting the object region while maintaining the configuration of the image decoding unit 1000 without generating additional feature information for deriving the object region.

That is, in a preferred embodiment of the present invention, the object region detection unit 2000 derives first object region information based on size information of bitstream data for a block included in the image data, and in the image decoding step The variable length decoding unit detects second object region derivation based on motion vector information derived from the decoded information, and the object region information includes the first object region information or a region corresponding to the second object region information. include information. Here, the object region information is derived by ORing the region corresponding to the first object region information and the region corresponding to the second object region information.

6 is a diagram illustrating some examples of macroblocks.

FIG. 6(A) shows a case in which a macroblock consists of one block, FIG. 6(B) shows a case in which it consists of 4 blocks, and FIG. 6(C) shows a case in which it consists of 7 blocks and FIG. 6(D) shows a case of 16 blocks.

As described above, in an embodiment of the present invention, the first object area information derivation unit 2100 determines data size information for each macroblock. In another embodiment of the present invention, the first object area information derivation unit 2100 determines data size information for a macroblock or a subblock included in the macroblock.

In relation to the embodiment in which the first object area information derivation unit 2100 determines the size information of data for each macroblock, as shown in FIG. There is a high possibility that the size of the bitstream data for the macroblock is higher than (A) of 6 . In this case, the first object area information derivation unit determines in a direction in which a macroblock as shown in FIG. 6(D) is more likely to correspond to the object area than a macroblock as shown in FIG. 6(A).

7 is a diagram illustrating an example of a macroblock including subblocks.

In encoding image data using variable size blocks, subblocks in the same macroblock may have different block sizes and motion vectors as shown in FIG. 7 .

Blocks #4, #5, #6, and #7 of FIG. 7 have motion vectors larger than blocks #1, #2, and #3. Accordingly, the second object region information derivation unit 2200 generates blocks #4 and #5. , #6, and #7 can be identified as object areas with a relatively high probability.

8 is a block diagram illustrating a process of an object region detection step according to an embodiment of the present invention.

As described above, in an embodiment of the present invention, the object region detecting step performed by the object region detecting unit 2000 includes the first step based on the size information of bitstream data for a block included in the image data. Deriving object region information, and detecting second object region derivation based on motion vector information derived from information decoded by the variable length decoding unit in the image decoding step, wherein the object region information is the first object region information Alternatively, information on a region corresponding to the second object region information is included.

In an embodiment of the present invention, the first object region information corresponds to information on blocks that meet a predetermined criterion (eg, when the size of the bitstream data is greater than or equal to a predetermined size), or It can be implemented as information about the upper left vertex and the lower right vertex of a rectangle formed according to the boundaries of the blocks to be used.

Similarly, in an embodiment of the present invention, the second object region information includes block(s) in which the motion vector information meets a preset criterion (eg, when the size of the motion vector is greater than or equal to the preset size; or Information on the upper-left and lower-right vertices of a rectangle formed according to the boundary of blocks corresponding to the size of the motion vector or larger and grouped in the direction of the motion vector) etc. can be implemented.

For example, FIG. 8A exemplarily illustrates an area corresponding to the first object area information, and FIG. 8B exemplarily illustrates an area corresponding to the second object area information. In this case, in a preferred embodiment of the present invention, the object region detection unit 2000 selects the region corresponding to (C) of FIG. 8 that may correspond to the first object region information or the second object region information (OR operation) as an object. It can be derived from area information.

As in the present invention, when an object region is detected in consideration of the block's bitstream data size and the block's motion vector size (and direction), as in FIG. The merging matched the real object domain and was able to have the maximum operation speed.

9 is a block diagram illustrating the operation of the first object area information derivation unit 2100 according to an embodiment of the present invention.

In a preferred embodiment of the present invention, the object region detection step includes data size information for blocks in a plurality of consecutive frames of a preset number and motion vector information in a preset number of a plurality of frames. Based on this, object area information is derived. For example, the object region is derived by comprehensively considering the size of bitstream data and motion vector information of blocks of frames for 5, 6, 7, etc.

In an embodiment of the present invention, as shown in FIG. 9, any one frame is selected (preferably I frame) from a plurality of preset number of frames (five in the case of FIG. 9), and the The size (absolute value) of the bitstream data for the corresponding frame is normalized based on the total information on the data size in the corresponding frame. That is, when the second frame is selected in FIG. 9 , the sizes (absolute values) of bitstream data for 7 blocks of the second frame may be S1, S2, S3, S4, S5, S6, and S7. Here, NS1, NS2, NS3, NS4, NS5, NS6, and NS7 corresponding to the normalized data size may be derived based on the information of S1 to S7. An important point here is that the normalization criterion is bitstream data size information in the corresponding frame. Through this process, more accurate object region detection can be performed.

In one embodiment of the present invention, before or after the process of normalizing the size (absolute value) of the bitstream data for each block based on the total information on the data size in the frame, the size of the block or macroblock is A process of extracting only blocks that are preset criteria (eg, top 70%) may be performed. This is to make the calculation faster.

For example, the values of S1 to S7 may be normalized in the following form. In the present invention, various types of known normalization techniques may be used.

S1 S2 S3 S4 S5 S6 S7 Block bitstream data size of the frame (absolute value) 20 40 40 20 50 100 200 Size information of the normalized bitstream data [0 ~ 10] One 2 2 One 2.5 5 10

S1 S2 S3 S4 S5 S6 S7 Block bitstream data size of the frame (absolute value) 2 4 4 2 5 10 20 Size information of the normalized bitstream data [0 ~ 10] One 2 2 One 2.5 5 10

10 is a diagram exemplarily illustrating the operation of the first object area information derivation unit on a block basis according to an embodiment of the present invention.

In a preferred embodiment of the present invention, the object region detection step includes data size information for blocks in a plurality of consecutive frames of a preset number and motion vector information in a preset number of a plurality of frames. Based on this, object area information is derived. For example, the object region is derived by comprehensively considering the size of bitstream data and motion vector information of blocks of frames for 5, 6, 7, etc.

In one embodiment of the present invention, as shown in FIG. 10 , the size (absolute value) of bitstream data for each block in a plurality of preset number of frames (five in the case of FIG. 9) is determined as the data in the frame. Normalize based on full information about size.

That is, for example, the size (absolute value) of bitstream data for 7 blocks of the first frame may be S11, S12, S13, S14, S15, S16, and S17. Here, NS11, NS12, NS13, NS14, NS15, NS16, and NS17 corresponding to the normalized data size may be derived based on the information of S11 to S17. An important point here is that the normalization criterion is bitstream data size information in the corresponding frame. Through this process, more accurate object region detection can be performed. As the normalization process, the method described with reference to FIG. 9 may be employed.

In this way, it is possible to derive five sets of information including normalized size information of the bitstream data size for each block of a plurality (5) frames as in column 2 of FIG. 10 .

Thereafter, as in column 3 of FIG. 10 , final size information of each block may be derived by summing the normalized size information of each block in 5 sets.

For example, the form of AS1 = f (NS11, NS21, NS31, NS41, NS51, NS61, NS71) or simply NS11+NS21+NS31+NS41+NS51+NS61+NS71 of the frame shown in column 3 of FIG. can be

Thereafter, the first object area information may be derived based on a block that meets a preset criterion (eg, greater than or greater than a specific value) in the comprehensive block data size determination value of each block such as AS1 to AS7.

That is, the detecting of the object region includes: deriving a comprehensive block data size determination value for each block based on the size information of the bitstream data of each block for each of a plurality of frames of a predetermined number; and deriving first object area information based on information on blocks in which the comprehensive block data size determination value meets a preset criterion, wherein the object area information includes the first object area information is determined based on

Preferably, the size information of the bitstream data of the block is a normalized value based on total information on the data size in each frame.

11 is a block diagram exemplarily illustrating the operation of the second object area information derivation unit according to an embodiment of the present invention.

In a preferred embodiment of the present invention, in the object region detection step performed by the second object region information derivation unit 2200, the size of the motion vector of each block for each of the plurality of frames of a preset number is a preset reference. , the motion vector judgment value of each block is given as a first value, and when the size of the motion vector of each block does not meet the preset criterion, the motion vector judgment value of each block is added The step of assigning a number to 2; is performed.

For example, for each block of 5 frames, if the size of the corresponding motion vector is a preset standard (eg, greater than a specific value), 2 is assigned to the corresponding block. Assuming that 0 is assigned to , it is possible to derive five motion vector judgment value maps including motion vector judgment values for 5 frames as in column 1 of FIG. 11 .

Thereafter, the step of deriving a comprehensive motion vector determination value for each block based on the motion vector determination values for each block of each of the plurality of frames in the predetermined number; is performed.

For example, in the case of summing the motion vector judgment values in a plurality of frames of each block, the total motion vector judgment value is 10 (2+2+2+2+2) in the case of the block at the (2, 2) position. can be derived from On the other hand, in the case of the block at the position (3, 3), the overall motion vector judgment value can be derived as 8 (2+2+2+2).

Thereafter, the step of deriving second object region information based on the information of the blocks in which the comprehensive motion vector determination value meets a preset criterion; is performed. For example, when the preset criterion is 8 or more, (2,2), (2,3), (3,2), (3,3) in FIG. 11 is a block according to the second object area information. This may be the case, and the final object area information may be determined based on information including the second object area information.

In another embodiment of the present invention, the step of deriving the second object region information includes whether the comprehensive motion vector determination value of each block meets a preset criterion and is derived according to the direction of the motion vector of each block. The second object area information may be derived based on the grouping information. As an example, based on information on grouped blocks in which a difference in the direction of a motion vector is within a preset range with respect to blocks having a motion vector having a size greater than or equal to a preset size through the process as in FIG. 11 , the second object Area information may be derived.

That is, the object region deriving and detecting step includes the object region based on size information of blocks in a plurality of frames of a preset number, and magnitudes and directions of motion vectors in a preset number of a plurality of frames. derive information

In addition, in one embodiment of the present invention, in the object region detection step (more specifically, the step performed by the second object region detection unit 2200), the size of the motion vector; Alternatively, noise in the object region may be removed by additionally applying a known filter, etc. to a region derived in consideration of the magnitude and direction of the motion vector.

12 is a diagram illustrating an example of an image screen and a detailed data processing map according to an operation of the first object area information derivation unit according to an embodiment of the present invention.

The upper left photo of FIG. 12 shows an exemplary frame image.

The upper right image of FIG. 12 shows an image map for blocks whose macroblock size is in the upper 70%.

The lower left image of FIG. 12 shows an image map for information summing the normalized bitstream data sizes of macroblocks of 7 frames.

The lower right image of FIG. 12 shows an image map for a block whose final size information of the block described with reference to FIG. 10 is equal to or greater than a preset standard.

13 is a diagram illustrating an example of an image screen and a detailed data processing map according to an operation of a second object region information derivation unit according to an embodiment of the present invention, and FIG. 14 is an object region detection according to an embodiment of the present invention. It is a diagram showing an example of object area detection according to the method.

15 schematically shows an encoder system for generating an encoded image according to an embodiment of the present invention.

The object region detecting unit and the image decoding unit according to an embodiment of the present invention described above may be applied to image data encoded using a variable size block as shown in FIG. 6 . As a representative example, it may be applied to image data encoded by the H.264 codec.

The encoder 10 shown in FIG. 15 includes a DCT unit (Discrete Cosine Transform) 11, a quantization unit 12, and an inverse quantization unit to generate data shown as image data in FIGS. 1 and 2 . (Inverse Quantization; IQ) 13, Inverse Discrete Cosine Transform (IDCT) 14, Frame Memory 15, Motion Estimation and Compensation (ME/MC) 16 and Variable It may include a Variable Length Coding (VLC) 17 . Likewise, it is preferable that the image decoding unit is configured to correspond to the configuration of the encoding unit.

To briefly explain this, the DCT unit 11 performs a DCT operation on image data input in units of pixel blocks of a preset size (eg, 4×4) in order to remove spatial correlation.

Thereafter, the quantization unit 12 quantizes the DCT coefficients obtained by the DCT unit 11 and expresses them as several representative values, thereby performing high-efficiency lossy compression.

Also, the inverse quantizer 13 inverse quantizes the image data quantized by the quantizer 12 .

The inverse transform unit 14 performs IDCT transformation on the image data inversely quantized by the inverse quantizer 13 .

The frame memory 15 stores the IDCT-transformed image data by the inverse transform unit 14 in units of frames.

Meanwhile, the motion estimator and compensator 16 estimates a motion vector (MV) per macroblock using the input image data of the current frame and the image data of the previous frame stored in the frame memory unit 15 to block the block. Calculate the sum of absolute difference (SAD) corresponding to the blockmatching error.

The variable length coding unit 17 removes statistical redundancy from the DCT and quantized data according to the motion vector estimated by the motion estimation and compensation unit 16 .

16 is a diagram schematically illustrating examples of frames of image data.

The video portion of a typical moving picture includes I frames (frames denoted by “I” in Fig. 16), P frames (frames denoted as “P” in Fig. 16), and B frames (frames denoted by “B” in Fig. 16). ) is composed of

I-frame is a key frame that includes all images, can function as an access point in a moving picture file, corresponds to an independently encoded frame, and has a low compression rate.

On the other hand, in the case of a P frame, as a frame generated by forward prediction with reference to a previous I frame or P frame, it does not correspond to an independently encoded frame. Such a P frame has a higher compression ratio than the I frame. Here, the "previous" frame means one of a plurality of frames that exist before the frame as well as the immediately preceding frame, and the "next" frame refers to not only the next frame but also one of a plurality of frames existing after the corresponding frame. means one

On the other hand, in the case of the B frame, as a frame generated by forward and backward prediction with reference to the previous frame and the subsequent frame, it does not correspond to an independently encoded frame. Such B frames have a higher compression ratio than I and P frames. Accordingly, the independently encoded frame may correspond to an I frame, and the non-independently encoded frame may correspond to the remaining B frames or P frames.

The B and P frames correspond to reference frames, and preferably, the object region detection unit detects the object region with respect to the reference frames.

17 exemplarily shows an internal configuration of a computing device according to an embodiment of the present invention. All or some of the components illustrated in FIG. 2 may include components of a computing device to be described later.

17, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input/output subsystem ( I/O subsystem) 11400 , a power circuit 11500 and a communication circuit 11600 may be included at least.

The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. have. The memory 11200 may include a software module, an instruction set, or other various data required for the operation of the computing device 11000 .

In this case, access to the memory 11200 from other components such as the processor 11100 or the peripheral device interface 11300 may be controlled by the processor 11100 . The processor 11100 may be configured as a single or plural number, and may include a GPU and a TPU type processor in order to improve the processing speed.

Peripheral interface 11300 may couple input and/or output peripherals of computing device 11000 to processor 11100 and memory 11200 . The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

The input/output subsystem 11400 may couple various input/output peripherals to the peripheral interface 11300 . For example, the input/output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor, keyboard, mouse, printer, or a touch screen or sensor as required to the peripheral interface 11300 . According to another aspect, input/output peripherals may be coupled to peripheral interface 11300 without going through input/output subsystem 11400 .

The power circuit 11500 may supply power to all or some of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or a power source. It may include any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, if necessary, the communication circuit 11600 may transmit and receive an RF signal, also known as an electromagnetic signal, including an RF circuit, thereby enabling communication with other computing devices.

This embodiment of FIG. 17 is only an example of the computing device 11000, and the computing device 11000 may omit some components shown in FIG. 17 or further include additional components not shown in FIG. 17, or 2 It may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 17 , and various communication methods (WiFi, 3G, LTE) are provided in the communication circuit 1160 . , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented as hardware, software, or a combination of both hardware and software including an integrated circuit specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the user terminal through a file provided by the file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file in response to a request from the user terminal.

According to an embodiment of the present invention, when performing analysis such as recognizing, tracking, tracking, or identifying an object from encoded image data, since the image analysis is performed only on the detected object area by detecting the object area in advance, The effect of processing the image at a faster speed can be exhibited.

According to an embodiment of the present invention, even in detecting the object area before image analysis, it is possible to reduce the amount of calculation for object area detection, thereby increasing the operation speed of the system as a whole.

According to an embodiment of the present invention, it is possible to exhibit the effect of detecting the object area determined that the object may exist with a fairly high level of accuracy despite the minimum amount of computation in the object area detection.

According to an embodiment of the present invention, H.264, H.265 (HEVC), H.264, H.265 (HEVC), In the case of a codec method using blocks according to a protocol such as H.266 (VVC), even if the codec is changed, an effect that can be easily applied can be exhibited.

According to an embodiment of the present invention, it can be utilized for real-time image analysis, reading, and detection such as high-resolution CCTV image detection by detecting an object area with high accuracy and high speed regardless of environmental factors such as the characteristics of a shooting device or a shooting location. can have an effect.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computing devices, and may be stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (13)

An object region detection method performed in a computing system including one or more processors and a main memory for storing instructions executable by the processor, the method comprising:
an image decoding step of decoding an image by performing a variable length decoding step, an inverse quantization step, an inverse transform step, and an addition step on the video data; and
size information of data for a block included in the image data; and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step;
The object region detection step includes:
extracting size information of block data from the bitstream of image data before the variable length decoding step is performed; and determining whether the size information of the data for the block satisfies a preset criterion; deriving the first object area information by;
Deriving second object region information based on one or more image decoding parameters including motion vector information extracted in the image decoding step;
the object area information is determined based on the first object area information and the second object area information;
The object region detection step includes:
deriving a comprehensive block data size determination value for each block by summing the size information of the bitstream data of each block for each of a plurality of frames of a predetermined number; and
and deriving first object area information based on the block information in which the comprehensive block data size determination value meets a preset criterion,
The size information of the bitstream data of the block is a value normalized based on total information on the data size in each frame.
delete The method according to claim 1,
The object region detection step includes:
When the image data is encoded into a plurality of macroblocks, and some of the macroblocks include subblocks,
Separating a macroblock or subblock with identifier information of a bitstream of the image data, and extracting size information of data of a block corresponding to the macroblock or subblock from this; and
Deriving first object area information by determining whether size information of data for a macroblock or subblock not having a subblock satisfies a preset criterion; .
The method according to claim 1,
The object region detection step includes:
When the block corresponds to a macroblock and the macroblock does not include a subblock, the second object region information is derived based on one or more image decoding parameters for the entire macroblock,
and deriving second object region information based on one or more image decoding parameters for each of the subblocks when the macroblock includes subblocks.
delete delete delete delete The method according to claim 1,
The object region detection step includes:
When the size of the motion vector of each block for each of the plurality of frames of the predetermined number meets the predetermined criterion, the motion vector determination value of each block is given as a first value, and the motion vector of each block assigning a motion vector determination value of each block as a second value when the size of n does not meet a preset criterion;
deriving a comprehensive motion vector judgment value for each block based on the motion vector judgment values for each block of the predetermined number of each of the plurality of frames; and
and deriving second object area information based on the information on blocks in which the comprehensive motion vector determination value meets a preset criterion;
and the object region information is determined based on information including the second object region information.
10. The method of claim 9,
The step of deriving the second object region information includes whether the comprehensive motion vector determination value of each block meets a preset criterion and the grouping information derived according to the direction of the motion vector of each block. An object area detection method for deriving area information.
delete An object region detection apparatus comprising one or more processors and a main memory for storing instructions executable by the processor, the apparatus comprising:
an image decoding unit for decoding an image by performing variable length decoding, inverse quantization, inverse transformation, and addition on the image data; and
size information of data for a block included in the image data; and an object region detection unit for deriving object region information of an image based on one or more image decoding parameters extracted from the image decoding unit;
The object area detection unit,
extracting size information of block data from the bitstream of image data before the variable length decoding step is performed; and determining whether the size information of the data for the block satisfies a preset criterion; deriving the first object area information by;
deriving second object region information based on one or more image decoding parameters including motion vector information extracted from the image decoding unit;
the object area information is determined based on the first object area information and the second object area information;
The object area detection unit,
deriving a comprehensive block data size determination value for each block by summing the size information of the bitstream data of each block for each of a plurality of frames of a predetermined number; and
performing the step of deriving the first object area information based on the information of the block in which the comprehensive block data size determination value meets a preset criterion;
The size information of the bitstream data of the block is a value normalized based on total information on the data size in each frame.
A computer program stored on a computer-readable medium comprising a plurality of instructions executed by one or more processors, the computer program comprising:
The computer program is
an image decoding step of decoding an image by performing a variable length decoding step, an inverse quantization step, an inverse transform step, and an addition step on the video data; and
size information of data for a block included in the image data; and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step;
The object region detection step includes:
extracting size information of block data from the bitstream of the image data before the variable length decoding step is performed; and determining whether the size information of the data for the block satisfies a preset criterion; deriving the first object area information by;
Deriving second object region information based on one or more image decoding parameters including motion vector information extracted in the image decoding step;
the object area information is determined based on the first object area information and the second object area information;
The object region detection step includes:
deriving a comprehensive block data size determination value for each block by summing the size information of the bitstream data of each block for each of a plurality of frames of a predetermined number; and
and deriving first object area information based on the block information in which the comprehensive block data size determination value meets a preset criterion,
The size information of the bitstream data of the block is a value normalized based on total information on the data size in each frame.

Images