KR20230040265A - Object Region Detection Method, Device and Computer Program For Traffic Information and Pedestrian Information Thereof - Google Patents

Abstract

The present invention relates to a traffic information and pedestrian information analysis method, device and computer program for the same and, more specifically, to a traffic information and pedestrian information analysis method using object region information detected based on parameter values derived in an image decoding process and relatively few computing resources to improve analysis accuracy, device and computer program for the same. The method includes a motion frame determination step; a first object derivation step; a comparison object area derivation step; a second object derivation step; and an analysis step.

Description

Traffic information and pedestrian information analysis method, device and computer program for this {Object Region Detection Method, Device and Computer Program For Traffic Information and Pedestrian Information Thereof}

The present invention relates to a method, apparatus, and computer program for analyzing traffic information and walking information, and more particularly, to a method using object area information detected based on a parameter value derived in an image decoding process, and relatively small computing resources. It relates to a method, device, and computer program for analyzing traffic information and walking information that can improve analysis accuracy by using

Recently, image data collected from smart phones, CCTVs, black boxes, high-definition cameras, and the like are rapidly increasing. Accordingly, requirements for extracting meaningful information by recognizing people or objects based on unstructured image data and visually analyzing and utilizing the contents are increasing.

Image data analysis technology refers to various technologies for situational awareness such as searching for a desired image or occurrence of an event by performing learning and analysis on these various images.

However, since the technology for recognizing, analyzing, and tracking image data is an algorithm that requires a significant amount of computation, that is, as the size of image data increases, a considerable load is placed on the computing device. Accordingly, it takes longer and longer to analyze image data whose size has increased. Accordingly, there is a steady demand for a method capable of reducing image information analysis time.

On the other hand, as awareness of security has been strengthened due to terrorism in recent years, the video security market has been continuously growing, and accordingly, the demand for intelligent image processing is also increasing.

Recently, a technology capable of efficiently compressing, transmitting, and verifying a high-resolution image based on a block-based video codec according to rules such as H.264 has spread. Such high-resolution images are also applied to monitoring images such as CCTV, but in analysis and tracking of such high-resolution images, as the resolution of images increases, conventional object detection methods require a higher amount of computation, and thus real-time images There was a point that the analysis of was not carried out smoothly.

Meanwhile, Prior Document 1 (Korean Patent Registration No. 10-1409826, registered on June 13, 2014) calculates a motion vector of a reference frame based on a histogram of motion vectors of blocks in a reference frame, and references it based on the global motion vector. A technique for determining the area type of a block is disclosed.

However, in the case of Prior Document 1, since motion vectors must be calculated for the entire region and histogram data must be calculated for the entire region, it is difficult to obtain a speed capable of real-time processing in the current high-resolution image, and also, Since motion vectors must be considered for all blocks, there is a problem in that calculations must be performed once for unnecessary blocks.

In addition, when only the motion vector is considered as a factor to be considered as in Prior Document 1, it may be difficult to accurately detect the object region. Therefore, in the case of Prior Document 1, in order to accurately determine the object region, since feature quantities inside the image must be recalculated, there is a problem in that it is practically difficult to quickly and accurately analyze a high-resolution image.

Prior Document 1: Korean Patent Registration No. 10-1409826, 'Motion prediction method using adaptive search range', registered on June 13, 2014)

The present invention relates to a method, apparatus, and computer program for analyzing traffic information and walking information, and more particularly, to a method using object area information detected based on a parameter value derived in an image decoding process, and relatively small computing resources. It relates to a method, device, and computer program for analyzing traffic information and walking information that can improve analysis accuracy by using

In order to solve the above problems, in one embodiment of the present invention, a method for analyzing traffic or walking information performed in a computing system including one or more processors and a main memory for storing instructions executable by the processor, image data Motion frame determination for determining whether the target frame is a frame with motion based on size information or image decoding parameters of data for a block in a state in which decoding is not performed on the first analysis frame including the target frame of step; a first object derivation step of decoding the target frame, extracting a decoded image, and deriving one or more first objects by performing object detection using a deep learning-based first machine learning model on the decoded image; a comparison object region derivation step of detecting one or more comparison object regions inside the target frame based on size information of block data or image decoding parameters with respect to the second analysis frame including the target frame of the image data; and a second method for deriving one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas. Object derivation step; and an analysis step of determining the first object and the second object as vehicle or pedestrian objects, performing tracking, and analyzing traffic or walking information.

In some embodiments of the present invention, the comparison object region detecting step compares the target frame and blocks in frames preceding one or more target frames based on at least one of size information and motion vector information of bitstream data. The object area can be detected.

In some embodiments of the present invention, the comparison object region detecting step may include summing size information of bitstream data of each block with respect to the target frame and frames preceding one or more target frames to form a comprehensive block for each block. Deriving a data size judgment value; and deriving a comparison object area based on information of blocks whose comprehensive block data size determination value meets a predetermined criterion.

In some embodiments of the present invention, the size information of the bitstream data of the block may be a normalized value based on overall information on the data size in each frame.

In some embodiments of the present invention, the comparison object region detection step may include, when the size of a motion vector of each block meets a predetermined criterion for each of the target frame and one or more frames preceding the target frame, respectively. assigning a motion vector judgment value of each block as a first numerical value, and assigning a motion vector judgment value of each block as a second numerical value when the magnitude of the motion vector of each block does not meet a predetermined criterion; deriving a comprehensive motion vector judgment value for each block by accumulating the motion vector judgment values for each block of each target frame and frames preceding one or more target frames; and deriving, as a comparison object area, blocks whose comprehensive motion vector judgment value meets a predetermined criterion.

In some embodiments of the present invention, the comparison object area detection step is based on whether the comprehensive motion vector judgment value of each block meets a predetermined criterion and grouping information derived according to the direction of the motion vector of each block. Second object area information may be derived.

In some embodiments of the present invention, in the first object derivation step, the decoded image input to the first machine learning model includes an image in which the original image is compressed, In the second object derivation step, the decoded image of the comparison object area input to the second machine learning model is compressed to a higher quality than the original image or an image in which the original image input in the first object derivation step is compressed. compressed images may be included.

In some embodiments of the present invention, the second machine learning model requires a relatively higher computational load than the first machine learning model, and can perform more accurate detection.

In order to solve the above problems, in one embodiment of the present invention, a traffic or walking information analysis system implemented as a computing system including one or more processors and a main memory for storing instructions executable by the processor, image data Motion frame determination for determining whether the target frame is a frame with motion based on size information or image decoding parameters of data for a block in a state in which decoding is not performed on the first analysis frame including the target frame of wealth; a first object region derivation unit that decodes the target frame, extracts a decoded image, and derives one or more first objects by performing object detection on the decoded image by a first machine learning model based on deep learning; a comparison object area deriving unit for detecting one or more comparison object areas inside the target frame based on size information of data or image decoding parameters for a second analysis frame including the target frame of the image data; and a second method for deriving one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas. object area extraction unit; and an analyzer configured to determine the first object and the second object as vehicle or pedestrian objects, perform tracking, and analyze traffic or walking information.

In order to solve the above problems, in one embodiment of the present invention, as a computer program stored in a computer-readable medium, including a plurality of instructions executed by one or more processors, the computer program, the image data A motion frame determination step of determining whether the target frame is a frame with motion based on size information of block data or image decoding parameters in a state in which decoding is not performed on the first analysis frame including the target frame. ; a first object derivation step of decoding the target frame, extracting a decoded image, and deriving one or more first objects by performing object detection using a deep learning-based first machine learning model on the decoded image; a comparison object region derivation step of detecting one or more comparison object regions inside the target frame based on size information of block data or image decoding parameters with respect to the second analysis frame including the target frame of the image data; and a second method for deriving one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas. Object derivation step; and an analysis step of determining the first object and the second object as vehicle or pedestrian objects, performing tracking, and analyzing traffic or walking information.

According to an embodiment of the present invention, traffic information and pedestrian information analysis capable of improving analysis accuracy using relatively small computing resources using object area information detected based on parameter values derived in an image decoding process. It is possible to exert an effect of providing a method, an apparatus, and a computer program for the same.

According to an embodiment of the present invention, it is possible to perform frame-by-frame analysis in order to increase the accuracy of analysis of traffic information or walking information, and simultaneously verify object detection based on decoding parameters.

According to an embodiment of the present invention, an error of a high-speed deep learning-based object detection model based on decoded image information can be corrected based on decoding parameters.

1 is a diagram schematically showing the overall structure of an object area detection system according to an embodiment of the present invention.
2 is a diagram schematically showing a detailed configuration of an object area detection system according to an embodiment of the present invention.
FIG. 3 is a diagram schematically illustrating a structure before decoding of a data stream of image data according to an embodiment of a video codec using blocks conforming to H.264 and the like.
4 is a diagram schematically illustrating a data field structure of a macroblock of image data according to an embodiment of a video codec using variable blocks.
5 is a diagram schematically illustrating a detailed configuration of an object area detection unit according to an embodiment of the present invention.
6 is a diagram showing several examples of macroblocks.
7 is a diagram illustrating an example of a macroblock including subblocks.
8 is a block-based diagram illustrating a process of detecting an object area according to an embodiment of the present invention.
9 is a diagram exemplarily showing the operation of the first object area information derivation unit on a block basis according to an embodiment of the present invention.
10 is a diagram exemplarily illustrating the operation of the first object area information derivation unit on a block basis according to an embodiment of the present invention.
11 is a diagram exemplarily showing the operation of the second object area information derivation unit on a block basis according to an embodiment of the present invention.
12 is a diagram showing an example of a video screen and a detailed data processing map according to the operation of the first object area information deriving unit according to an embodiment of the present invention.
13 is a diagram showing an example of a video screen and a detailed data processing map according to the operation of the second object area information deriving unit according to an embodiment of the present invention.
14 is a diagram illustrating an example of object area detection according to an object area detection method according to an embodiment of the present invention.
15 schematically illustrates the entire steps of a method for analyzing traffic information or walking information according to an embodiment of the present invention.
16 illustrates frames used for detection of a comparison object area according to an embodiment of the present invention.
17 exemplarily illustrates an object detection process according to an embodiment of the present invention.
18 exemplarily illustrates an object detection process according to an embodiment of the present invention.
19 exemplarily illustrates an object detection process according to an embodiment of the present invention.
20 schematically illustrates operations of a first machine learning model and a second machine learning model according to an embodiment of the present invention.
21 schematically illustrates operations of a first machine learning model and a second machine learning model according to an embodiment of the present invention.
22 schematically illustrates the internal configuration of an analysis system for traffic information or walking information according to an embodiment of the present invention.
23 schematically illustrates an encoder system for generating an encoded image according to an embodiment of the present invention.
24 is a diagram schematically illustrating examples of frames of video data;
25 illustratively illustrates the internal configuration of a computing device according to an embodiment of the present invention.

Various embodiments are now described with reference to the drawings, wherein like reference numbers are used throughout the drawings to indicate like elements. In this specification, various descriptions are presented to provide an understanding of the present invention. However, it is apparent that these embodiments may be practiced without these specific details. In other instances, well-known structures and devices are presented in block diagram form in order to facilitate describing embodiments.

As used herein, the terms “component”, “module”, “system”, “unit”, and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution, and a component may reside on a computer

It can be localized within, or distributed between two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. Components may for example signal with one or more packets of data (e.g. data and/or signals from one component interacting with another component in a local system, distributed system to other systems and data over a network such as the Internet). ) to communicate via local and/or remote processes.

Also, the terms "comprises" and/or "comprising" mean that the feature and/or element is present, but excludes the presence or addition of one or more other features, elements and/or groups thereof. It should be understood that it does not.

In addition, terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, are generally understood by those of ordinary skill in the art to which the present invention belongs. has the same meaning as Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the embodiments of the present invention, an ideal or excessively formal meaning not be interpreted as

1. Object area detection algorithm based on decoding parameters

1 is a diagram schematically showing the overall structure of an object area detection system according to an embodiment of the present invention.

The object region detection system in FIG. 1 includes an object region detection unit 2000 processing received image data in a broad sense, an image decoding unit 1000, and an image analysis unit 3000. Video data is video data encoded by a standardized codec, and preferably corresponds to data encoded using a variable size block, and preferably complies with standards including H.264 and H.265 codecs. Corresponds to video data encoded according to More preferably, it corresponds to video data encoded according to a convention using a variable size block.

The image data may correspond to images stored in the object region detection system or image data received in real time by another image collection device (eg, CCTV or monitoring device).

The image decoding unit 1000 corresponds to a device for decoding or decoding an image encoded according to a convention including an H.264 or H.265 codec, and is configured according to a decoding method of a corresponding codec.

The image analysis unit 3000 performs analysis such as object recognition, tracking, and identification on the decoded and received image, and the image analysis unit 3000 performs a pre-processing unit, a feature information extraction unit, and a feature information analysis according to the purpose of image analysis. It may include various configurations such as parts.

In the present invention, in order to further improve the image processing speed of the image analysis unit 3000, the object area detection unit 2000 is introduced. The object region detector 2000 extracts information extracted from the bitstream of the image data and the image decoding parameters extracted in the image decoding process of the image decoding unit 1000, not from the decoded image itself but from the undecoded image data bitstream. Object region information is detected through the information and parameter information extracted in the decoding process, and the object region information is transmitted to the image analysis unit 3000. The object area information may correspond to coordinate information of a rectangular area or non-standard area information composed of a plurality of blocks.

Therefore, the image analysis unit 3000 does not perform object recognition, feature point extraction, preprocessing, etc. for the entire image, but performs image analysis only for the region according to the object region information received from the object region detection unit 2000. can

The object area detection unit 2000, the image decoding unit 1000, and the image analysis unit 3000 may be configured as a single computing device, or may be configured by two or more computing devices.

That is, the object region detection system according to the present invention may be implemented by a computing system including one or more processors and a main memory storing instructions executable by the processor.

2 is a diagram schematically showing a detailed configuration of an object area detection system according to an embodiment of the present invention.

An object area detection apparatus according to an embodiment of the present invention is implemented by a computing system including one or more processors and a main memory storing instructions executable by the processor.

As shown in FIG. 2, an image decoding unit 1000 including a variable length decoding unit 1100, an inverse quantization unit 1200, an inverse transform unit 1300, an adder 1400, and a prediction unit 1500 Decodes the video data encoded by the video encoding method.

The configuration of the video decoding unit 1000 may be configured according to the configuration of an encoding unit that encodes video data. For example, the configuration of the video decoding unit 1000 shown in FIG. 2 is a configuration according to the configuration for decoding the H.264 codec in the embodiment, but the present invention is not limited thereto, and the configuration for performing block-based decoding/encoding Any codec method can be applied. For example, the present invention can also be applied to the H.265 codec.

The variable length decoding unit 1100 performs variable length decoding (decoding) of input video data. Through this, the variable length decoding unit 111 can separate the video data into motion vectors, quantization values, and DCT coefficients, or extract motion vectors, quantization values, and DCT coefficients from the video data.

The inverse quantization unit 1200 inversely quantizes the DCT coefficient output from the variable length decoding unit 1100 according to the extracted quantization value.

The inverse transform unit 1300 performs an inverse transform (IDCT) on the DCT coefficients inversely quantized by the inverse quantization unit 1200 to obtain a difference value image.

The prediction unit 1500 performs prediction according to whether a corresponding frame is an intra mode or an inter mode. The motion compensation unit 1530 of the prediction unit 1500 compensates for motion of current image data by using a motion vector and previous image data. Through this, the motion compensator generates a predicted image.

The configuration of the variable length decoding unit 1100, the inverse quantization unit 1200, the inverse transform unit 1300, the adder 1400, and the prediction unit 1500 of the video decoding unit 1000 is based on the encoding method of video data or It can be changed according to the codec, and this can be implemented by a person skilled in the art according to the codec of the corresponding image data. The present invention has an advantage in that it can be configured by adding an object region detection unit 2000 to an existing decoding unit that decodes video according to H.264, H.265, and the like.

The object area detection unit 2000 includes size information of data for blocks included in the image data; and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step.

Here, the block may correspond to a variable-sized block, or in the case of using a macroblock method, it may correspond to a macroblock or a subblock included in a macroblock.

The object region information derived in this way is transferred to the image analysis unit 3000, and the image analysis unit 3000 processes images only for the object region information, thereby significantly reducing the amount of computation required for image analysis.

FIG. 3 is a diagram schematically illustrating a structure before decoding of a data stream of video data according to an embodiment of a video codec conforming to a standard such as H.264.

The data stream shown in FIG. 3 corresponds to image data that is stored or transmitted without any decoding being input to the variable length decoding unit 1100 in FIG. 2 . Such video data or data stream is composed of NAL (Network Abstraction Layer), and each NAL is composed of Nal Unit and RBSP (Raw Byte Sequence Payload) as a payload. NAL may correspond to a unit in which parameter information such as SPS and PPS is described, or a unit in which slice data corresponding to VCL (Video Coding Layer) is described.

SLICE NAL corresponding to VCL consists of a header and data, where the data consists of a plurality of macroblock fields and a delimiter field. In the present invention, an encoding method of image data for performing object region detection is a method of encoding NAL state data into a macroblock having a certain block size. In FIG. 3, data fields divided into MBs encode data for blocks of a certain size.

The first object region information deriving unit 2100 of the object region detecting unit 2000 described later may use the data size of each macroblock, that is, the portion indicated by MB in FIG. 3 from image data on which decoding is not performed.

In the case of video data encoded with a general H.264 codec, in the MB data field of FIG. 3, data for a macroblock having a size of 16 pixels x 16 pixels is encoded and stored. Detailed block information of a macroblock can be checked in a partially encoded form.

4 is a diagram schematically illustrating a data field structure of a macroblock of image data according to an embodiment of a video codec using variable blocks.

The data field of the macroblock shown in FIG. 4 is decoded by the variable length decoding unit 1100. Basically, the data field of a macroblock includes a type field including information such as the size of the block; a prediction type field including information on whether encoding is done in intra mode or inter mode, and reference frame information and motion vector information in the case of inter mode; a Coded Picture Buffer (CPB) field containing information for maintaining a bit string of a previous picture input during decoding; a Quantization Parameter (QP) field including information about a quantization parameter; and a DATA field including information on DCT coefficients for the color of the corresponding block.

When a macroblock includes a plurality of subblocks, a plurality of data units of type-prediction type-CPB-QP-DATA shown in the second column of FIG. 4 are connected. The first object region information deriving unit 2100 described later may derive first object region information based on the data size of the bitstream of each subblock. The data size of the bitstream of a subblock may be extracted based on an identifier before variable length decoding, or may be extracted during or after variable length decoding.

The second determination unit 2200, which will be described later, determines the block size and type field of the entire macroblock decoded by the variable length decoding unit 1100 (when there is no subblock) or the type field of the subblocks constituting the macroblock. The object area is detected using the motion vector information known from the prediction type field.

Meanwhile, the color information of the DATA field includes information on colors of a plurality of systems (including color information of YCbCr systems in FIG. 4) in an encoded form. The color information of the DATA field is decoded by the inverse quantization unit 1200 and the inverse transform unit 1300, and the color value of the original image data in the adder 1400 and the image decoding unit 1000 Prediction error information corresponding to a difference between color values predicted by the prediction unit 1500 may be derived.

5 is a diagram schematically illustrating a detailed configuration of an object area detection unit 2000 according to an embodiment of the present invention.

The object area detection unit 2000 includes size information of data of blocks included in the image data; and an object region detection step of deriving object region information of an image based on one or more image decoding parameters extracted in the image decoding step.

extracting, by the object region detection unit 2000, size information of block data from a bitstream of image data before the variable length decoding step is performed; and determining whether size information of the data for the block satisfies a predetermined criterion; and a second object region information deriving unit 2200 for deriving second object region information based on one or more image decoding parameters extracted in the image decoding step. and an object region output unit 2300 that determines object region information based on the first object region information and the second object region information and transmits the object region information to an image analysis unit.

As shown in FIG. 2, the first object region information derivation unit 2100 determines the size of data for each macroblock from the NAL format data stream of video data that has not been decoded by the variable length decoding unit 1100. information can be derived. Based on the data size of each data field indicated as MB in FIG. 3, whether each macroblock corresponds to the object area is determined.

In the video encoding method using a variable size block, in the case of a macroblock in which a complex video is located, since a macroblock (16x16) is divided into a plurality of subblocks (8x8, 4x4, etc.) or contains various information, the corresponding macroblock has a large size. It gets bigger. In addition, in the video encoding method such as H.264, if there are frequently occurring values and non-frequently occurring values among macroblock data, a shorter length code is assigned to the frequently occurring value and a longer code is assigned to the non-frequently occurring value. Encode in a way to reduce the amount of data.

The first object region information deriving unit 2100 utilizes this characteristic of the encoding method using variable size blocks to obtain data of each macroblock from video data before the decoding procedure of the variable length decoding unit 1100 is completed. The size may be derived, and first object area information may be derived based on the size.

In this way, in the present invention, without changing the image decoding unit 1000, valid data capable of detecting the object region is extracted from the video data input to the image decoding unit 1000, and a simple operation is performed for the corresponding macroblock. Determine whether the object area exists.

In one embodiment of the present invention, even when not encoded in a separate macroblock/subblock form, the first object region information deriving unit 2100 determines the first object region information deriving unit 2100 based on the data size of the bitstream of the corresponding block. Object area information can be derived.

When the video data is encoded into a plurality of macroblocks, and some of the macroblocks include subblocks, the first object region information derivation unit 2100 performs only the bitstream data size of the macroblocks as described above. However, in one embodiment of the present invention, the first object region information deriving unit 2100 classifies macroblocks or subblocks with delimiter information of the bitstream of image data and corresponds to macroblocks or subblocks. extracting size information of bitstream data of a block; and determining whether size information of bitstream data for a macroblock or subblock having no subblocks satisfies a predetermined criterion; thereby deriving first object region information.

In one embodiment of the present invention, the second object region information deriving unit 2200 determines the object region based on one or more image decoding parameters for the entire macroblock when the macroblock does not include a subblock. and if the macroblock includes subblocks, the object region is detected based on at least one image decoding parameter for each of the subblocks.

The inventor of the present invention implemented the second object area information derivation unit 2200 using various information such as block size information, motion vector size information, grouping information based on motion vector direction angle, and prediction error information through various simulations. As a result, in the case of detecting the object area in conjunction with the size of the bitstream data of the block, only the motion vector size information is used or grouping information (or motion vector direction angle information) based on the motion vector size information and the motion vector direction angle is used. It was confirmed that the case satisfies both accuracy and operation speed.

In a preferred embodiment of the present invention, the one or more video decoding parameters include motion vector information of a macroblock or a subblock constituting a macroblock. include Each macroblock or subblock of an intermode frame includes motion vector information (direction and magnitude), and preferably, the second object region information deriving unit 2200 uses motion vector magnitude information among motion vector information. It is used to determine whether the corresponding macroblock or subblock can correspond to the object area or a value related to the determination.

In the case of an area where an object exists, there is a higher possibility that there is motion than a background area. Meanwhile, each image block of a reference frame (eg, P frame) in an image encoding method using a variable block includes size information of a motion vector for the reference frame. In an embodiment of the present invention, the second object region information deriving unit 2200 performs object region detection using the encoding characteristic. That is, the second object region information derivation unit 2200 determines that the greater the motion vector of the macroblock or subblock to determine whether or not it is an object region, the higher the probability that the macroblock or subblock corresponds to the object region, or a score for determining whether or not it is an object region. For , a higher score is given than blocks with small motion vectors.

Here, as shown in FIG. 5, the motion vector information is derived from information decoded by the variable length decoding unit 1100. In this way, parameters capable of detecting the object region may be derived while maintaining the configuration of the image decoding unit 1000 without generating separate feature information for deriving the object region.

That is, in a preferred embodiment of the present invention, the object region detection unit 2000 derives first object region information based on size information of bitstream data for a block included in the image data, and in the image decoding step Second object area derivation is detected based on motion vector information derived from information decoded by the variable length decoding unit, and the object area information is a region corresponding to the first object area information or the second object area information. contains information Here, the object area information is derived by performing an OR operation on an area corresponding to the first object area information and an area corresponding to the second object area information.

6 is a diagram showing several examples of macroblocks.

6(A) shows a case in which a macroblock consists of one block, FIG. 6(B) shows a case in which a macroblock consists of four blocks, and FIG. 6(C) shows a case in which a macroblock consists of seven blocks. , and FIG. 6 (D) shows a case consisting of 16 blocks.

As described above, in one embodiment of the present invention, the first object area information deriving unit 2100 determines size information of data for each macroblock. In another embodiment of the present invention, the first object area information deriving unit 2100 determines size information of data for a macroblock or a subblock included in a macroblock.

Regarding the embodiment in which the first object area information derivation unit 2100 determines the size information of data for each macroblock, when it is composed of a plurality of subblocks as shown in FIG. It is highly likely that the size of bitstream data for a macroblock is higher than in (A) of 6. In this case, the first object region information derivation unit performs determination in a direction in which the macroblock as shown in FIG. 6(D) is more likely to correspond to the object region than the macroblock as shown in FIG. 6(A).

7 is a diagram illustrating an example of a macroblock including subblocks.

In the encoding of video data using variable size blocks, subblocks in the same macroblock may have different block sizes and motion vectors, as shown in FIG. 7 .

Blocks #4, #5, #6, and #7 of FIG. 7 have motion vectors larger than those of blocks #1, #2, and #3. Accordingly, the second object region information deriving unit 2200 may use blocks #4 and #5. , #6 and #7 can be identified as object areas with a relatively high probability.

8 is a block-based diagram illustrating a process of detecting an object area according to an embodiment of the present invention.

As described above, in one embodiment of the present invention, the object area detection step performed by the object area detection unit 2000, First object region information is derived based on size information of bitstream data for a block included in the image data, and based on motion vector information derived from information decoded by the variable length decoding unit in the image decoding step, A second object region derivation is detected, and the object region information includes information of a region corresponding to the first object region information or the second object region information.

In one embodiment of the present invention, the first object area information corresponds to information about blocks whose size of the bitstream data meets a predetermined standard (eg, when the size is greater than or equal to the predetermined size), or the size of the bitstream data corresponds to the predetermined standard. It can be implemented with information about the upper left vertex and the lower right vertex of the rectangle formed along the boundaries of the blocks.

Similarly, in one embodiment of the present invention, the second object area information is the block(s) in which the information of the motion vector meets a preset criterion (eg, when the size of the motion vector is greater than or equal to the preset size; or Corresponds to information about blocks whose motion vector size is equal to or greater than a preset size and is grouped in the direction of the motion vector), or information about the upper left vertex and lower right vertex of a rectangle formed along the boundary of the corresponding blocks. etc. can be implemented.

For example, (A) of FIG. 8 exemplarily shows an area corresponding to the first object area information, and (B) of FIG. 8 exemplarily shows an area corresponding to the second object area information. In this case, in a preferred embodiment of the present invention, the object area detection unit 2000 determines the area corresponding to (C) of FIG. 8 (C) that can correspond to the first object area information or the second object area information (OR operation). It can be derived from area information.

As in the present invention, in the case of detecting an object region by considering the size of the bitstream data of a block and the magnitude (and direction) of a motion vector of a block, each derived region is OR-operated as shown in FIG. 8(C). The merging conformed to the real object area, and also had the maximum operation speed.

9 is a diagram exemplarily showing the operation of the first object area information deriving unit 2100 on a block basis according to an embodiment of the present invention.

In a preferred embodiment of the present invention, the object region detection step may include size information of data for blocks in a plurality of consecutive frames of a preset number and motion vector information of a plurality of frames of a preset number. Based on this, object area information is derived. For example, the object area is derived by comprehensively considering the size of bitstream data and motion vector information of blocks of 5, 6, 7, etc. frames.

In one embodiment of the present invention, as shown in FIG. 9, selecting any one frame (preferably an I frame) among a plurality of preset numbers of frames (five in the case of FIG. 9), and in each block The size (absolute value) of bitstream data for each frame is normalized based on the entire information on the size of data in the corresponding frame. That is, when the second frame is selected in FIG. 9, the size (absolute value) of bitstream data for the seven blocks of the second frame may be S1, S2, S3, S4, S5, S6, and S7. Here, NS1, NS2, NS3, NS4, NS5, NS6, and NS7 corresponding to the sizes of normalized data can be derived based on the information of S1 to S7. An important point here is that the normalization criterion is bitstream data size information in a corresponding frame. Through this process, more accurate object area detection can be performed.

In one embodiment of the present invention, before or after the process of normalizing the size (absolute value) of bitstream data for each block based on the entire information on the data size in the corresponding frame, the size of the block or macroblock is A process of extracting only blocks that are a predetermined criterion (eg, top 70%) may be performed. This is to make the calculation faster.

For example, the values of S1 to S7 may be normalized in the following form. Various types of known normalization techniques may be used in the present invention.

S1 S2 S3 S4 S5 S6 S7 Block bitstream data size of the frame (absolute value) 20 40 40 20 50 100 200 Size information of normalized bitstream data [0 ~ 10] One 2 2 One 2.5 5 10

S1 S2 S3 S4 S5 S6 S7 Block bitstream data size of the frame (absolute value) 2 4 4 2 5 10 20 Size information of normalized bitstream data [0 ~ 10] One 2 2 One 2.5 5 10

10 is a diagram exemplarily showing the operation of the first object area information derivation unit on a block basis according to an embodiment of the present invention.

In a preferred embodiment of the present invention, the object region detection step may include size information of data for blocks in a plurality of consecutive frames of a preset number and motion vector information of a plurality of frames of a preset number. Based on this, object area information is derived. For example, the object area is derived by comprehensively considering the size of bitstream data and motion vector information of blocks of 5, 6, 7, etc. frames.

In one embodiment of the present invention, as shown in FIG. 10, the size (absolute value) of bitstream data for each block in a plurality of preset numbers of frames (five in the case of FIG. 9) is the data in the frame. Normalize based on full information about size.

That is, for example, the sizes (absolute values) of bitstream data for the seven blocks of the first frame may be S11, S12, S13, S14, S15, S16, and S17. Here, based on the information of S11 to S17, NS11, NS12, NS13, NS14, NS15, NS16, and NS17 corresponding to the size of the normalized data may be derived. An important point here is that the normalization criterion is bitstream data size information in a corresponding frame. Through this process, more accurate object area detection can be performed. For such a normalization process, the method described with reference to FIG. 9 may be employed.

In this way, 5 sets of information including normalized size information of the bitstream data size for each block of a plurality of (5) frames as in the second column of FIG. 10 can be derived.

Then, as shown in column 3 of FIG. 10, final size information of each block may be derived by summing the normalized size information of each block in the 5 sets.

For example, AS1 = f (NS11, NS21, NS31, NS41, NS51, NS61, NS71) or simply NS11+NS21+NS31+NS41+NS51+NS61+NS71 in the frame shown in column 3 of FIG. It can be.

Thereafter, the first object region information may be derived based on blocks meeting a predetermined criterion (for example, greater than or equal to a specific value) in the comprehensive block data size determination value of each block, such as AS1 to AS7.

That is, the object region detecting step may include deriving a comprehensive block data size determination value for each block based on size information of bitstream data of each block for each of a plurality of frames of a predetermined number; and deriving first object area information based on information of blocks whose comprehensive block data size determination value meets a predetermined criterion, wherein the object area information includes information including the first object area information. is determined based on

Preferably, the size information of the bitstream data of the block is a normalized value based on overall information on the size of data in each frame.

11 is a diagram exemplarily showing the operation of the second object area information derivation unit on a block basis according to an embodiment of the present invention.

In a preferred embodiment of the present invention, the object region detection step performed by the second object region information deriving unit 2200 is based on a preset reference to the size of a motion vector of each block for each of a plurality of frames of a preset number. , the motion vector judgment value of each block is given as a first numerical value, and the motion vector judgment value of each block is given when the magnitude of the motion vector of each block does not meet the predetermined criterion. The step of giving 2 numerical values; is performed.

For example, for each block of 5 frames, if the size of the corresponding motion vector is a predetermined standard (eg, greater than or equal to a specific value), 2 is assigned to the corresponding block, otherwise, the corresponding block Assuming that 0 is assigned to , 5 motion vector decision value maps including motion vector decision values for 5 frames as shown in column 1 of FIG. 11 can be derived.

Thereafter, a step of deriving a comprehensive motion vector judgment value for each block based on the motion vector judgment values for each block of each of the plurality of frames of the predetermined number is performed.

For example, in the case of summing motion vector judgment values in a plurality of frames of each block, in the case of a block at position (2, 2), the total motion vector judgment value is 10 (2+2+2+2+2) can be derived as On the other hand, in the case of the block at the (3, 3) position, the comprehensive motion vector judgment value can be derived as 8 (2 + 2 + 2 + 2).

Thereafter, a step of deriving second object area information based on information of blocks whose comprehensive motion vector judgment value meets a predetermined criterion is performed. For example, when the preset criterion is 8 or more, in FIG. 11, (2,2), (2,3), (3,2), and (3,3) are blocks according to the second object area information. It may correspond, and the final object area information may be determined based on information including the second object area information.

In another embodiment of the present invention, the step of deriving the second object region information is derived according to whether the comprehensive motion vector judgment value of each block meets a predetermined criterion and the direction of the motion vector of each block. Second object area information may be derived based on the grouping information. For example, based on information about blocks whose motion vectors are grouped within a predetermined range with respect to blocks whose motion vectors have a predetermined size or more through the same process as in FIG. 11, the second object Area information may be derived.

That is, the object region derivation and detection step may include the object region based on size information of data for blocks in a plurality of frames of a preset number and magnitudes and directions of motion vectors in a plurality of frames of a preset number. derive information

Further, in one embodiment of the present invention, in the object region detection step (more specifically, the step performed by the second object region detection unit 2200), the size of the motion vector; Alternatively, noise in the object area may be removed by additionally applying a known filter to the area derived by considering the size and direction of the motion vector.

12 is a diagram showing an example of a video screen and a detailed data processing map according to the operation of the first object area information deriving unit according to an embodiment of the present invention.

The upper left photo of FIG. 12 shows an exemplary frame image.

The upper right image of FIG. 12 shows an image map of blocks corresponding to the upper 70% of macroblock sizes.

The lower left image of FIG. 12 shows an image map for information obtained by summing the normalized bitstream data sizes of macroblocks of 7 frames.

The lower right image of FIG. 12 shows an image map of a block whose final size information of the block described with reference to FIG. 10 corresponds to a predetermined criterion or higher.

13 is a diagram showing an example of a video screen and detailed data processing map according to the operation of the second object area information deriving unit according to an embodiment of the present invention, and FIG. 14 is an object area detection according to an embodiment of the present invention. It is a diagram showing an example of object area detection according to the method.

2. Teaching information or gait information analysis algorithm using decoding parameters

15 schematically illustrates the entire steps of a method for analyzing traffic information or walking information according to an embodiment of the present invention.

The traffic or walking information analysis method of the present invention is performed in a computing system including one or more processors and a main memory storing commands executable by the processor.

The computing system may include some or all of the modules for performing the object area detection method described with reference to FIGS. 1 to 14 .

A method for analyzing traffic or walking information according to embodiments of the present invention is based on size information of block data or image decoding parameters in a state in which decoding is not performed on a first analysis frame including a target frame of image data. a motion frame determination step (S100) of determining whether or not the target frame is a motion frame; A first object derivation step (S200) of decoding the target frame, extracting a decoded image, and deriving one or more first objects by performing object detection by a deep learning-based first machine learning model on the decoded image; Comparison object region derivation step of detecting one or more comparison object regions inside the target frame based on size information or image decoding parameters of data for a block with respect to the second analysis frame including the target frame of the image data (S300) ; and a second method for deriving one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas. object derivation step (S400); and an analysis step (S500) of determining the first object and the second object as vehicle or pedestrian objects, performing tracking, and analyzing traffic or walking information.

In step S100, in a state in which decoding is not performed on the first analysis frame including the target frame of video data, whether the target frame is a frame with motion is determined based on size information of block data or image decoding parameters. judge

In step S100, based on the data size of the macroblock by the first object region information deriving unit 2100 of FIG. 5 for a single frame (target frame), it is determined whether the corresponding frame is a frame with motion or Based on the motion vector of the block by the second object region information output unit 2200, it can be determined whether the corresponding frame is a frame with motion.

Alternatively, in another embodiment of the present invention, with respect to a plurality of frames (first analysis frame) including a single target frame, the data size of a macroblock by the first object region information deriving unit 2100 of FIG. Based on this, it can be determined whether the corresponding frame is a frame with motion, or it can be determined whether the corresponding frame is a frame with motion based on the motion vector of the block by the second object region information output unit 2200 of FIG. .

In this step S100, the method using the decoding parameters and block data size information described with reference to FIGS. 1 to 14 may be used.

For example, in the case of performing the operation based on a single target frame, based on the total data size of a plurality of macroblocks constituting a single target frame or the total sum of motion vector sizes of a plurality of blocks, It can be determined whether or not there is motion.

Alternatively, for example, in the case of performing based on a plurality of frames, the sum or plurality of data sizes of a plurality of macroblocks described with reference to FIGS. 9 to 11 for the target frame and the plurality of frames preceding the target frame. Based on the sum of motion vector magnitudes of the blocks of , it may be determined whether there is motion in the target frame.

In another embodiment of the present invention, it is possible to determine whether there is motion in the target frame in various forms other than the above-described method. Preferably, in order to increase the operation speed, it is determined whether the corresponding target frame is a frame with motion based on information included in the NAL unit before decoding the corresponding target frame. However, in another embodiment of the present invention, a method of determining whether there is motion based on a decoded image after a corresponding target frame is decoded may also be employed.

In S200, the target frame is decoded to extract a decoded image, and object detection is performed on the decoded image by a first machine learning model based on deep learning to derive one or more first objects.

The first machine learning model is a model used to analyze general traffic information or walking information, and can be implemented through a known artificial neural network such as CNN, and a detailed description thereof will be omitted.

In S200, an object is detected for a frame in which motion is determined in the same manner as before, and this is designated as a first object.

On the other hand, in the case of the first machine learning model learned by the human labeling task, when the object is occluded or affected by the environment such as ambient lighting, all objects are accurately identified as the first object, as in human perception. There may be cases where it cannot be detected.

However, in the present invention, based on decoding parameters, in step S300, based on size information of data or image decoding parameters for the second analysis frame including the target frame of the image data, one or more data within the target frame are determined. By detecting the comparison object area, it can be supplemented.

In one embodiment of the present invention, the comparison object area detection step compares the target frame and blocks in frames preceding one or more target frames based on at least one of size information and motion vector information of bitstream data. The object area can be detected.

In one embodiment of the present invention, the step of detecting the comparison object region is a composite block for each block by summing the size information of the bitstream data of each block with respect to the target frame and frames preceding one or more target frames. Deriving a data size judgment value; and deriving a comparison object area based on information of blocks whose comprehensive block data size determination value meets a predetermined criterion.

Preferably, the size information of the bitstream data of the block may be a normalized value based on overall information on the size of data in each frame.

Such a process may be implemented by the method described with reference to FIG. 10 .

For example, if the current analysis target frame is the 100th frame, the comparison object area in the 100th frame is derived by summing the size information of each macroblock for each of the 96th, 97th, 98th, 99th, and 100th frames can do. This is to eliminate noise-related errors, which may cause errors in detection of the comparison object area when the data size of the macroblock temporarily increases due to noise.

In one embodiment of the present invention, the comparison object region detection step may include, when the size of a motion vector of each block meets a predetermined criterion for each of the target frame and one or more frames preceding the target frame, respectively. assigning a motion vector judgment value of each block as a first numerical value, and assigning a motion vector judgment value of each block as a second numerical value when the magnitude of the motion vector of each block does not meet a predetermined criterion; deriving a comprehensive motion vector judgment value for each block by accumulating the motion vector judgment values for each block of each target frame and frames preceding one or more target frames; and deriving, as a comparison object area, blocks whose comprehensive motion vector judgment value meets a predetermined criterion.

In another embodiment of the present invention, rather than using the first numerical value and the second numerical value, the motion vector determination value of the block itself is cumulatively summed to derive a comprehensive motion vector determination value and compare it with a preset standard, Blocks can also be derived as a comparison object area.

Preferably, the comparison object area detection step is based on whether the comprehensive motion vector judgment value of each block meets a predetermined criterion and grouping information derived according to the direction of the motion vector of each block to the second object area. information can be derived.

Such a process may be implemented by the method described with reference to FIG. 11 .

For example, if the current analysis target frame is the 100th frame, the motion size information of each block is summed up for each of the 96th, 97th, 98th, 99th, and 100th frames to derive a comparison object area in the 100th frame can do. This is to eliminate noise-related errors, which may cause errors in detection of the comparison object area when the magnitudes of motion vectors of some blocks temporarily increase due to noise.

In another embodiment of the present invention, a comparison object area detected through macroblock size information as shown in FIG. 10 and a comparison object area detected through motion vector information as shown in FIG. can also be derived.

In step S400, one or more second objects are derived by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas. A second object derivation step (S400) is performed.

Specifically, in step S400, it is determined whether first objects exist in one or more comparison object areas detected in the target frame. If the first object does not exist in some comparison object areas, it is considered that there is a possibility that an object such as a pedestrian or a vehicle exists in the comparison object area but is not detected by the first machine learning model, and a second Object derivation by machine learning model is performed.

In step S500, an analysis step of determining the first object and the second object as vehicle or pedestrian objects, performing tracking, and analyzing traffic or walking information is performed.

In the analysis step, various fields of analysis may be performed using the recognized object and tracking information of the object.

(1) Smart Intersection System

Detailed analysis: aggregation of passing traffic volume by direction, measurement of occupancy rate, space occupancy information of moving flow subject to response, U-turn vehicle detection, vehicle type classification, number of initial queues, length of queues, jaywalking detection, stopped vehicle detection, tail-biting detection

(2) Traffic information collection system (VDS)

Detailed analysis details: traffic count, vehicle speed measurement, occupancy measurement, stopped vehicle detection, pedestrian detection, accident vehicle detection

(3) Unexpected situation detection system

Detailed analysis: stopped vehicle detection, pedestrian detection, reverse running vehicle detection

(4) Crosswalk Pedestrian Safety System

Detailed analysis: Detection of people waiting at crossing, detection of pedestrians on the sidewalk

(5) Alley Pedestrian Safety System

Detailed analysis details: collision prediction situation detection, collision prediction notification system

16 illustrates frames used for detection of a comparison object area according to an embodiment of the present invention.

In one embodiment of the present invention, the comparison object area can be detected using the macroblock data size for a single frame and the motion vector of the block.

However, in the case of a single frame, when a motion vector or macroblock size temporarily increases due to noise or video compression in some frames, erroneous comparison object region detection may be performed.

In a preferred embodiment of the present invention, in order to solve this problem, when the nth frame is the analysis target frame as shown in FIG. 16, n-1, n-2, . . . , by summing the data size or motion vector information of the macroblocks for the n-kth frame to detect the comparison object area of the analysis target frame, it is possible to have an advantage of performing more accurate comparison object area detection.

17 exemplarily illustrates an object detection process according to an embodiment of the present invention.

17 shows a case where there are actually three objects, the first object also includes all three objects, and the comparison object area also includes three first objects. This corresponds to the case where the first object and comparison object regions of the first machine learning model are accurately detected.

18 exemplarily illustrates an object detection process according to an embodiment of the present invention.

In FIG. 18, the first machine learning model accurately detects three first objects, but corresponds to a case in which the human object was not detected as a decoding parameter in the comparison object area. In this case, by selecting all three first objects as final objects, a malfunction in the comparison object area does not lead to a malfunction in actual object detection.

19 exemplarily illustrates an object detection process according to an embodiment of the present invention.

In FIG. 19, the human object is omitted in the first machine learning model, but the comparison object area according to the decoding parameter corresponds to a case in which a block including a human object is determined to be a comparison object area.

In this case, an additional object region is detected by the second machine learning model for the comparison object region to which the person belongs. Preferably, additional object region detection is performed on the decoded image corresponding to the comparison object region.

20 schematically illustrates operations of a first machine learning model and a second machine learning model according to an embodiment of the present invention.

In one embodiment of the present invention, in the first object derivation step, the decoded image input to the first machine learning model includes an image in a form in which the original image is compressed, and in the second object derivation step, the decoded image is input to the first machine learning model. 2 The decoded image of the comparison object region input to the machine learning model includes an original image or a compressed image compressed to a higher quality than the original image input in the first object deriving step.

That is, in the first object derivation step for detecting objects in the entire image area, the operation speed is increased by using the compressed image. In this case, false detection of the object area may occur. In this way, the falsely detected part is found using the comparison object area that can be detected at high speed using the decoding parameter.

In the case of the comparison object area where the first object is not included, it may be missed by the detection of the first machine learning model by the compressed image, so the second machine learning model has an uncompressed original image or the first Object detection accuracy may be partially increased by inputting an image of the comparison object region having a compression rate lower than that of the image input to the machine learning model (ie, better quality or higher resolution) to the second machine learning model.

21 schematically illustrates operations of a first machine learning model and a second machine learning model according to an embodiment of the present invention.

In one embodiment of the present invention, the second machine learning model requires a relatively higher computational load than the first machine learning model, and can perform more accurate detection.

The embodiment in FIG. 21 can be used simultaneously with the method of differentiating the compression quality of images input to the first machine learning model and the second machine learning model as shown in FIG. 20.

That is, when compared to the first machine learning model, the second machine learning model receives a decoded image of better quality, and in the case of the second machine learning model, it has higher detection accuracy by having more filters and layers. may be

22 schematically illustrates the internal configuration of an analysis system for traffic information or walking information according to an embodiment of the present invention.

A traffic or walking information analysis system according to an embodiment of the present invention may be implemented as a computing system including one or more processors and a main memory storing commands executable by the processor.

The traffic or walking information analysis system 4000 does not perform decoding on the first analysis frame including the target frame of image data, based on size information of block data or image decoding parameters, and the target frame a motion frame determining unit 4100 that determines whether or not this is a motion frame; a first object region derivation unit 4200 that decodes the target frame, extracts a decoded image, and derives one or more first objects by performing object detection on the decoded image by a first machine learning model based on deep learning; A comparison object area derivation unit (4300) for detecting one or more comparison object areas inside the target frame based on size information of block data or image decoding parameters with respect to the second analysis frame including the target frame of the image data. ; and a second method for deriving one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas. an object area derivation unit 4400; and an analyzer 4500 that determines the first object and the second object as vehicle or pedestrian objects, performs tracking, and analyzes traffic or walking information.

The description of these configurations corresponds to the steps described with reference to FIGS. 15 to 20 .

23 schematically illustrates an encoder system for generating an encoded image according to an embodiment of the present invention.

The object region detection unit and the image decoding unit according to an embodiment of the present invention described above may be applied to image data encoded using a variable size block as shown in FIG. 6 . As a representative example, it can be applied to image data encoded by the H.264 codec.

The encoder 10 shown in FIG. 23 includes a DCT unit (Discrete Cosine Transform) 11, a quantization unit 12, and an inverse quantization unit to generate data shown as image data in FIGS. 1 and 2. (Inverse Quantization; IQ) 13, Inverse Discrete Cosine Transform (IDCT) 14, Frame Memory 15, Motion Estimation and Compensation (ME/MC) 16 and Variable A variable length coding (VLC) 17 may be included. Similarly, it is preferable that the video decoding unit is configured to correspond to the configuration of the encoding unit.

To briefly explain this, the DCT unit 11 performs a DCT operation on input image data in units of pixel blocks of a predetermined size (eg, 4x4) in order to remove spatial correlation.

Thereafter, the quantization unit 12 performs quantization on the DCT coefficients obtained by the DCT unit 11 and expresses them with several representative values, thereby performing highly efficient lossy compression.

In addition, the inverse quantization unit 13 inversely quantizes the image data quantized by the quantization unit 12 .

The inverse transform unit 14 performs IDCT transformation on the image data inversely quantized by the inverse quantization unit 13 .

The frame memory 15 stores the IDCT-converted image data in the inverse transform unit 14 in units of frames.

Meanwhile, the motion estimating and compensating unit 16 estimates a motion vector (MV) for each macroblock by using the video data of the current frame and the video data of the previous frame stored in the frame memory unit 15. A sum of absolute difference (SAD) corresponding to a block matching error is calculated.

The variable length coding unit 17 removes statistical redundancy from DCT and quantized data according to the motion vector estimated by the motion estimation and compensation unit 16 .

24 is a diagram schematically illustrating examples of frames of video data;

The video portion of a general motion picture includes I frames (frames indicated by “I” in FIG. 24), P frames (frames indicated by “P” in FIG. 24), and B frames (frames indicated by “B” in FIG. 24). ) is composed of

An I frame includes all images as a key frame, can function as an access point in a video file, corresponds to an independently encoded frame, and has a low compression rate.

Meanwhile, in the case of a P frame, a frame created by forward prediction by referring to a previous I frame or P frame does not correspond to an independently encoded frame. Such a P frame has a higher compression ratio than an I frame. Here, the “previous” frame means not only the immediately preceding frame but also one of a plurality of frames existing before the corresponding frame, and the “post” frame means not only the immediately following frame but also one of a plurality of frames existing after the corresponding frame. means one

On the other hand, in the case of a B frame, it is a frame created by forward and backward prediction with reference to previous and subsequent frames, and does not correspond to an independently encoded frame. Such a B frame has a higher compression rate than I and P frames. Accordingly, the independently encoded frame may correspond to an I frame, and the non-independently encoded frame may correspond to the remaining B frames or P frames.

The B and P frames correspond to reference frames, and preferably, the object region detection unit performs object region detection on these reference frames.

25 illustratively illustrates the internal configuration of a computing device according to an embodiment of the present invention. All or some of the components shown in FIG. 2 may include components of a computing device described later.

As shown in FIG. 25, a computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input/output subsystem ( I/O subsystem 11400, a power circuit 11500, and a communication circuit 11600 may be included at least.

The memory 11200 may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. there is. The memory 11200 may include a software module, a command set, or other various data necessary for the operation of the computing device 11000.

In this case, access to the memory 11200 from other components, such as the processor 11100 or the peripheral device interface 11300, may be controlled by the processor 11100. The processor 11100 may be composed of single or multiple processors, and may include GPU and TPU type processors in order to improve calculation processing speed.

Peripheral interface 11300 may couple input and/or output peripherals of computing device 11000 to processor 11100 and memory 11200 . The processor 11100 may execute various functions for the computing device 11000 and process data by executing software modules or command sets stored in the memory 11200 .

Input/output subsystem 11400 can couple various input/output peripherals to peripheral interface 11300. For example, the input/output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor, keyboard, mouse, printer, or touch screen or sensor to the peripheral interface 11300 as needed. According to another aspect, input/output peripherals may be coupled to the peripheral interface 11300 without going through the input/output subsystem 11400.

The power circuit 11500 may supply power to all or some of the terminal's components. For example, power circuit 11500 may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator or power It may contain any other components for creation, management and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, the communication circuit 11600 may include an RF circuit and transmit/receive an RF signal, also known as an electromagnetic signal, to enable communication with other computing devices.

The embodiment of FIG. 25 is just one example of the computing device 11000, and the computing device 11000 may omit some of the components shown in FIG. 25, further include additional components not shown in FIG. It may have a configuration or arrangement combining two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented as hardware including one or more signal processing or application-specific integrated circuits, software, or a combination of both hardware and software.

Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in computer readable media. In particular, the program according to the present embodiment may be composed of a PC-based program or a mobile terminal-specific application. An application to which the present invention is applied may be installed in a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of a user terminal.

According to one embodiment of the present invention, when performing analysis such as recognizing, tracking, tracking, and identifying an object from encoded image data, an object region is detected in advance and image analysis is performed only for the detected object region. It is possible to achieve an effect of processing images at a higher speed.

According to an embodiment of the present invention, even in detecting an object region before image analysis, an operation speed for the object region detection can be reduced, thereby increasing the overall system operation speed.

According to an embodiment of the present invention, in spite of a minimum amount of computation in detecting an object area, an effect of detecting an object area where it is determined that an object may exist can be achieved with a fairly high level of accuracy.

According to an embodiment of the present invention, H.264, H. 265 (HEVC), H.265 (HEVC), A codec method using a block conforming to a rule such as H. 266 (VVC) can exert an effect that can be easily applied even if the codec is changed.

According to an embodiment of the present invention, it can be used for real-time image analysis, reading, and detection such as high-resolution CCTV image detection by detecting an object area with high accuracy and high speed regardless of environmental factors such as the characteristics of a recording device and a recording location. effect can be exerted.

The device described above may be implemented as a hardware component, a software component, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (10)

A traffic or walking information analysis method performed in a computing system including one or more processors and a main memory storing instructions executable by the processor,
a motion frame determination step of determining whether a target frame is a frame with motion with respect to a first analysis frame including a target frame of image data;
a first object derivation step of decoding the target frame, extracting a decoded image, and deriving one or more first objects by performing object detection using a deep learning-based first machine learning model on the decoded image;
a comparison object region derivation step of detecting one or more comparison object regions inside the target frame based on size information of block data or image decoding parameters with respect to the second analysis frame including the target frame of the image data; and
A second object that derives one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas derivation step; and
and an analysis step of determining the first object and the second object as vehicle or pedestrian objects, performing tracking, and analyzing traffic or walking information. The method of claim 1,
In the step of detecting the comparison object area,
A method for analyzing traffic or walking information, wherein a comparison object region is detected based on at least one of size information and motion vector information of bitstream data for blocks in the target frame and frames preceding one or more target frames.
The method of claim 1,
In the step of detecting the comparison object area,
deriving a comprehensive block data size determination value for each block by summing size information of bitstream data of each block with respect to the target frame and frames preceding one or more target frames; and
and deriving a comparison object area based on information of blocks whose comprehensive block data size determination value meets a predetermined criterion.
The method of claim 3,
The size information of the bitstream data of the block is a normalized value based on the entire information on the data size in each frame.

The method of claim 1,
In the step of detecting the comparison object area,
When the size of the motion vector of each block meets a predetermined criterion for each of the target frame and one or more frames preceding the target frame, a motion vector judgment value of each block is given as a first numerical value, respectively. assigning a motion vector judgment value of each block as a second numerical value when the size of the motion vector of the block does not meet a predetermined criterion;
deriving a comprehensive motion vector judgment value for each block by accumulating the motion vector judgment values for each block of each target frame and frames preceding one or more target frames; and
A method for analyzing traffic or walking information, comprising: deriving blocks whose comprehensive motion vector judgment values meet a predetermined criterion as a comparison object area.
The method of claim 5,
The comparison object area detection step derives second object area information based on grouping information derived according to whether the comprehensive motion vector judgment value of each block meets a preset criterion and the direction of the motion vector of each block. , Traffic or pedestrian information analysis method.
The method of claim 1,
In the first object derivation step, the decoded image input to the first machine learning model includes an image in which the original image is compressed,
In the second object derivation step, the decoded image of the comparison object area input to the second machine learning model is compressed to a higher quality than the original image or an image in which the original image input in the first object derivation step is compressed. A method for analyzing traffic or walking information, including compressed images.
The method of claim 1,
The second machine learning model requires a relatively higher computational load than the first machine learning model and can perform more accurate detection.
A traffic or walking information analysis system implemented as a computing system including one or more processors and a main memory storing instructions executable by the processors,
A motion of determining whether the target frame is a frame with motion based on size information of block data or image decoding parameters in a state in which decoding is not performed on the first analysis frame including the target frame of image data. frame judgment unit;
a first object region derivation unit that decodes the target frame, extracts a decoded image, and derives one or more first objects by performing object detection on the decoded image by a first machine learning model based on deep learning;
a comparison object area deriving unit for detecting one or more comparison object areas inside the target frame based on size information of data or image decoding parameters for a second analysis frame including the target frame of the image data; and
A second object that derives one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas area extraction unit; and
An analyzer configured to determine the first object and the second object as vehicle or pedestrian objects, perform tracking, and analyze traffic or walking information; a traffic or walking information analysis system comprising:
A computer program stored on a computer-readable medium comprising a plurality of instructions executed by one or more processors,
The computer program,
A motion of determining whether the target frame is a frame with motion based on size information of block data or image decoding parameters in a state in which decoding is not performed on the first analysis frame including the target frame of image data. frame judgment step;
a first object derivation step of decoding the target frame, extracting a decoded image, and deriving one or more first objects by performing object detection using a deep learning-based first machine learning model on the decoded image;
a comparison object region derivation step of detecting one or more comparison object regions inside the target frame based on size information of block data or image decoding parameters with respect to the second analysis frame including the target frame of the image data; and
A second object that derives one or more second objects by performing object detection by a deep learning-based second machine learning model on a decoded image of a comparison object area in which the first object does not exist among one or more comparison object areas derivation step; and
An analysis step of determining the first object and the second object as vehicle or pedestrian objects, performing tracking, and analyzing traffic or walking information.

Images