KR101906796B1 - Device and method for image analyzing based on deep learning - Google Patents

Abstract

The present invention discloses a video analyzing apparatus and a video analyzing method. A video analyzing apparatus according to the present invention comprises: a receiving unit for receiving a video image photographed by a photographing apparatus in real time; a color probability image generating unit for calculating a color probability by modeling a color distribution of pixel included in a video image received in at least two-dimensional color space based on color and luminance information and generating a color probability image using the calculated color probability; a background image generation unit for modeling the background of the received video image to generate a background image; a motion image generation unit for generating a motion image using the received video image and background image; and an image correction unit for correcting the candidate probability image by generating a candidate probabilistic image using the color probability image and the motion image, extracting a plurality of characteristics for the candidate probabilistic image, calculating a random probability by an application of a randomness test algorithm to a plurality of extracted characteristics and verifying the candidate probability image using the calculated randomness probability.

Description

TECHNICAL FIELD [0001] The present invention relates to a depth learning based image analysis apparatus and an image analysis method,

The present invention relates to an image analysis apparatus and an image analysis method for detecting and tracking color pedestrian differential pedestrians, and more particularly, to an image analysis apparatus and an image analysis method for pedestrian detection and tracking using color difference degree pedestrians, It is an image processing technology that performs image analysis in real time at the same time as detaching and tracks the same pedestrian from stored image data. When the designated pedestrian is displayed in the bounding box, the color name is used to generate the 7th and 7th digits of the difference data, store it in the label tag, and search for the same pedestrian in other images.

Recently, due to the development of intranet and internet technology, real-time streaming of data has become possible. In particular, as the hardware becomes larger and the abundant system resources become available, it is possible to analyze the real-time image of high-resolution images over Full-HD and to analyze the results in real time.

In order to solve the above-described problems, the present invention has been developed to extract a pedestrian moving object in a CCTV image, extract an image, And to provide a video analysis apparatus and an image analysis method for tracking and locating video.

According to an aspect of the present invention, there is provided an image analyzing apparatus including a receiving unit for receiving a video image photographed by a photographing apparatus in real time, a moving object name of a pedestrian, a color and brightness information A color probability image generating unit for modeling a color distribution of pixels included in the received image image in at least two-dimensional color space to calculate a color probability, and generating a color probability image using the calculated color probability; A background image generation unit for modeling the background of the received video image to generate a background image, a motion image generation unit for generating a motion image using the received video image and the background image, Quot; moving object ") and the motion A candidate probability image is generated using the unknown image, a plurality of characteristics of the candidate probability image are extracted, a randomness test algorithm is applied to the extracted plurality of characteristics to calculate a random probability, And an image corrector for correcting the candidate probabilistic image by verifying the candidate probabilistic image using the random probability.

In one embodiment of the present invention, the color probability image generating unit may calculate a color probability based on the following equation.

는 x, y 좌표의 픽셀에서의 빛 반사 각에 대한 색상 확률을 의미하고, rate는 x, y 좌표 픽셀에서의 Cr/Cb을 의미한다.here,

Denotes the chromatic probability for the light reflection angle at the x, y coordinate pixel, and rate means Cr / Cb at the x, y coordinate pixel.

를 산출하고, 제2 문턱 값으로 이진화하여

를 산출하며, 다음의 수학식에 의해서 최종 빛 반사 각 이동에 따른 움직임 이미지를 생성할 수 있다.In one embodiment of the present invention, the motion image generation unit generates a motion image using the received image and the background image, and binarizes the motion image to a first threshold value

And binarizes it to a second threshold value

And a motion image according to the movement of the final light reflection angle can be generated by the following equation.

In one embodiment of the present invention, the image correction unit may calculate a plurality of randomness probabilities for the plurality of characteristics, calculate a random probability average using the plurality of randomness probabilities, Calculating N random probability averages by repeatedly computing the random probability average over N frames of data, marking the random probability average as Label Tag for a specific pedestrian and storing the same as metadata, The method comprising the steps of: (a) calculating a mean value of a plurality of randomly selected random probabilities from a plurality of images, using the calculated average values as a threshold value for restoring an object image of a moving pedestrian contained in a motion image according to the movement of the light reflection angle, It can be used to search for the same pedestrian as a specific pedestrian.

In one embodiment of the present invention, the plurality of characteristics include seven characteristics of two pedestrian's arms, two legs, one trunk, one head and one entire pedestrian, and the random probability average is . ≪ / RTI >

는 특정 프레임에서의 7개의 특징에 대한 무작위성 확률이며,

는 무 작위성 확률 평균을 나타낸다.here,

Is the probability of randomness for the seven features in a particular frame,

Represents the random probability average.

In one embodiment of the present invention, the image corrector may determine a restored image for tracking the moving pedestrian object color brightness differential image using the following equation.

는 상기 문턱 값, line은 확실한 컬러 빛 굴절 각을 이루고 있는 선으로 판정, Should_line은 흐린 선으로 판정, Maybe_line은 격자 선으로 판정, Not_line은 공백으로 판정하는 것을 의미한다.Where W is the brightness differential pedestrian image,

, The line is determined as a line forming a certain color light refraction angle, the Should_line is determined as a blurred line, the Maybe_line is determined as a lattice line, and the Not_line is determined as a blank line.

In one embodiment of the present invention, the image correcting unit may further include an object detecting unit for detecting an object using the object characteristic learning data, and an object tracking unit for tracking the object using the object characteristic learning data.

In one embodiment of the present invention, the object detection unit generates a pedestrian object using data of the video image, extracts object characteristic learning data of color morpheme information of an object stored for each of the clusters, The object is detected on the basis of the extracted object characteristic learning data, the object detection process is continuously repeated, and the learned learning data can be recorded using the extracted object characteristic learning data. The object tracking unit extracts object characteristic learning data of color morpheme information of an object stored by the clusters, predicts the morpheme information of the object, extracts the most similar morpheme information, and tracks the object based on the extracted morphological information.

According to another aspect of the present invention, there is provided an image analysis method comprising: receiving in real time a video image picked up by an image pickup apparatus; Calculating a color probability by modeling a color distribution of a pixel, generating a color probability image using the calculated color probability, modeling a background of the received image image to generate a background image, Generating a candidate image by using the color probability image and the motion image, extracting a plurality of characteristics for the candidate probability image, The randomness test algorithm for the extracted characteristics is applied Calculating a no-jakwiseong probability, by using the calculated free jakwiseong probability verifying the candidate probability image, and a step of correcting said candidate probability image.

In one embodiment of the present invention, the step of generating the color probability image may calculate the color probability based on the following equation.

는 x, y 좌표의 픽셀에서의 색상 확률을 의미하고, rate는 x, y 좌표 픽셀에서의 Cr/Cb을 의미한다.here,

Denotes the chromatic probability in the pixels of the x and y coordinates, and rate denotes the Cr / Cb in the x, y coordinate pixels.

를 산출하는 단계, 상기 움직임 이미지를 제2 문턱 값으로 이진화하여

를 산출하는 단계 및 다음의 수학식에 의해서 최종 움직임 이미지를 생성하는 단계를 포함할 수 있다.In one embodiment of the present invention, the step of generating a motion image comprises generating a motion image using the received image and the background image, binarizing the motion image to a first threshold value

, Binarizing the motion image to a second threshold value

And generating a final motion image by the following equation: < EMI ID = 1.0 >

In one embodiment of the present invention, the step of correcting the image comprises: calculating a plurality of randomness probabilities for the plurality of characteristics; computing a random probability average using the plurality of randomness probabilities; Calculating N random probability averages by repeatedly calculating the random probability average for N frames of the received image data, writing the random probability average to a label for a specific pedestrian, , Restoring a color brightness difference image included in the motion image using a value calculated by averaging the N random probability averages as a threshold value and restoring the color brightness difference image included in the motion image using the stored non- And searching for the same pedestrian as the specific pedestrian.

In one embodiment of the present invention, the plurality of characteristics include seven characteristics of two pedestrian's arms, two legs, one trunk, one head and one entire pedestrian, and the random probability average is . ≪ / RTI >

는 특정 프레임에서의 7개의 특징에 대한 무작위성 확률이며,

는 무 작위성 확률 평균을 나타낸다.here,

Is the probability of randomness for the seven features in a particular frame,

Represents the random probability average.

In one embodiment of the present invention, the step of reconstructing the image may determine a reconstructed image of the color brightness differential image using the following equation.

는 상기 문턱값, line은 확실한 선으로 판정, Should_line은 흐린 선으로 판정, Maybe_line은 격자 선으로 판정, Not_line은 공백으로 판정하는 것을 의미)(Where W is the restored color brightness differential image,

, The line is determined as a certain line, the Should_line is determined as a blurred line, the Maybe_line is determined as a lattice line, and the Not_line is determined as a blank line)

In one embodiment of the present invention, the step of correcting the image may further include the step of detecting the object using the object characteristic learning data and the step of tracking the object using the object characteristic learning data.

In one embodiment of the present invention, the step of detecting the object may include generating a pedestrian object using data of the image image, extracting object characteristic learning data of color morpheme information of the object stored for each of the clusters, Detecting the object based on the extracted object characteristic learning data, continuously repeating the object detecting step, and recording the learned learning data using the extracted object characteristic learning data. The step of tracking the object may include extracting learning data of color morpheme information of an object stored for each group, extracting the most similar morpheme information by predicting the object morpheme information, and extracting the most similar morpheme information As shown in FIG.

According to various embodiments of the present invention, by capturing a color difference differential image due to the occurrence of a moving object event and performing image correction in real time, distortion phenomenon such as image blurring or image blurring of the moving object is reduced, The recognition rate of the user is increased.

1 is a schematic view illustrating an intelligent image processing system according to an embodiment of the present invention.
2 is a block diagram for explaining an image processing apparatus according to an embodiment of the present invention.
3A to 3D are diagrams illustrating a histogram of Cb and Cr distributions according to an embodiment of the present invention.
4 is a diagram for explaining a process of extracting a color brightness difference image from a captured image according to an embodiment of the present invention.
5A and 5B are views for explaining an embodiment of restoring a moving object image from a source color brightness difference image according to an embodiment of the present invention.
6 is a flowchart illustrating an image processing method according to another embodiment of the present invention.
FIG. 7 is a flowchart for explaining a real-time color brightness difference image information obtaining method according to another embodiment of the present invention.
8 is a flow chart for explaining an object detection step of the image processing method according to an embodiment of the present invention.
9 is a flow chart for explaining an object tracking step of the image processing method according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is a schematic view illustrating an image processing system according to an embodiment of the present invention.

The image processing system 10 according to the present invention includes an image processing apparatus 100, an image capturing apparatus 200, and a video stream communication apparatus 300. [ In addition, the image processing system 10 can perform wired / wireless communication with the remote server 400. The image processing system 10 may be implemented in various types of mobile terminals such as a cellular phone, a PDA, a smart phone, a tablet, and a laptop.

The image processing apparatus 100 processes an image photographed by the photographing apparatus 200, analyzes information such as a color brightness difference image included in the image, and transmits the analyzed information to the remote server 400.

The photographing apparatus 200 is an image photographing apparatus capable of photographing a subject at a full-HD grade or higher resolution. The photographing apparatus 200 may be composed of a plurality of digital cameras based on the RTP / RTSP protocol.

The video stream communication apparatus 300 can perform data streaming communication between the video processing apparatus 100 and the photographing apparatus 200. [ The video stream communication apparatus 300 can transmit a video sequence of image data photographed by the photographing apparatus 200 to the image processing apparatus 100 in real time.

The remote server 400 may comprise a remote database for collecting, processing, and storing information about the image processed by the image processing apparatus 100.

Hereinafter, for convenience of explanation, the image processing apparatus 100 will be described focusing on an embodiment in which the image data photographed by the photographing apparatus 200 is received through the video stream communication apparatus 300. FIG. It will be apparent to those of ordinary skill in the art that these embodiments are illustrative only and can be modified in various ways.

2 is a block diagram illustrating an image processing apparatus according to an embodiment of the present invention. Referring to FIG. 2, the image processing apparatus 100 includes a receiving unit 110, an image processing unit 130, a color brightness difference object separation processing unit 150, a transmitting unit 170, and a control unit 190.

The receiving unit 110 receives the image data photographed from the image capturing apparatus 200. For example, the receiving unit 110 may be implemented as a communication module based on the RTP / RTSP protocol. The receiving unit 110 may be implemented as a video streaming communication module to receive the image data sequence.

The image processing unit 130 includes a color probability image generating unit 131, a background image generating unit 132, a motion image generating unit 133, and an image correcting unit 134. Each configuration of the image processing unit 130 may be configured as an independent module or in the form of an integrated module. Each configuration of the image processing unit 130 will be separately described below.

The color brightness difference differential object separation processing unit 150 can extract information from the color brightness pre-difference image included in the image processed by the image processing unit 130. [

The transmitting unit 170 transmits information about the color brightness differential metadata through wired / wireless communication with the remote server. The transmitter 170 may be implemented as a wired / wireless communication module. For example, the transmission unit 170 may be implemented by various types of wired / wireless communication modules such as the Internet, an intranet, a wireless LAN, a WiFi, and the like.

The control unit 190 can control the overall operation of the image processing apparatus 100. [ That is, the control unit 190 may control the overall operation of the image processing apparatus 100 to operate the image processing apparatus 100 such as image data reception, image image processing, data transmission, and power supply.

Input image data

According to an embodiment of the present invention, the image processing apparatus 100 can receive image data photographed from the photographing apparatus 200 in real time.

Specifically, the image processing apparatus 100 can receive the image data sequence from the photographing apparatus 200 through the image stream communication apparatus 300. [

The photographing apparatus 200 may be implemented as a digital camera capable of photographing a high-resolution image of full-HD or higher. The photographing apparatus 200 includes an image sensor capable of converting light energy into electrical energy, and the image sensor can be implemented in various types such as a CMOS and a CCD. The photographing apparatus 200 photographs a label attached to a moving object moving at high speed in real time. The video stream communication apparatus 300 may transmit the photographed video data sequence to the video processing apparatus 100. [

The video stream communication apparatus 300 is a communication apparatus to which a real-time streaming technology based on the RTP / RTSP protocol is applied. The video stream communication apparatus 300 may be integrally installed in the image processing apparatus 100 or the photographing apparatus 200, or may be configured as an independent apparatus.

The video stream communication apparatus 300 can convert the video data photographed by the photographing apparatus 200 into BGR24 type data and transmit the converted data to the image processing apparatus 100. [ At this time, the BGR 24 type data is sRGB format data having 24 bits per pixel. The video stream communication apparatus 300 can convert the video data into image data of various formats in addition to the data of the BGR24 type.

Color brightness difference  Image of color probability distribution according to movement of object image modelling

The color probability image generating unit 131 may calculate a color probability for pixels constituting image data in a YCbCr color space and generate a color probability image using the calculated color probability.

Specifically, the color probability image generating unit 131 may model a histogram of Cb and Cr of a pixel using the input image data. The chrominance probability image generation unit 131 analyzes the histograms of Cb and Cr to analyze the chrominance probability distribution of the pixels of the input image data, and calculates a minimum value, a maximum value, and a Cb / Cr value of Cb and Cr Value can be calculated.

3A to 3D are diagrams illustrating a histogram of Cb and Cr distributions according to an embodiment of the present invention.

FIG. 3A is a histogram of Cb and Cr distributions for color pixels. In FIG. 3A, the horizontal axis represents pixel values, and the vertical axis represents the number of pixels. The line Cb distribution is indicated by a dotted line, and the line Cr distribution is indicated by a solid line.

FIG. 3B is a histogram of Cb and Cr distributions. In FIG. 3B, the horizontal axis represents pixel values and the vertical axis represents the number of pixels. In FIG. 3B, the line Cb distribution is represented by a first line (Line 1), the line Cr distribution is represented by a second line (Line 2), the non-line Cb distribution is represented by a third line (Line 3) And the non-line Cr distribution is represented by the fourth line (Line 4).

3C is a histogram of the Cb / Cr distribution for the color pixel, the horizontal axis of FIG. 3C is the ratio of Cb / Cr, and the vertical axis is the number of pixels.

FIG. 3D is a histogram of the Cb / Cr distribution. In FIG. 3D, the horizontal axis indicates the ratio of Cb / Cr, and the vertical axis indicates the number of pixels. In FIG. 3D, the line Cb / Cr distribution is represented by the first line, and the non-line Cb / Cr distribution is represented by the second line.

)을 다음의 수학식 1에 의해서 정의한다.The color probability image generating unit 110 generates a color probability image (x, y) of a pixel of the input image data,

) Is defined by the following equation (1).

Equation 1

Here, rate means Cb / Cr, and each pixel in the inputted image data satisfies the following condition.

If the above condition is not satisfied, the color probability of the pixel of the coordinate is 0. The color probability image generating unit 110 calculates the color probability for the pixels of the image data, and outputs the color probability image for the calculated color probability .

Color brightness difference  Object background image modelling

)를 산출하고, 산출된 배경 이미지의 픽셀 값에 의해서 배경 이미지를 생성할 수 있다.The background image generation unit 132 may generate a background image by modeling the background of a specific frame of the image image. That is, the background image generation unit 132 generates a background image by modeling the background of the frame before generating the motion image. The background image generation unit 132 generates a background image by using the following equation (2)

), And a background image can be generated by the pixel value of the calculated background image.

Equation 2

는 n번째 프레임의 배경 이미지의 (i,j) 위치에서의 픽셀 값이고,

는 입력된 영상 이미지의 (i,j) 위치에서의 픽셀 값을 의미한다. 그리고, 첫번째 프레임의 배경 이미지의 픽셀 값(

)는 첫 번째 입력된 영상 이미지의 픽셀 값 (

)과 함께 0으로 초기화된다. 여기서 사용되는 입력된 영상 이미지 및 배경 이미지는 모두 그레이(gray) 이미지이다.here,

Is the pixel value at the (i, j) position of the background image of the n-th frame,

Denotes a pixel value at the (i, j) position of the input image. Then, the pixel value of the background image of the first frame (

) Represents the pixel value of the first input image image (

) Is initialized to zero. The input image image and background image used here are all gray images.

Color brightness difference  Object Motion Image Extraction Algorithm

FIG. 4 is a view for explaining a process of extracting a color brightness difference image from a sensed image according to an embodiment of the present invention.

Referring to FIG. 4, the image processing apparatus 100 can extract the pedestrian image 403 from the input original image 401.

The input original image 401 includes a background image and a moving pedestrian image 403. The motion image generation unit 133 can generate a motion image using the background image and the input image image generated by the background image generation unit 132. [

Specifically, the motion image generating unit 133 calculates the pixel value M (i, j) of the motion image using the following Equation (3) and outputs the calculated motion image pixel value M (i, j) To generate a motion image.

Equation 3

Here, M (i, j) is the pixel value of the motion image, I (i, j) is the pixel value of the input image, and B (i, j) is the pixel value of the background image.

The motion image generating unit 133 calculates the calculated motion image pixel value M (i, j) as the first image pixel value Mb (i, j) by the first threshold value. The motion image generating unit 133 may remove the noise through an open operation of the calculated first image pixel value Mb (i, j).

)으로 이진화할 수 있다. The motion image generating unit 133 generates a second image pixel value (i, j) by the second threshold value for the first image pixel value Mb (i, j)

). ≪ / RTI >

)을 정규화하여 1 또는 0 값으로 최종적으로 이진화함으로써 최종적인 움직임 이미지를 산출할 수 있다.The motion image generation unit 133 generates a motion image using the second image pixel value (

) Is normalized and finally binarized to a value of 1 or 0, so that a final motion image can be calculated.

Color brightness difference  Object Image Calibration Algorithm

The image correction unit 134 generates a candidate probability image using the color probability image and the motion image. The image correction unit 134 extracts a plurality of characteristics for the candidate probability image.

The plurality of features means the characteristic of discriminating the light reflection line of the moving pedestrian. For example, a plurality of features include a start reflection zone of a moving pedestrian, an end zone of light reflection, a width of a reflection angle, a color difference brightness from an original image, a width of a boundary line between objects, resolution, illuminance, The width of the object, and the like.

The image correcting unit 134 calculates the random probability by applying a randomness test algorithm on the extracted characteristics.

The image corrector 134 may use the Wald-Wolfowitz test algorithm as a no-confidence test algorithm. The image correcting unit 134 may calculate the randomness test result through the following Equations (4) to (6).

Equation 4

Here, n ₁ and n ₂ are the numbers of 0 and 1, which are the bits of the characteristic subjected to the randomness test.

Equation 5

Equation 6

Here, R denotes the number of times of changing between 0 and 1, and Z denotes an anisotropy test result on the characteristic.

After the randomness test is completed, the test result Z can be used as a threshold value to provide the random probability P, which is calculated using the following equation (7).

Equation 7

The image correcting unit 134 may calculate the random probability P calculated by Equation (7) for each characteristic. That is, for each of the light reflection start zone, the light reflection end zone, the width of the boundary line of the object, the width of the reflection angle line, the resolution, the illuminance and the color brightness of the moving pedestrian, ₁ to P &_lt; ₇ >

내지

)를 산출하고, 7개의 무 작위성 확률(

내지

)을 이용하여 다음의 수학식 8에 의해서 무 작위성 확률 평균(

)을 산출할 수 있다. 이때 동일 확률 평균(

)은 지정된 보행자에 대하여 Label Tag으로 표기한 후 메타데이터를 저장한다. The image corrector 134 determines the probability of non-randomization (i.e.,

To

), And the seven random probabilities (

To

) Is used to calculate the random probability average (< RTI ID = 0.0 >

) Can be calculated. At this time, the same probability average (

) Stores the metadata after labeling the specified pedestrian with a label tag.

Equation 8

는 특정 프레임에서의 7개의 특징에 대한 무작위성 확률을 의미한다. 7개의 특성은 보행자를 검출함에 있어서 7등분(팔2, 다리2, 몸1, 머리1, 전체1)으로 세분화하여 각각의 컬러명도차분 확률을 산출하여 메타데이터를 생성하고 저장한 후, 다른 영상의 보행자 색 정보와 비교하여 동일 보행자를 추정한다. here,

Means the probability of randomness for seven features in a particular frame. The seven characteristics are subdivided into 7 equal parts (arm 2, leg 2, body 1, head 1, whole 1) in detecting pedestrians, and each color brightness and difference probability is calculated to generate and store metadata, And the same pedestrian is estimated.

That is, the image correcting unit 134 calculates the randomness probability for seven features in a specific frame in order to discriminate the boundary line of the color brightness difference object. Among the seven characteristics, the width and height of the bearer (the color lightness difference object light reflection angle in the ascending movement) can be obtained by the randomness test in terms of the bit combination group related to the color that forms the edge of the color brightness difference object.

On the other hand, in order to average the combinations of bits 0 and 1 at the outer boundaries of the bearer (the color lightness difference object light reflection angle in the ascending movement), the minimum and maximum values of the bit group are not lost by using the nonparametric T-test algorithm of the Wald- Wolfowitz algorithm The average of the number of cases is obtained.

개의 연속 프레임에 대해서 각각 반복 수행한다. 각각의 프레임에 대해서 산출된

개의 복원 전 확률을 이용하여 다음의 수학식 9에 의해서 복원 후 확률(

)을 산출한다.The image correction unit 134 calculates the restoration probability calculated by Equation (8)

Repeat for each successive frame. For each frame,

(9) using the probability of restoration after the restoration

).

Equation 9

는

개의 연속적인 프레임 각각에 대해서 산출된 복원 전 확률의 산술 평균을 의미한다. 수학식 9에 의해서 산출된 복원 후 확률(

)은 다음의 수학식 10에서 컬러명도차분객체 이미지의 라인을 판정하기 위한 문턱 값으로 이용할 수 있다.here,

The

Means an arithmetic mean of the computed restoration probabilities for each of consecutive frames. The post-restoration probability calculated by the equation (9)

) Can be used as a threshold for determining the line of the color difference differential object image in the following equation (10).

Equation 10

)을 문턱 값으로 이용하여 4단계의 컬러명도차분 후 객체 라인 복원 판정을 수행한다. 예를 들어, 복원 후 확률(Pw)이 0.1보다 클 경우에는 해당 비트 그룹은 line으로 판정하고, Pw가 0.07보다 클 경우에는 해당 비트 그룹은 Should_line으로 판정하며, Pw가 0.05보다 클 경우에는 해당 비트 그룹은 Maybe_line으로 판정하고, Pw가 0.05보다 작은 경우에는 해당 비트 그룹은 Not_line으로 판정한다.The image correction unit 134 corrects the post-reconstruction probability (

) Is used as a threshold value to perform object line restoration determination after color brightness difference of four stages. For example, if the probability Pw after recovery is greater than 0.1, the corresponding bit group is determined to be a line. If Pw is greater than 0.07, the corresponding bit group is determined as Should_line. If Pw is greater than 0.05, The group is determined as Maybe_line. If Pw is smaller than 0.05, the corresponding bit group is determined as Not_line.

)이 0.1보다 크면, 해당 비트 그룹은 거의 왜곡이 발생하지 않은 것이므로 선명한 선으로 복원할 수 있음을 의미한다(line으로 판정). 복원 후 확률(

)이 0.1보다 작고 0.07보다 크면, 해당 비트 그룹은 초점이 약간 흐려지는 왜곡이 발생한 것이지만 선으로 복원할 수 있음을 의미한다(Should_line으로 판정). 복원 후 확률(

)이 0.07보다 작고 0.05보다 크면, 해당 비트 그룹은 피사체의 이동으로 촬영 순간 다소 밀림 현상과 부분적인 계단식 격자 또는 계단식 연속성을 가진 격자 라인으로 왜곡이 발생한 것이므로, 이러한 계단식 격자 또는 계단식 연속성을 선으로 복원할 수 있음을 의미한다(Maybe_line으로 판정). 복원 후 확률(

)이 0.05보다 작으면, 피사체의 이동으로 촬영 순간 번짐이나 밀림의 왜곡이 발생한 것으로, 이러한 번짐이나 밀림의 왜곡을 공백으로 복원할 수 있음을 의미한다(Not_line으로 판정).More specifically, the probability of restoration (

) Is larger than 0.1, it means that the corresponding bit group has almost no distortion and can be restored to a clear line (judged as a line). Probability after restoration (

) Is smaller than 0.1 and larger than 0.07, it means that the corresponding bit group is distorted slightly in focus but can be restored to a line (Judged as Should_line). Probability after restoration (

) Is smaller than 0.07 and larger than 0.05, the corresponding bit group is distorted by a grating line having a partial cascade lattice or a cascade continuity at the moment of photographing due to the movement of the subject, so that the cascade lattice or the cascade continuity is restored to a line (Judged as Maybe_line). Probability after restoration (

) Is less than 0.05, it means that the blur of the photographing moment and the distortion of the jumping are caused by the movement of the subject, and that such blurring and jarring distortion can be restored to blank (judged as Not_line).

In addition, the image correcting unit 134 can obtain a position and a boundary value (threshold value, threshold) of a pixel value of an image to be used for a region of interest (ROI) using a vertical / horizontal histogram. In addition, in the image processing of vertical / horizontal histogram, high frequency filter and low frequency filter interpolation are applied to detect the area of the object such as the pedestrian, and the correction effect of the object (the object image is corrected by blurring the part, Can be provided.

The high-frequency filter interpolation method may be defined by Equation (11) below.

Equation 11

는 누적 가중계수,

는 복원 계수,

는 가중 계수를 나타낸다.here,

Is a cumulative weighting coefficient,

Is the restoration coefficient,

Represents a weighting factor.

Further, the low-frequency filter interpolation method can be defined by the following equation (12).

Equation 12

를 나타낸다.here,

.

Further, the filter interpolation area can be defined by the following equation (13).

Equation 13

는 누적 가중계수이고, 이는 배경의 누적 행렬로 형성되는 과정에서 가중된 강도(복원)의 이미지를

복원 계수 복원 계수로부터 획득하고, 감산행렬

를 이용한 색소공간영역 이동에 따른 가중(복원)된 색상의 이미지를 획득할 수 있다. 따라서, 고주파 필터 및 저주파 필터를 이용하여 영상을 보다 선명하게 복원할 수 있다. The high frequency filter interpolation method is a filtering method for processing when a pixel change in an image area is larger than a certain threshold value and a low frequency filter interpolation method is a filtering method for processing when a pixel change in an image area is smaller than a specific threshold value in image processing. Thus, using the Bernoulli formula

Is a cumulative weighting factor, which is the sum of the weighted intensity (restoration)

From the restoration coefficient restoration coefficient, and the subtraction matrix

It is possible to obtain an image of weighted (restored) color according to the movement of the dye spatial domain using the color space. Therefore, the image can be more clearly restored by using the high-frequency filter and the low-frequency filter.

Hereinafter, an example of restoring the color brightness differential object line from the original moving pedestrian image will be described with reference to FIGS. 5A and 5B.

5A, the graph of the upper side of FIG. 5A shows that the color brightness differential object image is a Blab graph waveform of a Wald-Wolfowitz test T-test result of a line blur judgment (Should_line) and a lattice line judgment (maybe_line) to be. In the case of clouding, jumping and lattice generation, the graph waveforms are shown irregularly.

The graph in the bottom view of Figure 5A is a regular blob graph waveform when having a normal pedestrian color pattern. In the case of a normal pedestrian color pattern, the waveform of the blob graph is regularly shown. This indicates that the lines are dark and suitable for image reading.

Referring to FIG. 5B, the upper diagram of FIG. 5B is the captured original pedestrian image, and the lower diagram is the image of the corrected color brightness differential object image.

In the upper diagram of Fig. 5B, the line L1 has a blurring distortion due to the external illuminance at the time of imaging. When calculating the probability of restoration for the bit group, it is determined to be a blurred line (determined as Should_line) Is restored as shown by the line L1 'in the bottom view of FIG. 5B.

In the upper diagram of Fig. 5B, lines L2 and L3 show that the distortion of the lines and the step-like lattice shape occurred at the time of imaging due to the movement of the subject. When calculating the probability of restoration for the corresponding bit group, It is determined to be a grid line (determined as maybe_line), and is restored as shown by lines L2 'and L3' in the lower diagram of FIG. 5B.

5B, the line L4 indicates that blurring of the line has occurred due to the movement of the subject. When calculating the probability after restoration for the corresponding bit group, the line L4 has a value smaller than 0.03, quot; not_line "), and restores the blank space L4 'in the lower drawing of Fig. 5B.

The color brightness differential object separation processing unit 150 may analyze the color brightness differential image included in the image corrected by the image correction unit 134. [ That is, the color brightness differential object separation processing unit 150 recognizes a moving pedestrian distinguished by the light reflection start zone and the end zone, reads the color map of the pedestrian in the image by the light reflection moving angle system conversion algorithm, And the defense rate can be generated and recorded.

The color brightness differential object separation processing unit 150 is an algorithm for converting a moving value of a pedestrian into a geometry by analyzing a pedestrian in an image.

6 is a flowchart illustrating an image processing method according to another embodiment of the present invention.

Referring to FIG. 6, in the image processing method according to the present invention, the image processing apparatus 100 performs a step S601 of receiving an image captured by the photographing apparatus 200. FIG.

The image processing apparatus 100 performs step S602 of calculating the color probability. In step 602, the color distribution of the pixels included in the received image image in at least two-dimensional color space is modeled based on the color and luminance information, and the color probability is calculated based on the modeled color distribution.

The image processing apparatus 100 performs step S603 of generating a color probability image. In step 603, a color probability image is generated using the calculated color probability.

The image processing apparatus 100 calculates a color probability using Cb, Cr, and Cr / Cb values of the pixels of the input image.

The image processing apparatus 100 performs step S604 of generating a background image. In step S604, a background image is generated by modeling the background of the received image image.

The image processing apparatus 100 performs a step S605 of generating a motion image. In step S605, a motion image is generated using the image image received in step S601 and the background image generated in step S604.

를 산출하는 단계(S605-2) 및 움직임 이미지를 제2 문턱 값으로 이진화하여

를 산출하는 단계(S605-3) 및

및

를 이용하여 최종 움직임 이미지(M(x,y))를 생성하는 단계(S605-4)를 더 수행한다.In operation S605, a motion image is generated using the received image and the background image in operation S605-1. The motion image is binarized to a first threshold value

(Step S605-2) and binarizing the motion image to a second threshold value

(S605-3) and

And

(Step S605-4) of generating a final motion image (M (x, y)) using the motion image (step S605-4).

The image processing apparatus 100 performs step S606 of generating a candidate probability image. In step S606, a candidate probability image is generated using the color probability image generated in step S603 and the final motion image generated in step S605.

The image processing apparatus 100 performs a step S607 of extracting a plurality of characteristics. In step S607, the plurality of properties for the candidate probabilistic image are determined by the lightness of the carrier including the light reflection start zone, the light reflection end zone, the width of the line of the light reflection angle, the width of the white boundary line of the object, Width and the like.

The image processing apparatus 100 performs step S608 of calculating the random probability. In step S608, the randomness test algorithm is applied to the plurality of characteristics extracted in step 607 to calculate a plurality of randomness probabilities. Step S608 includes calculating a random probability average by arithmetically averaging a plurality of random probabilities (S608-1), calculating a random probability average for the N frames of the received image data by repeating N random randomness (S608-2) of calculating a probability average, and a step (S608-3) of setting a threshold value by calculating an arithmetic average of the N random probability averages.

The image processing apparatus 100 performs step S609 of verifying the candidate probability image. In step S609, it is determined whether or not to restore the candidate probability image using the threshold value set after step S608-3. If the threshold value of the corresponding bit group is larger than 0.1, it is determined as a line. If the threshold value of the bit group is smaller than 0.1 and larger than 0.07, it is determined as a should line. If the threshold value of the bit group is smaller than 0.07 and larger than 0.05, it is judged as a lattice line (Maybe_line). If the threshold value of the corresponding bit group is smaller than 0.05, it is judged that it is not a line (Not_line).

The image processing apparatus 100 performs step S610 of correcting the candidate probability image. In step S610, image correction is performed on the candidate probabilistic image based on the verification result in step S609.

FIG. 7 is a flowchart illustrating a method of acquiring movement value information of a real-time pedestrian brightness difference object according to another embodiment of the present invention.

Referring to FIG. 7, the image processing apparatus 100 receives an image frame in real time (S701). The image processing apparatus 100 separates image frame data input by a background modeling algorithm into a background and a foreground. The image processing apparatus 100 may be configured to perform a background motion based on a background moving-image background modeling algorithm, an object moving without a large movement in a fixed place, and a minute motion due to a refraction of light on a natural illumination and an artificial illumination Forced to deny recognition by assimilation.

The image processing apparatus 100 extracts a still image of a region of interest from an image frame input in real time using an object histogram edge algorithm (S702).

The image processing apparatus 100 calculates the width, length, and RGB color information of the object by an object classification algorithm, specifies the logical coordinates of the pedestrian's name on the pixel of the differential color label, the digital 16x zoom engine is operated at the coordinates (x2, y2) and the end (x2, y2) coordinates and converted into a bitmap image, and then transferred to the memory of the system (S703).

The image processing apparatus 100 corrects the photographed image by removing the noise included in the digitally enlarged image (S704). The image processing apparatus 100 extracts a shape image included in the corrected image (S705). The image processing apparatus 100 extracts a label image from the extracted shape image (S706).

The image processing apparatus 100 extracts the pedestrian image from the label image using the background modeling algorithm (S707). The image processing apparatus 100 extracts the characteristic information of the pedestrian image, performs a randomness test based on the extracted characteristic information, and corrects image blurring and image blurring included in the extracted pedestrian image (S708).

The image processing apparatus 100 recognizes the corrected pedestrian image (S709), and converts the coordinates and the color brightness difference object information of the recognized pedestrian into text information (S710). The image processing apparatus 100 transmits the converted text information to a remote server (S711).

8 is a flow chart for explaining an object detection step of the image processing method according to an embodiment of the present invention.

Referring to FIG. 8, in the object detection step of the image processing method according to an embodiment of the present invention, an object is detected using object property learning data and data is acquired (S801) (Step S803). In the case where the object characteristic learning data does not exist, the step of detecting the object learning data includes the steps of: Extracting data (S804), continuously repeating the step of detecting the pedestrian, and recording the learned learning data using the extracted object characteristic learning data.

In the step of acquiring the data, information of the camera's Instance (RGB24 argument) is obtained, and the RGB24 information is passed as an argument when calling the CIC7P algorithm. Here, CIC7P (CIC7P (Color Intensity Classification 7 Part Based Model) algorithm is an algorithm that includes a pedestrian detection step and a pedestrian tracking step, and is a color difference degree difference equalization detection algorithm. An algorithm for detecting and tracking a pedestrian by dividing a morpheme into 7 equal parts Then, when using the CIC7P algorithm component, initialize the Instance by grouping it into one class.

In step S802 of generating a pedestrian object using the image data, only the pedestrian name difference object is generated as an independent difference image.

Thereafter, it is determined whether or not object characteristic learning data exists, and if the object characteristic learning data exists, step S803 of detecting a pedestrian based on the object characteristic learning data is performed. Further, the step of detecting the pedestrian is continuously repeated.

If the object characteristic learning data does not exist, the object learning data of the color morpheme information of the object stored in the clusters is extracted (S804). Thereafter, the learning data learned using the object learning data of the color morpheme information of the object stored for each of the extracted clusters may be recorded and terminated, or the pedestrian may be detected using the learning data again.

9 is a flow chart for explaining an object tracking step of the image processing method according to an embodiment of the present invention.

Referring to FIG. 9, in the object tracking step of the image processing method according to an embodiment of the present invention, an object is traced using object characteristic learning data and data is acquired (S901) Step S902 of extracting learning data of information (S902), predicting the object morpheme information and extracting the most similar morphological information (S903), and tracking the object based on the extracted most similar morpheme information (S904) .

In the step of acquiring the data (S901), the camera's Instance (= RGB24 argument) information is acquired, and the RGB24 information is passed as an argument when calling the CIC7P algorithm. Then, it is determined whether the image to be read is a currently displayed image. If the image to be read is not the image currently being viewed, the object tracking step is terminated. If the image to be read is the image currently being viewed, the learning data of the color morpheme information of the object stored for each cluster is extracted (S902).

Then, it is determined whether or not the object is recognized as the same pedestrian in the image input to the plurality of cameras. In order to determine that the object is the same pedestrian, the object morpheme information is predicted in advance and the most similar morpheme information is extracted (S903). The object is tracked based on the extracted most similar morpheme information (S904), and a tracked object list and a feature list can be created based on the extracted information.

The above-described methods according to various embodiments of the present invention may be stored as a code in a computer-readable storage medium. The code for performing the above-described methods according to various embodiments of the present invention may be stored in a memory such as a random access memory (RAM), a flash memory, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an Electrically Erasable and Programmable ROM ), A register, a hard disk, a removable disk, a memory card, a USB memory, a CD-ROM, and the like.

Although illustrative embodiments and applications of the present invention have been shown and described, many changes and modifications may be made without departing from the scope of the present invention, and such modifications may be made by one of ordinary skill in the art to which the present invention pertains It can be clearly understood. Accordingly, the described embodiments are illustrative and not restrictive, and the invention is not limited by the accompanying detailed description, but is capable of modifications within the scope of the claims.

100: image processing apparatus 110:
130: image processing unit 131: reflection angle moving color probability image generating unit
132: background image generation unit 133: color difference object motion image generation unit
134: Image correction unit 150: Color brightness difference image separation processing unit
170: Transmitting section 190:
200: Image pickup device 300: Video stream receiving device
400: remote server

Claims (16)

A receiving unit that receives a video image photographed by a photographing apparatus in real time;
Calculating a color probability by modeling a color distribution of pixels included in the received image image in at least two-dimensional color space based on color and luminance information, and generating a color probability image using the calculated color probability A probability image generating unit;
A background image generation unit for modeling a background of the received video image to generate a background image;
A motion image generation unit for generating a motion image using the received image and the background image;
Generating a candidate probability image using the color probability image and the motion image, extracting a plurality of characteristics for the candidate probability image, and applying a randomness test algorithm for the extracted plurality of characteristics to calculate a random probability And correcting the candidate probabilistic image by verifying the candidate probabilistic image using the calculated non-random probability,
Wherein the color probability image generating unit comprises:
Calculates a color probability based on the following equation,

(here,

Denotes the chromatic probability at the pixel of the x, y coordinate, and rate denotes the Cr / Cb at the x, y coordinate pixel)
Wherein the motion image generating unit comprises:
Generating a motion image using the received video image and the background image,
The motion image is binarized to a first threshold value

And binarizes it to a second threshold value

Lt; / RTI >
A final motion image is generated by the following equation,

Wherein the image correction unit comprises:
Calculating a plurality of randomness probabilities for the plurality of characteristics,
Calculating a random probability average using the plurality of random probabilities,
Calculating the N random probability average by iteratively calculating the random probability average over N frames of the received image data,
The random probability average is expressed as a label tag for a specific pedestrian and then stored as metadata,
Using a value calculated by averaging the N random probability averages as a threshold value for restoring a pedestrian image included in the motion image,
The stored metadata is used to search for pedestrians that are the same as the specific pedestrian in another image,
Wherein the plurality of characteristics includes seven characteristics of a pedestrian's arms, two legs, one trunk, one head, and one entire pedestrian, and the random probability average is defined by the following equation,

here,

Is the probability of randomness for the seven features in a particular frame,

Represents the random probability average.
The image correction unit may determine a restored image of the pedestrian image using the following equation,

(Where W is the restored pedestrian image,

, The line is determined as a certain line, the Should_line is determined as a blurred line, the Maybe_line is determined as a lattice line, and the Not_line is determined as a blank line)
The image corrector calculates a position and a boundary value of a pixel value of an image to be used in a ROI by using a vertical / horizontal histogram, and applies a high frequency filter interpolation method and a low frequency filter interpolation method in the vertical / horizontal histogram image processing, Sensing,
The high-frequency filter interpolation method is defined by the following equation,

here,

Is a cumulative weighting coefficient,

Is the restoration coefficient,

Represents the weighting factor.
The low-pass filter interpolation method is defined by the following equation,

here,

Lt; / RTI >
The filter interpolation area is defined by the following equation,

The high frequency filter interpolation method is used when a pixel change in an area is larger than a specific threshold value and the low frequency filter interpolation method is used when a pixel change in an area is smaller than a specific threshold value,
Wherein the image correction unit comprises:
An object detector for detecting an object using object characteristic learning data; And
And an object tracking unit for tracking an object using the object characteristic learning data,
The object detection unit generates a pedestrian object using data of the image image, extracts object characteristic learning data of color morpheme information of the object stored by the clusters, detects an object based on the extracted object characteristic learning data, The object characteristic learning data is continuously repeated, and the learned learning data is recorded using the extracted object characteristic learning data,
Wherein the object tracking unit extracts object characteristic learning data of color morpheme information of an object stored by the clusters, predicts the morpheme information of the object, extracts the most similar morpheme information, and tracks the object based on the extracted morphological information. Device.
delete delete delete delete delete delete delete Receiving a video image photographed by a photographing apparatus in real time;
Calculating a color probability by modeling a color distribution of pixels included in the received image in at least two-dimensional color space based on color and luminance information;
Generating a color probability image using the calculated color probability;
Generating a background image by modeling the background of the received video image;
Generating a motion image using the received image and the background image;
Generating a candidate probability image using the color probability image and the motion image;
Extracting a plurality of characteristics for the candidate probabilistic image;
Calculating a random probability by applying a randomness test algorithm to the extracted plurality of characteristics;
And correcting the candidate probabilistic image by verifying the candidate probabilistic image using the calculated non-random probability,
Wherein the step of generating the color probability image comprises:
Calculates a color probability based on the following equation,

(here,

Denotes the chromatic probability due to the movement of the light reflection angle in the pixels of x and y coordinates, and rate denotes Cr / Cb in the x and y coordinate pixels)
Wherein the step of generating the motion image comprises:
Generating a motion image using the received image and the background image;
The motion image is binarized to a first threshold value

;
The motion image is binarized to a second threshold value

; And
Generating a final motion image by: < EMI ID =

Wherein the correcting the image comprises:
Calculating a plurality of randomness probabilities for the plurality of characteristics;
Calculating a random probability average using the plurality of random probabilities;
Calculating N random probability averages by iteratively calculating the random probability average over N frames of the received video data;
Storing the random probability average as metadata in a label tag for a specific pedestrian;
Restoring a pedestrian image included in the motion image using a value calculated by averaging the N random probability averages as a threshold value; And
And retrieving the same pedestrian as the specific pedestrian from another image using the stored metadata,
Wherein the plurality of characteristics includes seven characteristics of a pedestrian's arms, two legs, one trunk, one head, and one entire pedestrian, and the random probability average is defined by the following equation,

here,

Is the probability of randomness for the seven features in a particular frame,

Represents the random probability average.
Wherein restoring the image comprises: determining a restoration image of the pedestrian image using the following equation:

(Where W is the restored pedestrian image,

, The line is determined as a certain line, the Should_line is determined as a blurred line, the Maybe_line is determined as a lattice line, and the Not_line is determined as a blank line)
In the step of correcting the image, a position and a boundary value of a pixel value to be used in a ROI are calculated using a vertical / horizontal histogram, and a high frequency filter interpolation method and a low frequency filter interpolation method are used in the image processing of a vertical / horizontal histogram, Lt; / RTI >
The high-frequency filter interpolation method is defined by the following equation,

here,

Is a cumulative weighting coefficient,

Is the restoration coefficient,

Represents the weighting factor.
The low-pass filter interpolation method is defined by the following equation,

here,

Lt; / RTI >
The filter interpolation area is defined by the following equation,

The high frequency filter interpolation method is used when a pixel change in an area is larger than a specific threshold value and the low frequency filter interpolation method is used when a pixel change in an area is smaller than a specific threshold value,
Wherein the correcting the image comprises:
Detecting an object using object property learning data; And
Further comprising tracking the object using the object property learning data,
Wherein detecting the object comprises:
Generating a pedestrian object using data of the image;
Extracting object characteristic learning data of color morpheme information of an object stored for each of the clusters;
Detecting an object based on the extracted object characteristic learning data;
Continuously repeating the object detection step; And
And recording the learned learning data using the extracted object characteristic learning data,
The method of claim 1,
Extracting learning data of color morpheme information of an object stored for each group;
Predicting object morpheme information and extracting the most similar morpheme information; And
And tracing the object based on the extracted most similar morpheme information. delete delete delete delete delete delete delete

Images