KR20130119741A - Apparatus and method for automatic detection of object from image taken by multiple color filter aperture camera - Google Patents

Abstract

Disclosed is an automatic object detection device and method of an image photographed by a camera having a multiple color filter aperture. A background generation unit generates a background image frame by detecting movement from a current image frame of a plurality of temporally continuous image frames which are photographed by the camera having different color filters mounted on a plurality of openings formed in the aperture thereof. An object detection unit detects an object area based on the difference between each of color channels of current image frame and each of color channels of background image frame. According to the present invention, the object can be automatically detected by the background image frame which is repetitively renewed, and the information of object can be exactly estimated by reflecting characteristics of MCA camera to separately detect the object for each color channel. [Reference numerals] (110) Background generation unit;(120) Object detection unit;(130) Color moving vector estimation unit;(140) Distance information estimation unit

Description

Apparatus and method for automatic detection of object from image taken by multiple color filter aperture camera}

The present invention relates to an automatic object detecting apparatus and method for an image captured by an imaging device having a multi-color filter aperture, and more particularly, to an imaging apparatus having an aperture provided with a plurality of color filters of different colors. The present invention relates to an apparatus and a method for automatically detecting an object region from a captured image.

Object detection and depth information estimation of images are important and difficult problems in video analysis applications such as security systems, intelligent traffic systems, three-dimensional digital cameras and advanced safety systems for vehicles.

Since a two-dimensional image obtained by a general camera does not provide enough information for estimating depth information, research on estimation of three-dimensional depth information has been made for a long time. The most common depth information estimation technique is to use a plurality of images such as stereo vision or additional signals such as contrast, focus and motion.

Stereo matching is a depth information estimation method that uses binocular disparity generated by a stereo camera. Although this method has many advantages, it has a fundamental limitation that it requires images of the same scene obtained by two cameras synchronized in time and space.

As an alternative to such a binocular system, research on a monocular system has also been conducted. Depth information estimation from defocus is a method of estimating depth information based on a single camera, and is a method of measuring the amount of defocus from images captured at different focus settings for the same scene. However, this method has a disadvantage in that it can be applied only to still pictures because the camera's vision must be fixed to capture a plurality of defocused images.

In recent years, computational cameras have been developed to provide new information that cannot be obtained from existing digital cameras, thereby providing new possibilities in consumer video equipment. Computational cameras use a combination of new optics and calculations to generate the final image, which has created new imaging features that traditional cameras could not achieve, such as improved field of view, increased spectral resolution, and increased dynamic range. .

On the other hand, the color shift model using a multiple color filter aperture (MCA) installed with a plurality of color filters, the depth of the objects located at different distances from the camera depending on the relative movement direction and the amount of movement between the color channels of the image Information can be provided. However, existing MCA-based depth information estimation methods require a process of manually selecting an object part in an image in order to estimate depth information of an object.

Korean Patent Laid-Open Publication No. 2009-0083030 discloses an aperture device in which a plurality of color filters of different colors are installed, and a method of restoring the focus of an image by moving a color channel in an image obtained by a camera equipped with the aperture device. . As such, there is a need for a method of estimating depth information of an object in an image based on movement of a color channel used for focus reconstruction of an image, and automatically detecting an object.

The technical problem to be achieved by the present invention, as described above, is provided with a multi-color filter aperture that can automatically detect the object in the image is refocused by the movement characteristics of the color channel, and can estimate the depth information of the detected object An object and method for detecting an automatic object of an image photographed by an imaging device is provided.

Another technical problem to be solved by the present invention is to provide a multi-color filter aperture capable of automatically detecting an object in an image whose focus is restored by the movement characteristics of a color channel and estimating depth information of the detected object. The present invention provides a computer-readable recording medium having recorded thereon a program for executing an automatic object detection method of an image photographed by an imaging device.

In order to achieve the above technical problem, an automatic object detecting apparatus for an image photographed by an imaging apparatus having a multi-color filter aperture according to the present invention is provided in an imaging apparatus in which different color filters are respectively provided in a plurality of openings formed in the aperture. A background generator configured to generate a background image frame corresponding to the current image frame by detecting a movement from the current image frame among a plurality of image frames that are photographed by time; And an object detector configured to detect an object area included in the current image frame based on a difference between each of the plurality of color channels of the current image frame and each of the plurality of color channels of the background image frame.

In order to achieve the above technical problem, an automatic object detection method of an image photographed by an imaging device having a multi-color filter aperture according to the present invention is provided in an imaging apparatus in which different color filters are respectively provided in a plurality of openings formed in the aperture. A background generation step of generating a background image frame corresponding to the current image frame by detecting a movement from the current image frame among a plurality of image frames that are photographed by time; And an object detecting step of detecting an object region included in the current image frame based on a difference between each of the plurality of color channels of the current image frame and each of the plurality of color channels of the background image frame.

According to the automatic object detection apparatus and method of the image photographed by the imaging device having a multi-color filter aperture according to the present invention, it is possible to automatically detect the object by the background image frame is repeatedly updated, the characteristics of the MCA camera By reflecting this, the object information can be accurately estimated by detecting the object separately for each color channel. In addition, it is possible to estimate the actual distance information from the camera to the object by using the property that a different color movement vector is obtained according to the position of the object.

1 is a diagram illustrating a structure of an MCA camera;
2 is a view illustrating characteristics of an image according to a distance of an object from an MCA camera;
3 is a block diagram showing the configuration of a preferred embodiment of an automatic object detection apparatus for an image captured by an imaging apparatus having a multi-color filter aperture according to the present invention;
4 is a view showing an embodiment of object detection according to the present invention;
5 is a diagram illustrating a positional relationship and color shift vectors between color channels;
FIG. 6 is a diagram illustrating an image captured by an MCA camera of two objects located at different distances;
7 is a view showing an image obtained by continuously shooting while moving the object in the distance range of 17 to 49 meters from the MCA camera,
8 is a graph showing normalized magnitudes of components of a color shift vector estimated in successive image frames;
9 is a diagram illustrating an embodiment of detecting an object region and estimating distance information from an image frame including a plurality of moving objects;
10 is a flowchart illustrating a preferred embodiment of a method for automatically detecting an object of an image captured by an imaging device having a multi-color filter aperture according to the present invention.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the automatic object detection apparatus and method of the image photographed by the imaging device having a multi-color filter aperture according to the present invention.

In order to explain the detailed configuration and operation of the present invention, the operation of the present invention will first be described after explaining the principle of an imaging apparatus (hereinafter referred to as an "MCA camera") having a multi-color filter aperture which can be referred to as the background of the present invention. Will be described in detail for each component.

1 is a diagram illustrating a structure of an MCA camera.

Referring to FIG. 1, three apertures are formed in the aperture inserted between the lenses of the MCA camera, and each aperture is provided with different color filters of red (R), green (G), and blue (B). . The aperture is also provided such that the center between the three openings coincides with the optical axis of the camera.

The light passing through the color filter installed in each opening of the aperture is formed at different positions of the camera sensor according to the distance between the lens and the object. When the object is located at a point away from the focal length of the camera, the color in the obtained image is obtained. Color deviation will occur.

FIG. 2 is a view showing characteristics of an image according to an object's distance from an MCA camera. FIG. 2 (a) shows when the object is located at a point far from the focal length of the camera. (C) shows the case where the object is located closer to the focal length of the camera.

Referring to FIG. 2, when the position of the object coincides with the focal length of the camera as shown in (b), color deviation does not occur in the image formed by the sensor. However, when the object moves away from or close to the focal length, As shown in a) and (c), it can be seen that a deviation occurs between the respective color channels in the image. Therefore, even in one image, a difference occurs in the position of the region corresponding to the object in each color channel.

The present invention has a configuration that automatically detects the object from the image by reflecting the color deviation appearing in the image taken by the MCA camera, and also estimates the distance information from the MCA camera to the object based on the degree of color deviation. .

3 is a block diagram showing the configuration of a preferred embodiment of an automatic object detection apparatus for an image captured by an imaging apparatus with a multi-color filter aperture according to the present invention.

Referring to FIG. 3, the automatic object detecting apparatus according to the present invention includes a background generating unit 110, an object detecting unit 120, a color motion vector estimating unit 130, and a distance information estimating unit 140.

The background generator 110 detects a motion from a current video frame among a plurality of video frames that are photographed by an MCA camera and generate a background video frame corresponding to the current video frame. That is, the automatic object detecting apparatus according to the present invention may generate a background and detect an object in real time for each image frame of a video image composed of a plurality of consecutive image frames.

The background generator 110 may estimate the motion of the current image frame using optical flow to generate a background image frame corresponding to the current image frame. The optical flow information corresponding to each pixel of the current image frame may be obtained from a relationship between the current image frame and the previous image frame temporally preceding the current image frame as shown in Equation 1 below.

Here, D (x, y) is the current video frame (x, y) optical flows corresponding to the pixel information, f _t is the current image frame, f _t-1 is the previous image frame, (d _x, d _y) is ( x, y) represents a shift in pixels, and minimizes D (x, y). In addition, in Equation 1, the size of the search area is set as (2w + 1) × (2w + 1).

If the value of the optical flow information D (x, y) in the (x, y) pixel of the current image frame is smaller than a preset Euclidean distance threshold, the pixel is determined to belong to the background, and the background generator 110 The background image frame generated corresponding to the previous image frame is updated using the pixels determined to belong to the background in the image frame.

Here, f _B ^t and f _B ^t-1 represent a background video frame corresponding to the current video frame and a background video frame corresponding to the previous video frame, respectively, and α is a mixture ratio preset in the range of [0,1]. to be.

The object detector 120 detects the object area included in the current image frame based on the difference between the background image frame and the current image frame of the current image frame. In this case, in the conventional methods, only the difference between image frames is calculated for object detection, whereas the object detector 120 of the automatic object detecting apparatus according to the present invention is configured between a plurality of color channels constituting the current image frame and the background image frame. By calculating the difference, the object region is detected for each color channel of the current video frame.

That is, when the difference between the R channel of the current video frame and the R channel of the background video frame is calculated, an object region corresponding to the R channel of the current video frame is obtained. The area is obtained. As described above, as the object region is detected for each color channel of the image frame, the characteristics of the MCA camera as shown in FIG. 2, that is, the color deviation that appears when the position of the object does not match the focal length may be reflected in the object detection process. Can be.

In detail, the object detector 120 may detect the object region from the current image frame by Equation 3 below.

here,

f _O ^c is a binary image corresponding to the color channel of the current image frame, and represents an object region in which pixels having a value of 1 in f _O ^c are detected from the corresponding color channel.

After detecting the object region as described above, the object detector 120 may additionally remove noise by using the same object morphological filter on the object regions positioned at similar points in each color channel.

4 is a diagram illustrating an embodiment of object detection according to the present invention, in which (a) of FIG. 4 is a current image frame, (b) is a background image frame corresponding to the current image frame, and (c) is a detected object. Area, and (d) shows the result of applying a mask to the object area.

As shown in (a) of FIG. 4, a plurality of objects are included in the current image frame. Referring to FIG. 4 (c), the focal point is focused by the color deviation characteristic according to the distance of the object from the MCA camera described above. The color deviation does not appear in the right object area, but the deviation between the color channels occurs in the unfocused object area.

Meanwhile, the automatic object detecting apparatus according to the present invention uses the color shift degree included in the object region detected by the object detecting unit 120 to obtain distance information, that is, depth information from an MCA camera to an object corresponding to the object region. It can be estimated.

In order to estimate the depth information of the object, as described above, the channel alignment process should be performed on the object region where the color channel misalignment occurs. The alignment process of a color channel includes a color shift vector representing direction and distance information of another color channel (eg, R channel and B channel) about a specific color channel (eg, G channel). By estimating CSV).

5 is a diagram illustrating a positional relationship and color shift vectors between color channels. As shown in (a) of FIG. 5, each color channel of the aperture of the MCA camera is located at each vertex of an equilateral triangle. Using this characteristic, depth information can be accurately obtained while reducing the amount of computation for estimating depth information of an object. It can be estimated. The color motion vector estimation and distance information estimation to the object described below are performed separately for each object region when a plurality of object regions are detected from the current image frame.

In detail, the color shift vector of the R channel and the B channel centering on the G channel in the i-th object area of the plurality of object areas may be expressed by Equation 4 below.

및

은 각각 GB 채널(G 채널과 B 채널)에 대한 색상 이동 벡터 및 GR 채널(G 채널과 R 채널)에 대한 색상 이동 벡터를 나타낸다. 또한 도 5의 (a)와 같은 MCA 카메라의 특성으로 인해 수학식 4에 나타난 두 색상 이동 벡터는 다음의 수학식 5와 같은 관계를 가진다.here,

And

Denotes a color shift vector for the GB channel (G channel and B channel) and a color shift vector for the GR channel (G channel and R channel), respectively. In addition, due to the characteristics of the MCA camera as shown in (a) of FIG. 5, the two color shift vectors shown in Equation 4 have the same relationship as in Equation 5 below.

및

은 다음 수학식 6의 2차 에러 함수를 최소화함으로써 추정될 수 있다.Color shift vector

And

Can be estimated by minimizing the quadratic error function of Equation 6 below.

Here, E ^GB represents an error function corresponding to the color shift vector of the GB channel, E ^GR represents an error function corresponding to the color shift vector of the GR channel, and Ω represents an object region. In addition, referring to Equation 6, an error function corresponding to the color shift vector of the GR channel may be represented by the color shift vector of the GB channel with reference to the relationship between the color shift vectors described above.

의 비선형 함수이므로, 수학식 6을 최소화하는

를 찾기 위해 뉴튼-랩슨(Newton-Raphson) 알고리즘과 같은 반복 접근법을 사용할 수 있다.As a result, the error function of Equation 6

Since it is a nonlinear function of, minimizing

We can use an iterative approach such as the Newton-Raphson algorithm to find.

The first order Taylor series estimation for the error function of Equation 6 may be expressed as Equation 7 below.

에 대하여,

와 같이 산출되며,

와

는 각각

의 수평 및 수직 방향 도함수이다.Where color channel

about,

Is calculated as

Wow

Respectively

Is the derivative of the horizontal and vertical directions.

In addition, when the estimated error is expressed in a vector form, the following Equation 8 is obtained.

,

및

와 같이 정의된다.Where s = f _t ,

,

And

Respectively.

E (v) because it is a quadratic function of the vector v, v that minimizes the error may be obtained by finding the value of making the results of the differential for the error function to v, as shown in the following equation (9) to zero.

Since Equation 9 is a linear equation, the vector v can finally be obtained as in Equation 10 below.

이고,

이다. 또한 검출된 객체영역의 크기가 충분히 크며 영상에 충분한 내용이 포함되어 있다면, 수학식 10에서 행렬 C는 역행렬이 존재하는 것으로 볼 수 있다.here,

ego,

to be. In addition, if the size of the detected object region is large enough and the image contains sufficient content, it can be seen that the matrix C in Equation 10 exists as an inverse matrix.

는 0이 된다. 따라서 벡터 v는 도 5의 (b)에 도시된 것과 같은 삼각 특성 및 조리개의 컬러 필터 사이의 각도를 사용하여 단일 파라미터

에 의해 표현될 수 있으며, 이는 다음의 수학식 11과 같다.On the other hand, Equation 10 may be further simplified based on the characteristics of the MCA camera. If the horizontal axes of the G and B channels are the same, the vertical component of the color shift vector

Becomes zero. Thus, the vector v is a single parameter using the angle between the triangular characteristic and the color filter of the aperture as shown in Fig. 5 (b).

It can be expressed by, which is represented by the following equation (11).

,

이고,

이다.here,

,

ego,

to be.

The numerator and denominator of Equation 11 are all 1x1 matrices, and the final motion vector v which is a combination of estimated color motion vectors for each color channel can be estimated without using an inverse matrix.

FIG. 6 is a diagram illustrating an image captured by an MCA camera of two objects located at different distances. (A) of FIG. 6 shows the movement of the estimated color channels with respect to the two objects A and B included in the image, and (b) and (c) show the color channels around the A and B objects, respectively. Shows sorted results.

이며, 도 6의 (c)에서 B 객체에 대응하는 영역에 대하여 추정된 최종 이동 벡터는

이다. 이와 같이 A 객체와 B 객체가 MCA 카메라로부터 서로 다른 거리에 위치함에 따라 상이한 이동 벡터가 얻어지므로, A 객체와 B 객체 사이의 상대적인 거리를 추정하는 것이 가능하다.In FIG. 6B, the final motion vector estimated for the area corresponding to the A object is

In FIG. 6C, the final motion vector estimated for the region corresponding to the B object is

to be. In this way, since different motion vectors are obtained as the A and B objects are located at different distances from the MCA camera, it is possible to estimate the relative distance between the A and B objects.

The automatic object detecting apparatus according to the present invention may further include a distance information estimator 140 for estimating absolute distance information from the MCA camera to the object, and the distance information estimator 140 may have a magnitude of the final motion vector v . The distance information between the object included in the object area and the MCA camera is estimated based on the information.

In detail, a conversion function may be set that indicates a relationship between the distance to the object and the movement amount of the color channel, that is, the magnitude of the movement vector. The transform function may be obtained by locating objects from a MCA camera at a predetermined distance, and then repeatedly capturing the same scene including the object at each position of each object to estimate a color shift vector.

7 is a view showing an image obtained by continuously photographing while moving the object in the distance range of 17 to 49 meters from the MCA camera. (A) to (c) of FIG. 7 show the 30th, 330th, and 660th image frames, respectively, obtained sequentially in time, and (d) to (f) of (a) to (c). The color channel alignment is performed on the object region detected from the image frame.

및

가 추정될 수 있으며, 도 8은 연속적인 영상프레임마다 추정되는 색상 이동 벡터의 각 성분의 크기를 정규화하여 도시한 그래프이다. 도 8의 (a)는 영상프레임의 번호에 따른 색상 이동 벡터의 크기 정보를 도시한 것이고, (b)는 MCA 카메라로부터 객체까지의 거리에 따른 색상 이동 벡터의 크기 정보를 도시한 것이다.Two components of the color shift vector in this color channel alignment process

And

8 may be estimated. FIG. 8 is a graph showing normalized sizes of components of the color motion vector estimated for each successive image frame. FIG. 8 (a) shows size information of the color motion vector according to the number of the image frame, and (b) shows size information of the color motion vector according to the distance from the MCA camera to the object.

Referring to FIGS. 7 and 8 (a), it can be seen that as the object gets closer to the focal position (about 21 meters) of the MCA camera, that is, as the number of image frames increases, the magnitude of the motion vector converges to zero. have. When the object approaches the MCA camera past the focal position, the magnitude of the motion vector is diverged as shown in (a) of FIG. 8. 8 (b) shows the size of the motion vector by quantizing the distance from the MCA camera to the object in units of 1 meter.

When the graph as shown in FIG. 8B is constructed in advance, the distance information estimator 140 substitutes the size information of the motion vector corresponding to the object region detected from the current image frame into the graph and includes it in the object region. Accurate distance information to the object can be estimated.

9 is a diagram illustrating an embodiment of detecting an object region and estimating distance information from an image frame including a plurality of moving objects.

(A) of FIG. 9 shows each image frame of the original image, (b) shows that the object region is detected from each image frame, and (c) shows that the color channel alignment process corresponds to each object region. (D) shows the result of the distance information is estimated for each object area.

Referring to FIG. 9, even when a plurality of objects are included in an image, since different color motion vectors are estimated for each object, distance information may be estimated for each object.

10 is a flowchart illustrating a preferred embodiment of a method for automatically detecting an object of an image captured by an imaging device having a multi-color filter aperture according to the present invention.

Referring to FIG. 10, the background generator 110 detects a motion from a current video frame among consecutive video frames captured by an MCA camera and generates a background video frame corresponding to the current video frame (S1010). In addition, the object detector 120 detects an object area included in the current image frame based on the difference between each of the plurality of color channels of the current image frame and each of the plurality of color channels of the background image frame (S1020). Accordingly, the present invention can detect the object region in real time whenever an image frame is input, and this process can be performed automatically without specifying the object part in advance.

Furthermore, the color shift vector estimator 130 estimates a color shift vector representing a direction and distance of movement between object regions detected from each color channel of the current image frame, and estimates the color corresponding to each color channel. The final motion vector corresponding to the object region is calculated by combining the motion vectors (S1030).

In addition, the distance information estimator 140 may estimate distance information of the object included in the object region based on the size information of the final motion vector (S1040). In this case, as described above, it is preferable that a conversion function between the magnitude and the distance information of the motion vector is set in advance.

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may be implemented in the form of a carrier wave (for example, transmission via the Internet) . The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation in the embodiment in which said invention is directed. It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the appended claims.

110-background generator
120-object detector
130-color shift vector estimator
140-distance information estimator

Claims (13)

The image capturing apparatus, which is photographed by a different color filter in each of the plurality of openings formed in the aperture, detects a motion from a current video frame among a plurality of video frames that are temporally continuous to generate a background video frame corresponding to the current video frame. A background generator; And
And an object detector configured to detect an object region included in the current image frame based on a difference between each of the plurality of color channels of the current image frame and each of the plurality of color channels of the background image frame. Detection device. The method of claim 1,
Estimating a color shift vector representing a direction and distance of movement between the object regions detected from each color channel of the current image frame, and combining the estimated color shift vectors corresponding to each color channel to the object region. A color motion vector estimator for calculating a final motion vector corresponding to the color shift vector; And
And a distance information estimator for estimating distance information between the object included in the object area and the image pickup device based on the size information of the final motion vector. 3. The method of claim 2,
And the color motion vector estimator calculates a vector for minimizing an error function representing a deviation between the color channels represented by the color motion vectors and determines the final motion vector. 3. The method of claim 2,
And the distance information estimating unit estimates the distance information based on a preset function of the magnitude information of the final motion vector and the actual distance to the object. 3. The method of claim 2,
The object detector detects a plurality of object regions from the current image frame,
The color motion vector estimator calculates a final motion vector corresponding to each of the plurality of object regions,
The distance information estimating unit estimates distance information of an object included in each of the plurality of object areas. 6. The method according to any one of claims 1 to 5,
The background generator updates the background image frame by adding pixels having a smaller motion size than a preset threshold among pixels of the current image frame to a background image frame corresponding to a previous image frame temporally preceding the current image frame. Automatic object detection apparatus characterized in that. The image capturing apparatus, which is photographed by a different color filter in each of the plurality of openings formed in the aperture, detects a motion from a current video frame among a plurality of video frames that are temporally continuous to generate a background video frame corresponding to the current video frame. Background generation step; And
And an object detecting step of detecting an object region included in the current image frame based on a difference between each of the plurality of color channels of the current image frame and each of the plurality of color channels of the background image frame. Object detection method. 8. The method of claim 7,
Estimating a color shift vector representing a direction and distance of movement between the object regions detected from each color channel of the current image frame, and combining the estimated color shift vectors corresponding to each color channel to the object region. A color motion vector estimating step of calculating a final motion vector corresponding to; And
And a distance information estimating step of estimating distance information between the object included in the object area and the image pickup device based on the size information of the final motion vector. The method of claim 8,
In the color motion vector estimation step, an automatic object detection method comprising determining a final motion vector by calculating a vector for minimizing an error function representing a deviation between the color channels represented by each color motion vector. . The method of claim 8,
And in the distance information estimating step, estimating the distance information based on a predetermined conversion function between the size information of the final motion vector and the actual distance to the object. The method of claim 8,
In the object detecting step, detecting a plurality of object areas from the current image frame,
In the color motion vector estimating step, a final motion vector corresponding to each of the plurality of object regions is calculated,
In the distance information estimation step, the automatic object detection method characterized in that for estimating the distance information of the object included in each of the plurality of object areas. 12. The method according to any one of claims 7 to 11,
In the background generating step, the background image frame is added by adding pixels having a motion size smaller than a preset threshold among pixels of the current image frame to a background image frame corresponding to a previous image frame temporally preceding the current image frame. Automatic object detection method characterized in that for updating. A computer-readable recording medium having recorded thereon a program for executing the automatic object detecting method according to any one of claims 7 to 11.

Images