KR100220838B1 - Apparatus and method of target tracking in image telephone - Google Patents

Abstract

The present invention relates to a target tracking method and apparatus for moving a position of a camera by randomly setting the shape of an effective correlation area using a quadtree and merging to track a target, .

The present invention arbitrarily sets a correlation area using a quadtree and a merge to track a talker, and calculates the correlation and accordingly moves the position of the camera.

Therefore, the present invention estimates the correlation using the correlation area formed by the quad tree partitioning and merging instead of the conventional square window, so that it has the effect of reducing the calculation amount while improving the tracking performance for tracking the target.

Description

A method and apparatus for tracking a target of a videophone (Target Tracking Method and Device for Video Phone)

The present invention relates to a target tracking method and apparatus for moving a position of a camera by randomly setting the shape of an effective correlation area using a quadtree and merging to track a target, .

In general, tracking methods for tracking an object, ie, a target, include a center point tracking method and a correlation tracking method.

The center point tracking method is a method of tracking the center point A of the moving object after separating the moving object from the background as shown in FIG. 1A. At this time, the threshold value is used to separate the moving object, that is, the target, from the background. That is, the background and the object are binarized using the threshold value.

However, this central tracking technique has a disadvantage that it is highly affected by noise.

That is, if the image is relatively simple and the image area is easy and the restriction on the speed of the trackable object is relatively small, the tracking stability is good and the influence of noise is small. On the other hand, if the image is relatively complicated and the image area is not easy and the constraint on the speed of the trackable object is relatively large, the tracking stability is bad and the noise is increased.

1B, the correlation tracking method defines a moving object of a previous image, that is, a region B having a size appropriate to the position of the target, and calculates a correlation between the defined region B and the search region in the current image And estimates that the object has moved to the region B 'having the highest degree of correlation.

That is, the correlation tracking method defines a window region of a fixed size including a moving object, that is, a correlation region when a position of a moving object is given in a given n-th image, The position of the region with the highest degree of correlation is regarded as the position of the moving object in the (n + 1) -th image.

Here, in calculating the correlation, the area of the initial window, that is, the area used in calculating the correlation between the current frame and the previous frame, is mainly a square window.

Therefore, the correlation tracking method calculates the correlation directly from the image of the current frame without performing the image segmentation process, so that the tracking performance is maintained even for a relatively complicated image, but the calculation amount is increased.

In other words, since the correlation tracking method generally uses the intensity information of the image without binarizing the image as compared with the center-point tracking method, it is expected that some degree of tracking performance can be expected even when the image area can not be added by adding background scattering. However, the correlation tracking method has a disadvantage in that the computation amount is increased because the correlation degree must be calculated for all cases between the correlation region and the search region in order to estimate the movement of the moving object.

The present invention for improving the above-mentioned disadvantages of the present invention is that, in a videophone, a shape of an effective correlation area is arbitrarily set by using a quadtree and merging to track a talker, The object of the present invention is to provide a target tracking method and apparatus for improving tracking performance while reducing computational complexity.

1A is a view for explaining a conventional center point tracking technique;

1B is a diagram for explaining a conventional correlation tracking technique;

2 is a flowchart of a target tracking method according to the present invention.

FIG. 3 is a view for explaining the quad tree segmentation step of FIG. 2; FIG.

4 is a view for explaining the merging step of FIG. 2; FIG.

5 is a view for explaining the correlation degree calculating step of FIG. 2; FIG.

FIG. 6 is a block diagram of a target tracking apparatus according to the present invention.

DESCRIPTION OF THE REFERENCE NUMERALS

300: camera 400: motor part

410: motor driver 420: motor

500: control unit 600: image storage unit

610: A / D converter 620 630: Memory

700: correlation area determination unit 710: quad tree partition unit

720: merging unit 730: correlation area extracting unit

800: Correlation calculation unit 900: Motion estimation unit

In order to achieve the above object, according to the present invention, there is provided a tracking method of a target object of a video telephone, comprising the steps of: dividing a brightness value of each pixel in an initial window around a target of an initial image input in a tracking mode, Area setting step; A retrieval region extracting step of extracting a retrieval region for retrieval from an input current image; A motion estimation and motor driving step of estimating a motion according to the degree of correlation between the determined correlation area and the extracted search area and moving the camera, And a call termination step of repeating the driving step.

According to another aspect of the present invention, there is provided an apparatus for tracking a target object of a video telephone, comprising: image storage means for A / D converting an image input in a tracking mode; A correlation area determination unit for determining a correlation area by dividing and merging brightness values of each pixel in an initial window around a target of an initial image stored in the image storage unit; Correlation calculation means for extracting a search region from a current image output from the image storage means according to a correlation region determined by the correlation region determination means and calculating a correlation degree; A motion estimation means for estimating a motion according to the correlation calculated by the correlation calculation means; And control means for controlling the operation of the image storage means and moving the camera in accordance with the motion estimated by the motion estimation means.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2, the target tracking method of a video telephone according to the present invention includes a correlation area setting step 100, 101, 102, 103, 104 and 105, a search area extraction step 106 and 107, The driving steps 108, 109 and 110, and the call termination steps 111 and 112, respectively.

In the correlation area setting step 100, 101, 102, 103, 104, and 105, the brightness values of each pixel in the initial window are divided and merged with the target of the initial image input in the tracking mode to determine a correlation area An initial image storage step (100, 101) for converting an initial image input in a tracking mode into an A / D (Analog / Digital) and storing the initial image, an initial window setting step of setting an initial window at a position of a target in the stored initial image A quad tree dividing step 103 for dividing the set initial window into a quad tree using a homogeneity check function according to a brightness value of each pixel, A merging step 104 for merging the squares adjacent to the merged rectangle according to the variance, and a correlation area determination step 105 for determining the merged rectangle as the correlation area.

Here, the initial image is an image taken by the camera when the target is at a preset position.

In the search region extracting step (106, 107), a search region for search is extracted from the input current image.

Here, the search area is composed of pixels added by 8 pixels in the up, down, left, and right directions on the basis of the correlation area extracted from the previous image.

In the motion estimation and motor driving steps (108, 109, 110), motion is estimated according to the correlation between the determined correlation area and the extracted search area, and the camera is moved.

Here, the correlation function for calculating the correlation is MAE (Mean Absolute Error).

In the call termination steps 111 and 112, the search region extraction step, the motion estimation and the motor driving step are repeated for the image input until the call is terminated.

Hereinafter, a method for tracking a target of a video telephone according to the present invention will be described in detail with reference to the accompanying drawings.

First, the correlation area setting step (100, 101, 102, 103, 104) is performed to determine a correlation area for tracking, which will be described in detail below.

Since there may be times when a call is initiated and the user does not want to use the tracking function depending on the situation, the user first decides whether to use the tracking function (100). The distinction between trace mode and non-trace mode can be implemented simply with a switch. That is, if the switch is ON, the tracking function is performed. If the switch is OFF, the tracking function is not performed.

And an initial image storage step (100, 101) in which an initial image input from a camera is A / D (Analog / Digital) converted and stored in a tracking mode for performing a tracking function by turning on a switch.

That is, the NTSC signal coming from the camera is output as an 8-bit digital signal through the A / D converter. At this time, a high-bit A / D converter may be used. Since this digital signal is a video signal, it is stored as a digital image I (x, y) having a value within a specified range on the two-dimensional matrix on the memory.

This digital image is an image with a size of 64 x 64 pixels, which is stored in memory and used for target tracking.

In the correlation tracking, it is assumed that the initial position, that is, the position where the target caller can enjoy the dial to make a call, is known by the locking process of the initial target. Thus, in this initial position, the initial image taken by the camera can be taken when the target is at a preset location for the call.

An initial window setting step 102 is performed to set a window made up of a square of 16 x 16 pixels including a target person's face as an initial window from the initial image taken as described above.

That is, an initial window composed of 16 x 16 pixels including the face of the caller is set from the initial image of 64 x 64 pixels size, and the size of the initial window can be set variously.

After the initial window is set, the quad tree segmenting step 103 is performed to divide the set initial window into quad-trees using the homogeneity measure according to the brightness values of the pixels.

That is, the initial window is divided according to the uniformity check function using a quadtree partitioning technique. The uniformity check function uses variance as shown in Equation (1) below.

υ = 1 / N ΣΣI (x, y) - M [Expression 1]

x y

In Equation (1), N is the number of pixels in the window, M is the average of pixel brightness inside the target window, and I (x, y) is the brightness value of each pixel.

As the variance value (υ), which is a result of the calculation according to the above formula (1), becomes smaller, the pixels in the window have a brightness value similar to the average, so that the distribution of brightness values is uniform.

Therefore, if the variance value (v) calculated by applying the above uniformity check function is greater than a threshold value that is set, and if the variance value (v) is not greater than the threshold value, the process is repeated.

That is, as shown in FIG. 3, when the variance value (v) is calculated by the above-mentioned (Formula 1) for the window C to be the object, (D) determining that the brightness value of a pixel in the window C is uniform if the variance value (v) is less than the threshold value set, and does not divide the window (C) into a quad tree.

The above process is repeated to make the target window one of the quad-tree divided squares D, and the quadtree is divided according to the variance value (υ) calculated by the above (Equation 1).

After the quad tree segmentation step 103 is performed, a merging step 104 is performed to merge the quad-tree segmented quadrangle in which the target is located and the neighboring quadrangles adjacent to the quad tree segment.

That is, assuming that the quad tree is divided as shown in FIG. 4 and the current talker position is an 'A' quadrangle, the quadrangle adjacent to the periphery around the quadrangle (F) where the talker is located is divided into 7 quadrangles (Q1, Q2, Q3 , Q4, Q5, Q6, Q7). The merging process of the rectangle (F) and the surrounding rectangles is performed.

The merging is performed by calculating the variance between the rectangle (F) in which the dialogue character is located and each of the adjacent rectangles (Formula 1), and when the calculated variance value (v) is equal to or less than the threshold value, two rectangles are added. At this time, the area where? Is made is the sum of two squares.

For example, as shown in FIG. 4, the variance value (v) is calculated for each of the quadrangles (F) and adjacent quadrangles (Q1, Q2, Q3, Q4, Q5, Q6, Q7) When the variance value (v) with the quadrangles (Q1, Q3, Q5, Q7) becomes equal to or less than the set threshold value, it is merged.

When the rectangles are merged as described above, a correlation area determination step 105 is performed to determine the area formed by the merged rectangles as a correlation area.

That is, as shown in FIG. 5A, the region including the merged region, that is, the rectangle (F, Q1, Q3, Q5, Q7) is determined as the correlation region (105).

After the correlation area is determined as described above, a search area extraction step (106, 107) is performed to extract a search area for searching from the input current image.

That is, the input current image is A / D converted (Analog / Digital) and stored (106). As shown in FIG. 5 (b), in the current image, An area made up of pixels added by 8 pixels, that is, an area made up of 32 x 32 pixels centering on an initial window made up of 16 x 16 pixels, is extracted as a search area.

After the search area is extracted as shown in FIG. 5 (b), the motion estimation and motor drive steps 108, 109, and 110 are performed.

First, the degree of correlation between the determined correlation region and the extracted search region is calculated (108).

In order to calculate the correlation between the correlation region of the initial image and the search region of the current image, a correlation function to be used must be determined.

Correlation functions for the correlation calculation include NCCF (Normalized Cross Correlation Function), MSE (Mean Square Error), and MAE (Mean Absolute Error) as shown in Equation 2, Equation 3 and Equation 4 below. ).

[Formula 2]

NCCF (p, q) = [ ΣI n (i, j) · I n + 1 (i + p, j + q)] / [(ΣI n 2 (i, j)) 1/2 · (ΣI n + ₁ ² (i + p, j + q)) ^1/2 ]

MSE (p, q) = E ([I n (i, j) - I n + 1 (i + p, j + q)] 2) [ Equation 3]

MAE (p, q) = E (┃I n (i, j) - I n + 1 (i + p, j + q) ┃) [ Expression 4]

Where E (·) represents the mean.

As the correlation function, NCCF is most suitable but MAE is used considering the calculation amount.

The higher the correlation between images, the smaller the value of MAE is. Therefore, when MAE is used as a correlation function, the position with the lowest value becomes the moving object in the next image, that is, the estimated position of the target.

According to the correlation function, the correlation between the search area of the current image and the correlation area of the initial image, which is the previous image, is calculated, and the part having the highest correlation is considered as the target of the search area, It is calculated by the difference of the current position.

That is, the movement of the face of the talker, which is the target, is estimated by calculating the difference between the position of the correlation area of the initial image, which is the previous image, and the position of the part having the highest correlation in the search area.

At this time, the motor is driven according to the estimated estimated movement, that is, the talker moves to move the camera (110).

At this time, if a new image is input from the camera, a new image is inputted and then an image is stored (step 106).

The video signal of one frame is stored in one memory and then shifted to the next memory and used as a previous video. That is, the video signals of the current video and the previous video are stored in the two memories, respectively.

When the video signals of the previous video and the current video are stored in the memory, the search area is extracted (107), the correlation is calculated and the motion is estimated (108, 109).

At this time, the portion having the highest correlation degree extracted in the motion estimation step 109, which is the highest correlation degree extracted from the previous image, can be extracted and set as a correlation region.

On the other hand, if a new image is not input from the camera, a call termination step of terminating the tracking function is performed by searching whether the call is terminated.

That is, when a new image is input, the process proceeds to the image storage step 106 and repeats the above process. When the new image is not inputted and the call is terminated, the tracking function is completed (111, 112).

If the call is not terminated but a new image is not input, the system waits until a new image is input.

6, the apparatus for tracking a target object of a video telephone according to the present invention includes an image storage unit 300, a correlation area determination unit 700, a correlation calculation unit 800, a motion estimation unit 900, A motor unit 400, and a control unit 500

The image storage unit 600 A / D converts the image input in the tracking mode and stores the image. The A / D converter 610 performs A / D conversion on the image input in the tracking mode, the A / D converter 610 And a second memory 630 for storing a video signal output from the first memory 620. The first memory 620 stores a current video signal outputted from the first memory 620,

The correlation area determination unit 700 determines a correlation area by dividing and merging the brightness values of each pixel in the initial window centered on the target of the initial image stored in the image storage unit 600, An initial window is set at the position of the target in the initial image stored in the second memory 630 of the quad-tree partitioning unit 600, and a quad-tree partitioning unit, which performs a quad-tree division using a homogeneity check function according to the brightness value of each pixel, A merging unit 720 for merging a quadrangle in which the target is located and a quadrangle adjacent to the target in a quad tree divided by the quad tree dividing unit 710, And a correlation region extracting unit 730 for extracting a region formed by the output merged rectangle as a correlation region.

Here, the initial image is an image taken by the camera when the target is at a preset position.

The correlation calculation unit 800 extracts a search region from the current image output from the image storage unit 600 according to the correlation region determined by the correlation region determination unit 700 and calculates the correlation.

Here, the search area is composed of pixels that are added by 8 pixels in the up, down, left, and right directions in the current image based on the correlation area extracted from the previous image, and the correlation function for calculating the correlation is MAE (Mean Absolute Error).

The motion estimation unit 900 estimates motion according to the degree of correlation calculated by the correlation calculation unit 800. [

The control unit 500 controls the operation of the image storage unit 600 and controls the movement of the camera according to the motion estimated by the motion estimation unit 900.

The motor unit 400 includes a motor driver 410 for driving the motor under the control of the controller 500 and a motor 420 for moving the camera 300 by the motor driver 410.

The operation of the target tracking device of the image phone according to the present invention constructed as above will be described.

First, the A / D converter 610 is turned on under the control of the controller 500 to A / D convert the NTSC signal coming from the camera 300 and output it as an 8-bit digital signal.

At this time, when the tracking mode is turned on and a telephone conversation is started, the original image coming from the camera 300 is A / D converted by the A / D converter 610 and stored in the first memory 620.

The initial image stored in the first memory 620 is shifted back to the second memory 630 and input to the quad tree partitioning unit 710. [ At this time, the initial image may be directly input to the quad tree partition unit 710 in the first memory 620.

The initial image output from the first memory 620 is input to the quad tree partitioning unit 710, and an initial window is set to an area of 8 x 8 pixels in which the target face of the caller is located.

After the initial window is set as described above, quad-tree division is performed as shown in (Equation 1) and FIG. 3 above.

4, the image output from the quad tree partitioning unit 710 is output to the correlation region extracting unit 730, as shown in FIG. 5, Regions are extracted. The detailed process of the merging unit 720 and the correlation region extracting unit 730 is as described above.

The initial image stored in the second memory 630 is a previous image and the NTSC signal from the camera 300 is A / D converted by the A / D converter 610 under the control of the controller 500, And stored in the memory 620.

As described above, the window of the previous image stored in the second memory 630 is moved. That is, in order to calculate the correlation between the search area in the current image and the correlation area in the previous image, the window of the previous image must be moved to match the correlation area.

As described above, in the window of the previous image, the correlation area is extracted in the shape of the correlation area determined by the correlation area extraction unit 730 in order to calculate the correlation in the correlation calculation unit 800, In the image, a search area is extracted for correlation calculation.

The degree of correlation is calculated by the MAE in the correlation calculation unit 800 with the extracted correlation region and the search region.

Therefore, since the part having the highest correlation degree, that is, the part having the smallest MAE, can be regarded as the target, that is, the talker, the motion estimation unit 900 calculates the motion based on the difference between the previous position and the current position.

The controller 500 controls the motor driver 410 to drive the motor 420 using the motion information. Accordingly, the camera 300 moves and a new image is input to the A / D converter 610, and the controller 500 turns on the A / D converter 610 at this time.

In the controller 500, the A / D converter 610 and the first memory 620 are turned on and the A / D converter 610 is operated until the motion estimator 900 estimates motion and outputs motion information. And the first memory 620 are turned off to prevent the NTSC signal from being input from the camera 300.

Through such an operation, the camera 300 is moved according to the position of the target so that the camera 300 can track the target.

As described above, according to the present invention, the method and apparatus for tracking a target object of a video telephone use a correlation area formed by a quad tree segmentation and merging instead of a conventional square window to estimate correlation, It is possible to reduce the amount of calculations.

Claims (11)

A correlation area setting step (100, 101, 102, 103, 104, 105) for determining a correlation area by dividing and merging brightness values of each pixel in an initial window based on a target of an initial image input in a tracking mode; A retrieval region extracting step (106, 107) for extracting a retrieval region for retrieval from an input current image; A motion estimation and motor driving step (108, 109, 110) for moving a camera by estimating a motion according to a correlation between the determined correlation area and the extracted search area, and And a call terminating step (111, 112) for repeating the search region extracting step, the motion estimation and the motor driving step for an image input until the call is terminated. The method according to claim 1, wherein the initial image is an image taken by a camera when the target is at a predetermined position. The method according to claim 1 or 2, wherein the correlated region setting step An initial image storage step (100, 101) for A / D (Analog / Digital) conversion of an initial image input in the tracking mode and storing the converted image; An initial window setting step (102) of setting an initial window at a position of a target in the stored initial image; A quad tree partitioning step (103) of dividing the set initial window into quad tree using a uniformity check function according to the brightness value of each pixel; A merging step (104) of merging a quad-tree-divided rectangle in which the target is located and a quadrangle adjacent to the target in a dispersed manner; And a correlation area determination step (105) of determining a region formed by the merged rectangle as a correlation area. 2. The method according to claim 1, wherein the search area comprises pixels added by 8 pixels in the up, down, left, and right directions of the current image based on the correlation area extracted from the previous image. 2. The method of claim 1, wherein the correlation function for calculating the correlation is a Mean Absolute Error (MAE). Image storage means 600 for A / D-converting an image input in the tracking mode; A correlation area determination unit (700) for determining a correlation area by dividing and merging brightness values of each pixel in an initial window around a target of an initial image stored in the image storage unit (600); Correlation calculation means (800) for extracting a search region from a current image output from the image storage means (600) according to a correlation region determined by the correlation region determination means (700) and calculating a correlation degree; A motion estimation unit 900 for estimating a motion according to the degree of correlation calculated by the correlation calculation unit 800; And And control means (500) for controlling the operation of the image storage means (600) and controlling the movement of the camera in accordance with the motion estimated by the motion estimation means (900) Device. 7. The apparatus of claim 6, wherein the image storage means (600) comprises an A / D converter (610) for A / D-converting an image input in a tracking mode; A first memory 620 for storing a current image signal output from the A / D converter 610; And a second memory (630) for storing video signals output from the first memory (620). 7. The method according to claim 6, wherein the correlation area determination unit (700) sets an initial window at a target position in an initial image stored in the image storage unit (600) and calculates a homogence measure A quad tree partitioning unit 710 for partitioning the quad tree using the quad tree partitioning unit 710; A merging unit 720 for merging a quadrangle in which the target is located and a quadrangle adjacent to the target in the quad tree partitioning unit 710 according to the distribution; And a correlation area extractor (730) for extracting a region formed by the merged rectangle output from the merger (730) as a correlation area. The apparatus according to claim 6, wherein the initial image is a video taken by the camera when the target is at a predetermined position. 9. The apparatus according to claim 8, wherein the search area comprises pixels added by 8 pixels in the up, down, left, and right directions of the current image based on the correlation area extracted from the previous image. The apparatus according to claim 6, wherein the correlation function for calculating the correlation is a Mean Absolute Error (MAE).