KR100220834B1 - Apparatus and method of target tracking in image telephone - Google Patents

Abstract

In order to track a target, the present invention sets a correlation area of a previous image and a search area of a current image from an input image, divides the search area into subblocks of a predetermined size, extracts uniform subblocks of the correlation area The object of the present invention is to provide a target tracking method and apparatus for reducing a calculation amount and improving tracking performance by estimating a motion of a target by calculating a correlation with a sub-block of a search area. A first step of setting a correlation area in which a target is located in a previous image and setting a search area for searching in the current image to be input; A second step of dividing the correlation area and the search area set by the first step into subblocks of a predetermined size; A third step of extracting only subblocks having a uniform brightness value in a subblock of the correlation area divided by the second process, mapping the extracted subblocks to a search area, and then calculating a correlation; A fourth step of moving the camera by estimating motion according to the degree of correlation calculated in the third step; And a fifth step of repeating the first to fourth processes with respect to the image input until the call is terminated.

Description

Title: TARGET TRACKING METHOD AND DEVICE FOR VIDEO PHONE

The present invention relates to a target tracking method and apparatus for extracting a uniform sub-block of a correlation area to track a target and then calculating a correlation with a sub-block of the search area to change the direction of the camera.

In general, tracking methods for tracking an object, ie, a target, include a center point tracking method and a correlation tracking method.

The center point tracking method is a method of tracking the center point A of the moving object after separating the moving object from the background as shown in FIG. 1A.

At this time, the threshold value is used to separate the moving object, that is, the target, from the background. That is, the background and the object are binarized using the threshold value.

However, this central tracking technique has a disadvantage that it is highly affected by noise.

That is, if the image is relatively simple and the image area is easy and the restriction on the speed of the trackable object is relatively small, the tracking stability is good and the influence of noise is small.

On the other hand, if the image is relatively complicated and the image area is not easy and the constraint on the speed of the trackable object is relatively large, the tracking stability is bad and the noise is increased.

1B, the correlation tracking method defines a moving object of a previous frame, that is, a region B having a size appropriate to the position of the target, and calculates a correlation (B) between the defined region B and the search region in the current frame And estimates that the object has moved to the region B 'having the highest degree of correlation.

That is, the correlation tracking method defines a window region of a fixed size including a moving object, that is, a correlation region when a position of a moving object is given in a given n-th image, The position of the region with the highest degree of correlation is regarded as the position of the moving object in the (n + 1) -th image.

Here, in calculating the correlation, the area of the initial window, that is, the area used in calculating the correlation between the current frame and the previous frame, is mainly a square window.

Therefore, the correlation tracking method calculates the correlation directly from the input image of the current frame without performing the image segmentation process, so that the tracking performance is maintained even for a relatively complicated image, but the calculation amount is increased.

In other words, since the correlation tracking method generally uses the intensity information of the image without binarizing the image as compared with the center-point tracking method, it is expected that some degree of tracking performance can be expected even when the image area can not be added by adding background scattering.

However, in order to estimate the motion of a moving object, the correlation tracking method has a disadvantage in that it requires a large amount of computation because the degree of correlation must be calculated for all cases between the correlation region and the search region.

According to an aspect of the present invention, there is provided a method for improving the above disadvantages, comprising the steps of: setting a correlation area of a previous image and a search area of a current image from an input image, dividing the search area into subblocks of a predetermined size, And a target tracking method and apparatus for reducing the amount of calculation and improving the tracking performance by estimating the movement of the target by calculating a correlation with the sub-block of the search area.

1A is a view for explaining a conventional center point tracking technique;

1B is a diagram for explaining a conventional correlation tracking technique;

2 is a flowchart of a target tracking method according to the present invention.

FIG. 3A is a view for explaining the search area setting step of FIG. 2

FIG. 3B is a view for explaining the correlation area setting step of FIG. 2; FIG.

3C is a diagram for explaining the sub-block extraction step in the correlation area;

FIG. 3D is a view for explaining the sub-block extraction step in the search area

FIG. 3E is a view for explaining the retrieval region extracting step of FIG. 2;

FIG. 3F shows a mapping by a correlation area in a search area

FIG.

4 is a block diagram of a target tracking apparatus according to the present invention.

DESCRIPTION OF THE REFERENCE NUMERALS

100: camera 200: image storage unit

210: analog / digital conversion unit 220: current image storage unit

230: previous image storage unit 300: block division unit

310 search area setting unit 320 sub block sub-

330: correlation area setting unit 340: sub block dividing unit

350: Uniform sub block extraction unit 400: Correlation calculation unit

500: motion estimation unit 600:

700: motor section 710: motor driving section

720: Motor

In order to achieve the above object, the present invention provides a method of tracking a target object of a video telephone set, comprising: setting a correlation area in which a target is located in a previous image to be input and setting a search area for searching in the current image, process; A second step of dividing the correlation area and the search area set by the first step into subblocks of a predetermined size; A third step of extracting only subblocks having a uniform brightness value in a subblock of the correlation area divided by the second process, mapping the extracted subblocks to a search area, and then calculating a correlation; A fourth step of moving the camera by estimating motion according to the degree of correlation calculated in the third step; And a fifth step of repeating the first through fourth processes with respect to an image input until the call is terminated.

According to another aspect of the present invention, there is provided an apparatus for tracking a target object of a video telephone, including: image storage means for A / D converting an image input in a tracking mode; Block dividing means for setting a search region and a correlation region from a current image and a previous image stored in the image storage means and dividing the search region and the correlation region into subblocks of a predetermined size to extract only a uniform subblock with respect to a subblock of the correlation region; Correlation calculating means for calculating a correlation by mapping a uniform sub-block of the correlation region divided by the block dividing means to a sub-block of the search region; A motion estimation means for estimating a motion according to the correlation calculated by the correlation calculation means; And control means for controlling the operation of the image storage means and moving the camera in accordance with the motion estimated by the motion estimation means.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

As shown in FIG. 2, the method for tracing a target of a video telephone according to the present invention includes a region setting process ST1, ST2, ST3, and ST4 as a first process, a subblock division process ST5 as a second process, (ST6), a motion estimation and a motor driving process (ST8 and ST9) as a fourth process, and a call end process (ST10 and ST11) as a fifth process.

The region setting process ST1, ST2, ST3, and ST4 of the first process is a process for extracting a correlation region in which a target is located from a previous image input in the tracking mode and extracting a search region for searching from the current image, (ST1, ST2, ST3) for A / D (Analog / Digital) conversion of a previous image and a current image input in a tracking mode, a correlation area is set from the stored previous image, And a second step (ST4) of setting an area.

Assuming that the size of the input image is 64x64, the size of the correlation area set by the image taken by the camera when the previous image, i.e., when the conversation person is at a preset position for communication, is set to 16x16 And the search area set from the current image is set to a 32 × 32 area by adding 8 pixels each in the upper, lower, right, and left sides of the correlation area.

The second step, the sub-block segmentation step ST5, is a pre-processing step for calculating the correlation degree. The 16 × 16 correlation area and the 32 × 32 search area are divided into a 4 × 4 size sub- It is a division process.

In the correlation calculation steps ST6 and ST7 as the third process, a rectangle having uniform brightness values of divided sub-blocks is searched for the correlation area divided into 4 × 4 sub-blocks by the second process, Calculating a correlation between the correlation region and the search region by mapping the average brightness value of the pixels in the sub-block of the correlation region to a predetermined threshold value, and extracting only the sub-blocks having a brightness value smaller than the threshold value (ST6), and a second step (ST7) of mapping the uniform subblock extracted in the first step (ST6) to the search area and calculating the correlation.

The motion estimation and motor driving processes ST8 and ST9 as the fourth process are processes for estimating motion and moving the camera according to the degree of correlation calculated in the third process, A first step (ST8) of estimating motion; And a second step (ST9) of moving the camera in accordance with the estimated motion.

Here, the correlation function for calculating the correlation is MAE (Mean Absolute Error).

The call termination steps ST10 and ST11 as the fifth step are a step of repeating the first to fourth steps for the input image until the end of the call.

A target tracking method of a video telephone according to the present invention, which is performed as described above, will be described in detail with reference to the accompanying drawings.

First, since there may be a case where the telephone is started and the user does not want to use the tracking function depending on the situation of the user, the user first determines whether to use the tracking function (ST1).

The distinction between trace mode and non-trace mode can be implemented simply with a switch. That is, if the switch is ON, the tracking function is performed. If the switch is OFF, the tracking function is not performed.

(ST1, ST2), which is a first process of converting an initial image input from a camera into an A / D (Analog / Digital) and storing a previous image and a current image, , ST3).

That is, the NTSC signal coming from the camera is output as an 8-bit digital signal through the A / D converter. At this time, a high-bit A / D converter may be used. Since this digital signal is a video signal, it is stored as a digital image I (x, y) having a value within a specified range on the two-dimensional matrix on the memory.

This digital image is an image with a size of 64 x 64 pixels and is stored in memory and used for tracking the target.

In the correlation tracking, it is assumed that the initial position, that is, the position where the target caller can enjoy the dial to make a call, is known by the locking process of the initial target.

Therefore, the initial image and the next image, that is, the previous image and the current image, are stored in the memory.

A correlation area having a size of 16x16 that can include the target person's face is set from the previous image taken in this way, and a search area of 32x32 size is set from the current image (ST4).

That is, as shown in FIG. 3A, a search area of 32 × 32 size for calculating a correlation with a previous image from a current image of 64 × 64 pixels is set. From the previous image, as shown in FIG. A correlation area including a face of a caller of 16 x 16 size is set.

When the correlation area and the search area are set as described above, each area is divided into 4 × 4 subblocks as shown in FIGS. 3C and 3D. This is because when the correlation is calculated between the correlation area and the search area, It is a preprocessing process (ST5) to reduce the amount of calculation by calculating the degree of correlation in units of subblocks instead of the correlation calculation in units of pixels.

As shown in FIG. 3E, when a sub-block is divided into sub-blocks, a process of extracting a sub-block having a uniform brightness value in a correlation area of 16 × 16 size (ST6) is performed. same.

The reason for extracting the uniform subblock is that the shape of the uniform subblock forms a certain shape of the target, so that the effect similar to the segmentation of the image can be obtained.

Therefore, the uniformity function is obtained for each sub-block of the correlation area. That is, in order to find a block having a uniform brightness value in each sub-block, a uniformity check function is obtained by the following equation [1] do.

[Formula 1]

Here, mean is the average value of the pixels in the sub-block.

If the uniformity check function of each sub-block thus obtained is larger than a predetermined threshold value, it is not a uniform sub-block. On the other hand, if the uniformity checking function is small, it is determined that the brightness value is a uniform sub- Block is extracted.

As shown in FIG. 3F, the uniform sub-block of the extracted correlation region is mapped to the search region to calculate the correlation (ST7). The correlation region is shifted by four pixels in the search region, The correlation is calculated only for the uniform subblock (hatched block).

Therefore, instead of calculating the degree of correlation with the search area corresponding to the entire sub-block of the correlation area, the amount of calculation can be reduced by calculating the correlation with the search area only for the uniform sub-block.

At this time, a correlation function to be used for calculating the correlation between the correlation region of the previous image and the search region of the current image should be determined.

As shown in the following formulas [2], [3] and [4], the correlation function for calculating the correlation is a normalized cross correlation function (NCCF), a mean square error (MSE) have.

[Formula 2]

[Formula 3]

[Formula 4]

Where E (·) represents the mean.

As the correlation function, NCCF is most suitable but MAE is used considering the calculation amount.

The higher the correlation between images, the smaller the value of MAE is. Therefore, when MAE is used as a correlation function, the position with the lowest value becomes the moving object in the next image, that is, the estimated position of the target.

Since the correlation between the search area of the current image and the previous area of the previous image is calculated according to the correlation function and the part having the highest correlation is considered to be the target of the search area, . ≪ / RTI >

That is, the movement of the face of the talker, which is the target, is estimated by calculating the difference between the position of the correlation area in the previous image and the position having the highest correlation in the search area of the current image (ST8).

Therefore, the motor is driven in accordance with the estimated motion, that is, the talker moves to move the camera (ST9).

At this time, if a new image is inputted from the camera, a new image is inputted, and if the new image is input, the area is stored (ST3).

The video signal of one frame is stored in one memory and then shifted to the next memory and used as a previous video. That is, the video signals of the current video and the previous video are stored in the two memories, respectively.

Therefore, the image stored in the area setting process ST1, ST2, ST3, and ST4 is the current image being currently input, and the previous image including the initial image is shifted and stored in another memory for the next motion estimation.

When the video signals of the previous video and the current video are stored in the memory, the correlation area and the search area are extracted (ST4).

That is, a portion having the highest degree of correlation extracted in the motion estimation step ST8, which is the highest degree of correlation extracted from the previous image, is extracted as a shape of the determined correlation region and set as a correlation region, The search area having 32 x 32 pixels centered on the correlation area of the image is set in the current image.

The correlation calculation step ST7, the motion estimation step ST8, and the motor drive step ST9 for calculating the correlation thereafter are performed as described above.

On the other hand, if a new image is not input from the camera, a call termination step of terminating the tracking function is performed by searching whether the call is terminated.

That is, when a new image is inputted, the process goes to the image storing step ST3 and the above process is repeated. When the new image is not inputted and the call is ended, the tracking function is finished (ST11).

If the call is not terminated but a new image is not input, the system waits until a new image is input.

4, a target tracking device for a video telephone according to the present invention includes a control unit 600, an image storage unit 200, a block dividing unit 300, a correlation calculation unit 400, (500), and a motor unit (700).

The image storage unit 200 A / D converts the image input from the camera 100 in the tracking mode under the control of the control unit 600 and stores the converted image. The A / D conversion unit A / D converter 210, a current image storage 220 for storing the current image signal output from the A / D converter 210, and a controller 220 for storing the image signal output from the current image storage 220 And a previous image storage unit 230.

The block division unit 300 extracts a correlation area in which a target is located from a previous image stored in the image storage unit 200 and sets a search area for searching from the current image. A sub block dividing unit 320 for dividing the search area set by the search area setting unit 310 into sub blocks having a predetermined size, A sub block dividing unit 340 for dividing the correlation area set by the correlation area setting unit 330 into sub blocks of a predetermined size, And a uniform sub-block extractor 350 for extracting only one sub-block having a brightness value among the sub-blocks of the correlation area divided by the sub-block dividing unit 340.

The correlation area is set to a size of 16 x 16 that can include the target person's face from the previous image and the search area is set to a size of 32 x 32 from the current image.

The correlation area and the search area are divided into 4 × 4 subblocks, and only the subblocks having uniform brightness values of the respective subblocks of the correlation area are extracted.

The correlation calculator 400 maps the uniform subblocks of the correlation area to the subblocks of the search area divided by the block dividing unit 300 and calculates the correlation by mapping the subblocks in units of subblocks.

Here, the correlation function for calculating the correlation is made of MAE (Mean Absolute Error).

The motion estimation unit 500 estimates the motion according to the degree of correlation calculated by the correlation calculation unit 400.

The control unit 600 controls the operation of the image storage unit 200 and controls the camera 100 to move according to the motion estimated by the motion estimation unit 500.

The motor unit 700 includes a motor driving unit 710 for driving the motor under the control of the controller 600 and a motor 720 for moving the camera 100 by the motor driving unit 710.

The operation of the target tracking device of the image phone according to the present invention constructed as above will be described.

First, the A / D converter 210 is turned on under the control of the control unit 600 to A / D convert the NTSC signal coming from the camera 100 and output it as an 8-bit digital signal.

At this time, when the tracking mode is on and a telephone conversation is started, the original image coming from the camera 100 is A / D converted by the A / D converter 210 and stored in the current image storage 220.

The initial image stored in the current image storage unit 220 is input to the search area setting unit 310 of the block dividing unit 300 to set a search area of 32x32 size for correlation search.

In addition, the initial image is shifted and stored in the previous image storage unit 230 so that the correlation area setting unit 330 of the block dividing unit 300 determines a correlation area of 16 x 16 pixels Is set.

When the search area and the correlation area are set as described above, the sub-block sub-blocks 320 and 340 receive the sub-block sub-blocks 320 and 340 and divide them into 4 × 4 sub-blocks, respectively.

This is a preprocessing process for reducing the amount of computation by calculating the correlation degree in units of subblocks rather than the correlation calculation in units of pixels having a large amount of calculation at the time of calculating the correlation between the correlation region and the search region.

In the uniform sub-block extraction unit 350, a sub-block of a correlation area is applied to extract only a 4 × 4 size sub-block having a uniform brightness value. This is because a shape of a uniform sub- So that an effect similar to segmentation of an image can be obtained.

When the uniform sub-blocks of the correlation region are extracted as described above, the correlation calculation unit 400 maps the uniform sub-blocks of the correlation region output from the block division unit 300 to the search region as shown in FIG. 3F, The correlation is calculated only for the uniform subblocks by moving the correlation area by four pixels in the search area, that is, by one subblock.

At this time, the minimum motion unit in the present invention is equal to the size of the subblock, and the size of the subblock is negligible.

Therefore, instead of calculating the degree of correlation with the search area corresponding to the entire sub-block of the correlation area, the amount of calculation can be reduced by calculating the correlation with the search area only for the uniform sub-block.

This correlation is calculated by MAE (Mean Absolute Error).

Accordingly, since the part having the highest correlation degree, that is, the part having the smallest MAE can be regarded as the target, that is, the talker, the motion estimation unit 500 calculates the motion based on the difference between the previous position and the current position.

As a result, since the correlation area and the search area are divided into 4 × 4 subblocks, the motion of less than 4 pixels is ignored, and the motion estimation is performed only for the motion of 4 pixels or more.

Using the motion information, the controller 600 controls the motor driver 710 to drive the motor 720.

Accordingly, the camera 100 moves and a new image is input to the A / D converter 210, and the controller 600 turns on the A / D converter 210 at this time.

The controller 600 controls the A / D converter 210 and the current image storage unit 220 to turn on the A / D converter 210, and then, until the motion estimator 500 estimates motion and outputs motion information, (210) and the current image storage unit (220) so that the NTSC signal can not be input from the camera (100).

Through such an operation, the camera 100 is moved according to the position of the target so that the camera 100 can track the target.

As described above, according to the present invention, the target tracking method and apparatus of the image phone divides a correlation area and a search area into subblocks of a predetermined size, extracts only subblocks having brightness values of the subblocks of the correlation area The correlation is estimated by mapping it to the search area, thereby reducing the amount of calculation and improving the tracking performance.

Claims (15)

A first step of setting a correlation area where a target is located in a previous image to be input and setting a search area for searching in a current image to be input when the tracking mode is selected; A second step of dividing the correlation area and the search area set by the first step into subblocks of a predetermined size; A third step of extracting only subblocks having a uniform brightness value in a subblock of the correlation area divided by the second process, mapping the extracted subblocks to a search area, and then calculating a correlation; A fourth step of moving the camera by estimating motion according to the degree of correlation calculated in the third step; And And a fifth step of repeating the first to fourth processes for the input image until the end of the call. The method of claim 1, wherein the first step comprises A / D converting the previous image and the current image input in the tracking mode and storing the converted image; And a second step of setting a correlation area from the stored previous image and setting a search area from the current image. 3. The method according to claim 1 or 2, wherein the correlation area comprises 16 x 16 pixels. 3. The method according to claim 1 or 2, wherein the search area comprises 32 x 32 pixels. 2. The method according to claim 1, wherein the sub-block comprises 4x4 pixels. The method according to claim 1, wherein the third step comprises: a first step of comparing only the average brightness value of each sub-block of the correlation area with a predetermined threshold value and extracting only a uniform sub-block having a brightness value smaller than the threshold value; And a second step of mapping the uniform subblock extracted by the first step to a search area and calculating a degree of correlation. 7. The method according to claim 1 or 6, wherein the correlation function for calculating the correlation is a Mean Absolute Error (MAE). The method according to claim 1, wherein the fourth step comprises: a first step of extracting a part having the highest correlation degree and estimating motion; And moving the camera in accordance with the estimated movement. ≪ Desc / Clms Page number 19 > An image storage means for A / D converting the input image in the tracking mode and storing the converted image; Block dividing means for setting a search region and a correlation region from a current image and a previous image stored in the image storage means and dividing the search region and the correlation region into subblocks of a predetermined size to extract only a uniform subblock with respect to a subblock of the correlation region; Correlation calculating means for calculating a correlation by mapping a uniform sub-block of the correlation region divided by the block dividing means to a sub-block of the search region; A motion estimation means for estimating a motion according to the correlation calculated by the correlation calculation means; And And control means for controlling the operation of the image storage means and moving the camera in accordance with the motion estimated by the motion estimation means. The apparatus of claim 9, wherein the image storage means comprises: an A / D converter for A / D-converting an image input in a tracking mode; A current image storage unit for storing a current image signal output from the A / D converter; And a previous image storage unit for storing image signals output from the current image storage unit. 10. The apparatus of claim 9, wherein the block dividing unit comprises: a search area setting unit for setting a search area of a predetermined size for searching for a correlation degree from a current image; A sub-block dividing unit for dividing the search area set by the search area setting unit into sub-blocks of a predetermined size; A correlation area setting unit for setting a correlation area of a predetermined size at which a target is located from a previous image; A sub-block dividing unit for dividing the correlation area set by the correlation area setting unit into sub-blocks of a predetermined size; And a uniform sub-block extracting unit for extracting only a sub-block having a uniform brightness value from the sub-blocks of the correlation area divided by the sub-block dividing unit. 12. The apparatus according to claim 9 or 11, wherein the correlation area comprises 16 x 16 pixels. 12. The method according to claim 9 or 11, wherein the search area comprises 32 x 32 pixels. 12. The apparatus according to claim 9 or 11, wherein the sub-block comprises 4x4 pixels. 10. The apparatus according to claim 9, wherein the correlation function for calculating the correlation is a Mean Absolute Error (MAE).