KR100226583B1 - Target tracking method and device for video phone - Google Patents

Abstract

According to the present invention, in a video telephone, a correlation region of a previous image and a search region of a current image are set from an input image, and then divided into subblocks having a predetermined size with respect to the correlation region to merge subblocks having similar brightness values. Its purpose is to reduce the amount of calculation and improve tracking performance by estimating the motion of the target by calculating the correlation between the merged subblock and the search area. An area setting process of setting a correlation area and setting a search area for searching in the input current image; A block dividing / merging process of dividing the subblocks having a predetermined size into the correlated region set by the region setting process and then merging subblocks having similar brightness values into one region; A correlation calculation step of calculating a correlation by mapping subblocks of the correlation area merged by the block division / merging process to a search area; A motion estimation and motor driving process of estimating motion and moving the camera according to the correlation calculated by the correlation calculation process; And a call termination process of repeating a region setting process, a motion estimation process, and a motor driving process with respect to the input image until the call ends.

Description

Target tracking device and method of video telephone

The present invention relates to a method and apparatus for tracking an object in a video telephone, in which a similar subblock of a correlation area is merged to track a target, and then a direction of the camera is changed by calculating a correlation with the subblock of the search area.

In general, a tracking method for tracking an object, that is, a target, includes a center tracking method and a correlation tracking method.

The center point tracking technique is a method of tracking the center point A of the extracted moving object after separating the moving object from the background as shown in FIG. 1A.

At this time, the threshold value is used to separate the moving object, that is, the target from the background. In other words, the background and the object are binarized using the threshold.

However, this central tracking technique has a drawback of being affected by noise.

In other words, when the image is relatively simple and the image is easily segmented and the constraint on the speed of the traceable object is relatively low, the tracking stability is good and the noise is less affected.

On the contrary, when the image is relatively complicated and image segmentation is not easy and the constraint on the speed of the traceable object is relatively high, the tracking stability is poor and the noise is high.

In addition, the correlation tracking technique defines a region B of a size appropriate for the moving object of the previous frame, that is, the target position, as shown in FIG. 1B, and the correlation between the defined region B and the search region in the current frame. It is a method of estimating that an object has moved to the region B 'having the highest correlation by calculating.

That is, the correlation tracking technique defines a window area having a constant size, that is, a correlation area, including a moving object when a position of a moving object is given in a given n-th image, and for each position on the search area in the n + 1 th image. The correlation is calculated to regard the position of the region having the highest correlation as the position of the moving object in the n + 1 th image.

Here, the shape of the initial window when the correlation is calculated, that is, the area used when calculating the correlation between the current frame and the previous frame is mainly composed of a square window.

Therefore, since the correlation tracking technique calculates the correlation directly from the image of the current frame input without performing the image segmentation process, the tracking performance is maintained even for a relatively complex image, but a large amount of calculation is required.

That is, the correlation tracking technique generally uses contrast information of the image rather than the center point tracking technique, so that some tracking performance can be expected even when image scattering is impossible due to background scattering.

However, the correlation tracking technique has a disadvantage in that a large amount of calculation is required because the correlation must be calculated in all cases between the correlation region and the search region in order to estimate the movement of the moving object.

In order to solve the above disadvantages, the present invention provides a video telephone having a similar brightness value by setting a correlation region of a previous image and a search region of a current image from an input image, and then dividing the correlation region into subblocks having a predetermined size. It is an object of the present invention to provide a target tracking method and apparatus for reducing a computation amount and improving tracking performance by merging subblocks and calculating correlations between the merged subblocks and a search region.

1A is a diagram for explaining a conventional center point tracking technique.

1B is a diagram for explaining a conventional correlation tracking technique.

2 is a flowchart of a target tracking method according to the present invention.

3A is a diagram for explaining a search area setting step of FIG. 2;

FIG. 3B is a diagram for describing a correlation area setting step of FIG. 2.

3C is a diagram for explaining a subblock extraction step in a correlation region;

3D is a diagram for describing a like subblock merging state in a correlation region.

3E is a mapping by correlation region in search region to calculate correlation

Drawing showing status

4 is a block diagram of a target tracking device according to the present invention.

<Explanation of symbols for the main parts of the drawings>

100: camera 200: video storage

210: analog / digital converter 220: current image storage unit

230: previous image storage unit 300: block division / merge unit

310: search area setting unit 320: correlation area setting unit

330: subblock division unit 350: pseudo subblock merge unit

400: correlation calculation unit 500: motion estimation unit

600: control unit 700: motor unit

710: motor driving unit 720: motor

In order to achieve the above object, a target tracking method of a video telephone according to the present invention, in the tracking mode, sets a correlation region where a target is located in a previous image inputted and sets a search region for searching in the current image inputted. process; A block dividing / merging process of dividing the subblocks having a predetermined size into the correlated region set by the region setting process and then merging subblocks having similar brightness values into one region; A correlation calculation step of calculating a correlation by mapping subblocks of the correlation area merged by the block division / merging process to a search area; A motion estimation and motor driving process of estimating motion and moving the camera according to the correlation calculated by the correlation calculation process; And a call termination process of repeating a region setting process, a motion estimation process, and a motor driving process with respect to the input image until the call ends.

In addition, the target tracking device of the video telephone according to the present invention for achieving the above object, the image storage means for storing the A / D conversion of the image input in the tracking mode; Block dividing / merging means for setting a search region and a correlation region from the current image and the previous image stored in the image storage means, and then dividing only the correlation region into sub-blocks and merging sub-blocks having similar brightness values for the sub-blocks; Correlation coefficient calculating means for calculating a correlation by mapping subblocks of the correlation region merged by the block division / merge means to a search region; Motion estimation means for estimating motion according to the correlation degree calculated in said correlation calculation means; And control means for controlling the operation of the image storing means and controlling the camera to move according to the motion estimated by the motion estimating means.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

As shown in FIG. 2, the target tracking method of the video telephone according to the present invention includes a region setting process (ST1, ST2, ST3, ST4), a subblock division / merge process (ST5, ST6), and a correlation calculation process (ST7). , Motion estimation and motor driving processes ST8 and ST9, and call termination processes ST10 and ST11.

The region setting process (ST1, ST2, ST3, ST4) is a process for extracting a correlation region where a target is located from a previous image input in the tracking mode and extracting a search region for searching from the current image. A first step (ST1, ST2, ST3) of converting and storing the previous image and the current image to A / D (Analog / Digital), to set the correlation region from the stored previous image and to set the search region from the current image It is performed by the second step ST4.

Here, assuming that the size of the input image is 64 × 64 size, the size of the correlation area set by the previous image, that is, the image taken by the camera when the talker is at a preset position for the call is set to 16 × 16. The search area set from the current image is set to a 32 × 32 size area by adding 8 pixels each up, down, left, and right from the correlation area.

The block division / merge process (ST5, ST6) is a preprocessing process for calculating the correlation. The predetermined 16 × 16 size correlation area is divided into 4 × 4 sized subblocks of a constant size, and then subblocks having similar brightness values. Is the process of merging into one area.

The correlation calculation process ST7 is a process of calculating the correlation between the correlation region and the search region by mapping the region merged by the block division / merging process to the search region.

The motion estimation and motor driving process (ST8, ST9) is a process of moving the camera by estimating the motion according to the correlation calculated by the block division / merging process. Estimating (ST8); A second step ST9 is performed to move the camera according to the estimated movement.

Here, the correlation function for calculating the correlation is a mean absolute error (MAE).

The call termination process (ST10, ST11) is a process of repeatedly performing the area setting process (ST1, ST2, ST3, ST4) to the motion estimation and motor driving process (ST8, ST9) for the input image until the call is terminated to be.

The target tracking method of the video telephone according to the present invention performed as described above will be described in detail with reference to the accompanying drawings.

First, since the call is initiated and there may be times when the user does not want to use the tracking function, the user first decides whether to use the tracking function (ST1).

The distinction between tracking mode and non-tracking mode is simple with a switch. That is, if the switch is ON, the tracking function is performed. If the switch is OFF, the tracking function is not performed.

When the tracking mode is turned on to perform the tracking function, the first image storage step (ST1, ST2) converts an initial image input from the camera to A / D (Analog / Digital) to store the previous image and the current image. , ST3).

In other words, NTSC signal from camera is output as 8-bit digital signal through A / D converter. A high bit A / D converter can also be used.

Since this digital signal is a video signal, it is stored in the memory as a digital video (I (x, y)) having a value within a range specified on a two-dimensional matrix.

This digital image is an image having a size of 64 x 64 pixels, stored in a memory and used for tracking a target.

The correlation tracking assumes that the initial target is known by the locking process of the initial target, that is, the position where the target caller can dial to make a call.

Therefore, the initial image and the next image, that is, the previous image and the current image are stored in the memory.

From the previous image taken as described above, a correlation area of 16 × 16 size, which may include the caller's face as a target, is set, and a search area of 32 × 32 size is set from the current image (ST4).

That is, as shown in FIG. 3A, a search area of size 32 × 32 for calculating a correlation with the previous image is set from the current image having a size of 64 × 64 pixels, and as shown in FIG. 3B. A correlation area including a face of a 16 × 16 caller is set.

When the correlation region and the search region are set in this way, as shown in FIG. 3C, each correlation region is divided into 4 × 4 subblocks, which is a pixel unit having a large amount of computation when calculating the correlation between the correlation region and the search region. This is a preprocessing step for reducing the amount of calculation by calculating the correlation of subblock units rather than the correlation of (ST5).

When divided into subblocks, a process of merging subblocks having similar brightness values into one region is performed in a 16 × 16 correlation region as shown in FIG. 3D, which will be described in more detail. Is as follows.

The reason for extracting the similar subblocks is that the similar subblocks form the shape of the target to some extent, and thus have an effect similar to that of segmenting the image.

Therefore, it is determined whether or not each of the subblocks in the correlation region has a similar brightness value, and the determination thereof is examined by the following equation [1].

Same area = | M1-M2 | <T

Other areas = | M1-M2 | ≥ T

Where M is the average brightness of the subblock,

T is a constant threshold.

In other words, if the difference between the brightness values of any two blocks, that is, M1 and M2 is less than or equal to the predetermined threshold, it is determined to be a sub-block having similar brightness and included in the same area by merging them. It will not be included in the same area.

Through this process, a predetermined number of regions (four merge regions in FIG. 3D) of the correlation region are extracted.

As shown in FIG. 3E, the merged region in the extracted correlation region is mapped to the search region to calculate the correlation (ST7). The correlation is calculated while the merged region of the correlation region corresponds to the search region. do.

Therefore, rather than calculating the correlation with the search region corresponding to the entire subblock of the correlation region, the amount of computation can be reduced by calculating the correlation with the search region only for the merged region.

In this case, in order to calculate a correlation between the merged region of the correlation region of the previous image and the search region of the current image, a correlation function to be used must be determined.

Correlation functions for such correlation calculation include the Normalized Cross Correlation Function (NCCF), Mean Square Error (MSE), and Mean Absolute Error (MAE), as shown in the following equations [2], [3], and [4]: have.

Here, E (*) represents an average.

NCCF is the best correlation function, but MAE is used considering the amount of calculation.

The higher the correlation between images, the smaller the MAE is.

Therefore, when using MAE as the correlation function, the lowest position is the estimated position of the moving object, that is, the target in the next image.

According to the correlation function, the correlation between the search area of the current image and the correlation area of the previous image is calculated, and since the highest correlation is regarded as the dialog of the search area, that is, the target, the movement is the previous position and the current position. Is calculated by the difference.

That is, the difference between the positions of the merged regions of the correlation region in the previous image and the positions of the most highly correlated portions in the search region of the current image is calculated, and the motion vectors of the merged regions, i.e. ①, ②, ③ Finally, the final motion is estimated by extracting the motion vector ④ and then calculating the average of the motion vector (ST8).

Therefore, the camera is moved by driving the motor in accordance with the estimated movement, that is, the movement of the talker (ST9).

At this time, if the new image is input from the camera and the new image is input, the area storage step ST3 of A / D conversion is again performed.

An image signal of one frame is stored in one memory, shifted to the next memory, and used as a previous image.

That is, the video signals of the current video and the previous video are stored in the two memories, respectively.

Therefore, the image stored in the area setting process (ST1, ST2, ST3, ST4) is the current image currently input, the previous image including the initial image is shifted and stored in another memory for the next motion estimation.

As described above, when image signals of the previous image and the current image are stored in the memory, the correlation region and the search region are extracted (ST4).

That is, the highest correlation degree extracted in the motion estimation step ST8, which is the highest correlation degree extracted from the previous image, is extracted as a shape of the determined correlation area, and is set as a correlation area. A search region consisting of 32 x 32 pixels is set in the current image based on the correlation region of the image.

Subsequently, the subblock division and merging steps ST5 and ST6, the correlation calculation step ST7, the motion estimation step ST8, and the motor driving step ST9 for calculating the correlation are performed as described above.

On the other hand, if a new video is not input from the camera performs a call termination step of searching whether the call is terminated and ends the tracking function.

That is, when a new video is input, the process proceeds to the video storing step ST3 and repeats the above process.

In addition, if a call is not ended but a new video is not input, it waits until a new video is input.

Next, as shown in FIG. 4, the target tracking device of the video telephone according to the present invention includes an image storage unit 200, a block division / merge unit 300, a correlation calculation unit 400, and a motion estimation unit 500. , The control unit 600 and the motor unit 700.

The image storage unit 200 A / D converts and stores the image input from the camera 100 in the tracking mode under the control of the controller 600, and A / D converts the image input in the tracking mode. The D converter 210, the current image storage unit 220 for storing the current video signal output from the A / D converter 210, and stores the video signal output from the current image storage unit 220 The previous image storage unit 230 is configured.

The block dividing / merge unit 300 extracts a correlation region where a target is located from a previous image stored in the image storage unit 200 and sets a search region for searching from the current image. A search region setting unit 310 for setting a search region having a predetermined size for searching a target, a correlation region setting unit 320 for setting a correlation region having a predetermined size in which a target is located from a previous image, and the correlation region setting unit 320. The sub-block dividing unit 330 for dividing the correlation region set by the sub block into a sub-block having a predetermined size and the sub-blocks having similar brightness values from the sub-blocks of the correlation region divided by the sub-block dividing unit 330 are examined. Similar sub-block merging unit 340 for merging.

The correlation area is set to a size of 16 × 16 that can include the caller's face as a target from the previous picture, and the search area is set to a size of 32 × 32 from the current picture.

The correlated region thus set is divided into subblocks having a size of 4x4, and only subblocks having similar brightness values among the subblocks of the correlated region are extracted and merged into one region.

The correlation calculator 400 calculates a correlation by mapping the regions merged by the block dividing unit 300 to a search region and corresponding to each other.

Here, the correlation function for calculating the correlation is made of Mean Absolute Error (MAE).

The motion estimator 500 estimates the motion according to the correlation degree calculated by the correlation calculator 400.

The controller 600 controls the operation of the image storage unit 200 and controls the camera 100 to move according to the motion estimated by the motion estimator 500.

The motor unit 700 includes a motor driver 710 for driving a motor under the control of the controller 600, and a motor 720 for moving the camera 100 by the motor driver 710.

The operation of the target tracking device of the video telephone according to the present invention configured as described above will be described.

First, the A / D converter 210 is turned on under the control of the controller 600 to A / D convert NTSC signals from the camera 100 and output them as 8-bit digital signals.

In this case, when the tracking mode is turned on to start a phone call, the A / D conversion unit 210 converts the first image from the camera 100 to the current image storage unit 220.

The initial image stored in the current image storage unit 220 is input to the search region setting unit 310 of the block divider 300 to set a 32 × 32 size search region for correlation search.

In addition, the initial image is shifted and stored in the previous image storage unit 230 so that the correlation region of 16 x 16 pixels is located at the part where the caller's face as a target is located by the correlation region setting unit 330 of the block division unit 300. Will be set.

When the search area and the correlation area are set in this way, the sub-block dividing unit 330 receives the correlation area and divides the sub block into 4 × 4 sub-blocks.

This is a preprocessing process for reducing the amount of calculation by calculating the correlation between sub-blocks rather than the calculation of the pixel-by-pixel correlations when calculating the correlation between the correlation area and the search area.

The similar subblock merging unit 340 receives the subblocks of the correlation region and extracts only the 4 × 4 subblocks having similar brightness values, and merges them into one area. This is because the shape forms the shape of the target to some extent, and thus an effect similar to that of segmenting the image can be obtained.

When the sub-blocks having similar brightness values of the correlation region are merged as described above, the correlation calculator 400 maps the merge region of the correlation region output from the block divider 300 to the search region as illustrated in FIG. 3E. The degree of correlation is calculated. The degree of correlation is calculated only for the merged area while the correlation area is moved to the search area.

In this case, the minimum motion unit in the present invention is as much as the size of the merge region, and the following is ignored.

Therefore, the amount of calculation can be reduced by calculating the correlation with the search area only for the area merged into one area of the correlation area.

This correlation is calculated by means of Mean Absolute Error (MAE).

Therefore, the portion having the highest correlation among the search areas, that is, the smallest MAE, can be regarded as the target, that is, the dialogue person, so the motion estimator 500 calculates the motion based on the difference between the previous position and the current position.

Using the motion information, the controller 600 controls the motor driver 710 to drive the motor 720.

Accordingly, the camera 100 moves and a new image is input to the A / D converter 210. At this time, the controller 600 turns on the A / D converter 210.

Meanwhile, the controller 600 turns on the A / D converter 210 and the current image storage unit 220, and then estimates the motion by the motion estimator 500 to output the motion information. The 210 and the current image storage unit 220 are turned off to prevent the NTSC signal from being input from the camera 100.

Through this operation, the camera 100 is moved according to the position of the target so that the camera 100 can track the target.

As described above, the method and apparatus for tracking a target of an image telephone according to the present invention divides the correlation area into sub-blocks having a predetermined size, and then merges the sub-blocks having similar brightness values into one area to correlate them to the search area. Since the degree is estimated, the amount of calculation is reduced, and the tracking performance is improved.

Claims (16)

A region setting process of setting a correlation region where a target is located in the previous image to be input in the tracking mode and setting a search region for searching in the input current image; A block dividing / merging process of dividing the subblocks having a predetermined size into the correlated region set by the region setting process and then merging subblocks having similar brightness values into one region; A correlation calculation step of calculating a correlation by mapping subblocks of the correlation area merged by the block division / merging process to a search area; A motion estimation and motor driving process of estimating motion and moving the camera according to the correlation calculated by the correlation calculation process; And And a call termination process for repeating the process of area setting, motion estimation, and motor driving for the input image until the call is terminated. The method of claim 1, wherein the area setting process comprises: a first step of A / D converting and storing a previous image and a current image which are input in a tracking mode; And a second step of setting a correlation region from the stored previous image and setting a search region from the current image. 3. A method according to claim 1 or 2, wherein the correlation area consists of 16 x 16 pixels. 3. The method of claim 1 or 2, wherein the search area is composed of 32 x 32 pixels. 2. The method of claim 1, wherein the subblock of the correlation region is 4x4 pixels. 2. The method of claim 1, wherein the subblock merging of the block division / merge process merges when the difference between the brightness values of any two subblocks is equal to or less than a predetermined threshold, and otherwise does not merge. The method of claim 1, wherein the correlation function for calculating the correlation is a mean absolute error (MAE). The method of claim 1, wherein the motion estimation and motor driving process comprises: a first step of estimating motion by extracting a portion having the highest calculated correlation; And a second step of moving the camera according to the estimated movement. Image storage means for A / D converting and storing the image input in the tracking mode; Block dividing / merging means for setting a search region and a correlation region from the current image and the previous image stored in the image storage means, and then dividing only the correlation region into sub-blocks and merging sub-blocks having similar brightness values for the sub-blocks; Correlation coefficient calculating means for calculating a correlation by mapping subblocks of the correlation region merged by the block division / merge means to a search region; Motion estimation means for estimating motion according to the correlation degree calculated in said correlation calculation means; And And control means for controlling the operation of the image storage means and controlling the camera to move according to the motion estimated by the motion estimation means. 10. The apparatus of claim 9, wherein the image storing means comprises: an A / D converter configured to A / D convert an image input in the tracking mode; A current video storage unit for storing a current video signal output from the A / D converter; And a previous video storage unit for storing a video signal outputted from the current video storage unit. 10. The apparatus of claim 9, wherein the block dividing / merge means comprises: a search region setting unit for setting a search region having a predetermined size for searching for a correlation from a current image; A correlation region setting unit for setting a correlation region of a predetermined size in which a target is located from a previous image; A sub-block dividing unit dividing the correlation region set by the correlation region setting unit into sub-blocks having a predetermined size; And a similar sub-block merging unit for merging only sub-blocks having similar brightness values among the sub-blocks of the correlated region divided by the sub-block dividing unit into one region. 12. The target tracking device of claim 11, wherein the similar sub-block merging unit merges if the difference between the brightness values of any two sub-blocks is equal to or less than a predetermined threshold, and otherwise does not merge. 12. The target tracking device of a video telephone according to claim 9 or 11, wherein the correlation area is made of 16 x 16 pixels. 12. The method of claim 9 or 11, wherein the search area is composed of 32 x 32 pixels. 12. The target tracking device of a video telephone according to claim 9 or 11, wherein the subblock is composed of 4x4 pixels. 10. The apparatus of claim 9, wherein the correlation function for calculating the correlation is a mean absolute error (MAE).