KR100220836B1 - Speaker tracking method and device for video conference system - Google Patents

Abstract

The present invention relates to a method and apparatus for tracking a sister after recognizing a singer using an infrared image in a video conference system.

In the present invention, the faces of each model are databased by using the boundary pixels extracted from the model and the sister image signal, and the face image of the sister is extracted and matched in the same manner as the face extraction method of the model, ≪ / RTI >

Accordingly, the present invention has a function of extracting a closed curve by using boundary pixels and directions and recognizing the scepter and then tracking the scepter, so that even if the scepter moves during the videoconference, it can be traced by the camera.

Description

A method and apparatus for tracking a delegate in a video conference system (Speaker Tracking Method and Device for Video Conference System)

The present invention relates to a method and apparatus for tracking a sister after recognizing a singer using an infrared image in a video conference system.

In general, a video conferencing system allows a conference attendee to stay on the screen, even if they are at a distance.

Therefore, the conference person was required to be in front of the camera so that he could show his / her appearance to the remote party through the screen.

That is, the conventional video conferencing system has a problem that the conference person can not move because the conference person must be located in front of the camera in the video conference.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and apparatus for tracking a skeeter in a video conferencing system for recognizing a singer using an infrared image and tracking the singer to freely move the singer.

1 is a flowchart showing a model processing step according to the present invention.

FIG. 2 is a flow chart illustrating the skeiner recognition, matching, and tracking steps according to the present invention.

FIGS. 3A, 3B and 3C illustrate the boundary pixels and direction extraction steps of FIGS. 1 and 2

Drawings for

Figs. 4 and 5 are views for explaining the closed curve extracting steps of Figs. 1 and 2

Fig. 6 is a view for explaining the matching step of Fig. 2

FIG. 7 is a detailed flowchart showing the tracking step of FIG.

FIG. 8 is a block diagram of a skeiner tracking device according to the present invention.

DESCRIPTION OF THE REFERENCE NUMERALS

300: infrared camera 400: motor unit

410: motor 420: motor driver

500: control unit 600: image storage unit

610: A / D conversion section 510, 620: Memory

700: recognition unit 710: boundary pixel extraction unit

720: boundary direction extracting unit 730: closed curve extracting unit

800: initial positioning unit 900: tracker

In order to accomplish the above object, a method for tracking a skeiner of a video conference system according to the present invention is a method for tracking a singer in a video conference using a boundary pixel extracted by inputting a model video signal corresponding to one or more sceiver images, A model processing step of extracting a corresponding closed curve to recognize a model face; A model selection step of selecting one of the recognized one or more model image signals; A conference person image processing step of recognizing a conference person's face by extracting a closed curve corresponding to the conference person's face using the extracted boundary pixels input through the camera; A matching step of matching a closed curve of the selected model face with a closed curve of the recognized sister face; Tracking the skeptics to move the camera; And a model changing step of proceeding to a matching step when the other one of the recognized one or more model video signals is selected.

In order to achieve the above object, a sidenor tracking device of a video conferencing system according to the present invention is a device for tracking a video signal of each model and a sketcher corresponding to one or more video images of a skeiner during a video conference, Image storage means; A recognizing means for recognizing a model face by extracting a closed curve corresponding to each model and a face of a skeeter using the model outputted from the image storing means and the boundary pixels extracted by inputting the sketcher image signal; Control means for controlling the operation of the image storage means and the recognizing means, for matching the outline of the sister face of the model recognized by the recognizing means, and for tracking the matching sister; Initial position determining means for setting a window of a predetermined size at an initial position in which the matched conference is controlled by the control means; And tracking means for tracking the matched sketch by receiving the output of the initial positioning means under the control of the control means.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 and 2, a method for tracking a sister in a video conference system according to the present invention includes model processing steps 100 to 105, a model selection step 201, a singer image processing step 202 and 206, A matching step 207, a skeptic tracking step 208, and a model changing step 210.

The model processing steps 100 to 105 extract a closed curve corresponding to each model face using the extracted boundary pixels by inputting a model image signal corresponding to one or more images of a conference person during a video conference, A model image signal storing step 100 for A / D converting and storing a model image signal corresponding to an infrared image of a piconet to be recognized at the time of a video conference, a boundary pixel extracting step for extracting a boundary pixel from the stored model image signal, Extracting a boundary direction using the extracted boundary pixels, comparing a brightness value of each pixel of the model video signal with a brightness value of surrounding pixels, and adding a vector to the boundary direction, (102), a boundary pixel having a larger brightness value among the pixels of the model video signal according to the number of the models, A closed curve extracting step (103, 105) for extracting a closed curve while moving along the direction, and a model face storing step (104) for recognizing and storing the inside of the extracted closed curve as a model face.

In the model selection step 201, one of the recognized one or more model video signals is selected.

In the conference person image processing steps 202 to 206, a conference person's face is recognized by extracting a closed curve corresponding to the conference person's face using the extracted boundary pixels inputted through the camera, A mask image signal storing step 202 for A / D converting the mask image signal input through the mask image signal storing unit 202, a boundary pixel extracting step 203 for extracting a boundary pixel from the stored mask image signal, A boundary direction extracting step (204) of extracting a boundary direction by using the boundary direction, comparing the brightness value of each pixel of the conference video signal with an average of brightness values of surrounding pixels and adding a vector to the boundary direction, Extracts a closed curve while moving along the extracted boundary direction from a boundary pixel having a larger brightness value among the pixels of the conference video signal A closed curve extracting step 205, and a delegate face storing step 206 for storing the inside of the extracted closed curve as a delegate face.

Wherein the boundary pixels are determined by comparing vertical and horizontal differential values of the stored model video signal with a threshold value for setting a boundary, and the boundary direction is determined by comparing arc tangents of vertical and horizontal differential values of the boundary pixels, Value (aran (h / v)).

In the matching step 207, the closed curve of the selected model face is matched with the closed curve of the recognized sister face.

In the step of tracking the delegate, the step of setting a window 300 for tracking the matched delegate to move the camera, and setting a window having a predetermined size including the delegate's face at the initial position with the matched delegate, A binarization step of binarizing the pixel values in the set window, a threshold setting step of setting a threshold value for tracking the center point of the matched spermator, a binarization value search step of searching the binarized pixel value to find whether the binarized pixel value is greater than a predetermined threshold value, And a camera movement and tracking step (step 304) of moving the camera and performing tracking if the binarized value is greater than the threshold value (step 303).

In the model changing step 210, when the other one of the recognized one or more model video signals is selected, the process proceeds to the matching step 207.

The detailed operation of the video conferencing system according to the present invention will be described with reference to the accompanying drawings.

First, the infrared model video signal corresponding to all the meeting persons attending the video conference should be taken through the camera to extract the face shape of the model.

That is, the model processing steps 100 to 105 are performed to extract and recognize the outline of each model face by inputting respective model image signals corresponding to images of one or more delegates as a reference, .

First, a model image signal corresponding to an image of a conference person to be recognized at the time of a video conference is taken through an infrared camera, A / D converted, and stored (100).

A signal received from an infrared camera, for example, an NTSC signal, is converted into a digital image I (x, y) having a value within a specified range on a two-dimensional matrix after A / D conversion. This image is stored in memory and used as input image.

After the model image signal storage step 100 is performed, a boundary pixel extraction step 101 for extracting boundary pixels from the stored model image signal is performed.

(H, v) in the horizontal and vertical directions by differentiating the stored model video signal in the vertical and horizontal directions, and then calculates the difference (h ² + v ² ) ^1/2 of the differential value The boundary pixels are found by comparing threshold values.

A digital image I (x, y) signal having a value within a specified range on a two-dimensional matrix is differentiated in the horizontal and vertical directions as shown in the following equations (1) and (2).

[Formula 1]

I (x + 1, y-1) + I (x-1, y) + I -I (x + 1, y-1) = h

[Formula 2]

I (x, y + 1) -I (x, y + 1) -I (x + 1, y + 1) = v

That is, as shown in FIG. 3A, the horizontal differential includes brightness values A4, A5, and A6 of the left pixel and brightness values A1, A2, and A3 of the right pixel with the reference pixel x, A7 and A1 on the upper side and the brightness values A6, A8 and A3 on the lower side are centered on the reference pixel x and y, It is done by retrieving the car.

As shown in the above equation (1), the horizontal differential value is calculated to calculate the horizontal differential value h and the vertical differential value is calculated as shown in the above equation (2) to calculate the vertical differential value (v) (H ^2, v ² ) ^1/2 ) of the differential values (h, v). That is, the magnitude ((h ² + v ² ) ^1/2 ) of the differential value is calculated by squaring and adding the vertical and horizontal differential values (h, v), and then taking the square root again.

When the magnitude ((h ² + v ² ) ^1/2 ) of the calculated differential value is compared with a threshold value for determining a boundary and is larger than the threshold value for threshold value determination, the reference pixel A (x, y) .

The size of the differential value and the comparison process are performed for each pixel of the model video signal to extract all boundary pixels in the model video signal of one frame.

After extracting the boundary pixels, the boundary direction is calculated (102), and the boundary direction is calculated as the arctangent value (aran (h / v)) of the differential values in the vertical and horizontal directions of the boundary pixels.

For this purpose, if a lookup table is prepared in the internal ROM, an arctangent result for the corresponding value (h / v) can be obtained, so that the boundary direction can be calculated through this. At this time, the calculated arc tangent value (atan (h, v)) has a value ranging from -180 degrees to 180 degrees, so it is divided into 8 steps. (-1, -1), (-1,0), (-1,1), (0,1), and (0,1) in the , (0, -1), (1, -1), (1,0), and (1,1).

On the other hand, the z component is determined as '-1' or '1' by comparing the brightness value of the current pixel with the average brightness value of the surrounding eight pixels as shown in FIG. 3C. That is, if the brightness value of the current pixel is larger than the average value of the surrounding eight pixels, the downward vector is added by modifying the x and y directions. If the brightness value of the current pixel is smaller than the average value of the surrounding eight pixels, Modification is applied and an up vector is added. In other words, if the brightness value of the current pixel is larger than the average value of the surrounding eight pixels, the z component is corrected to -1 in the x and y directions, and if the brightness value of the current pixel is smaller than the average value of the surrounding eight pixels Modify the x and y directions so that the z component is '1'.

After the boundary direction extraction step 102 is performed, the closed curve extraction step 103 is performed to extract the model face.

As shown in FIG. 4, a boundary pixel having a large brightness value, that is, a boundary pixel having a brightness value higher than 1/10, moves along the extracted boundary direction x, y, and z, When this closed curve is formed, the face of the delegate is extracted.

On the other hand, if no closed curve is formed, a boundary pixel whose brightness value is included in the upper 1/10 or more is selected to track the closed curve.

When the closed curve is formed, the inside of the closed curve becomes the corner portion, and the pixel value I (x, y) is set to '1' for this portion.

At this time, if there is no starting pixel other than the pixel value I (x, y) '1', the above process is terminated. In other words, if there is no other model, the above closed curve extraction step ends.

Thus, the face of the model extracted from the model image signal is used for matching with the face of the skeeter extracted from the singer image signal stored in the memory.

A model selection step (201) of extracting a linear feature for all the model video signals and terminating the model processing step when there is no other model video signal and selecting one of the recognized one or more model video signals.

That is, a model to which a video signal is to be transmitted is selected through a camera (201).

For example, from among many conferences attending a videoconference, a chairman who mainly talks is selected.

A conference person image processing step (202 to 206) for extracting and recognizing the outline of a sister face from a video signal of a conference person inputted through a camera is performed.

The process of image processing of the conference person is performed in the same way as the above model processing step.

That is, A / D conversion of a conference image signal input through a camera is performed (202), boundary pixels are extracted from the stored conference image signals (203), and boundary directions are extracted using the extracted boundary pixels (204) a vector is added to the boundary direction by comparing a brightness value of each pixel of the conference video signal with an average of brightness values of neighboring pixels, and a vector is added to each pixel of the conference video signal (205). The inside of the extracted closed curve is recognized as a face of the skein and stored (206).

At this time, as shown in FIG. 5, the extracted closed curves can be several according to the number of the skeptics.

As shown in FIG. 6, the faces of the plurality of skeiners extracted from the slave video signal are compared with the faces of the selected model extracted from the model video signal to determine whether or not they are matched (207). Correlation calculations are used to determine matching.

In order to calculate the correlation between the selected model and a number of delegator faces, the correlation function to be used must be determined. Correlation functions for the correlation calculation include NCCF (Normalized Cross Correlation Function), MSE (Mean Square Error) and MAE (Mean Absolute Error) as shown in Equation 3, Equation 4 and Equation 5 below. ).

[Formula 3]

NCCF (p, q) = [ ΣI n (i, j) · I n + 1 (i + p, j + q)] / [(ΣI n 2 (i, j)) 1/2 · (ΣI n + ₁ ² (i + p, j + q)) ^1/2 ]

[Formula 4]

MSE (p, q) = E ([I n (i, j) - I n + 1 (i + p, j + q)] 2)

[Formula 5]

MAE (p, q) = E (┃I n (i, j) - I n + 1 (i + p, j + q) ┃)

Where E (·) represents the mean.

As the correlation function, NCCF is most suitable but MAE is used considering the calculation amount.

The higher the correlation between images, the smaller the value of MAE is. Therefore, when MAE is used as a correlation function, it is judged that the face of the lowest chairman is matched and becomes the face of the chairman corresponding to the model. At this time, the matching delegate face becomes the initial position, and the delegate tracking step is performed through the center point tracking method.

That is, if the matching is performed, the initial position can be estimated for the conference person, so that the camera is moved and tracked using the initial information (208), which will be described in detail with reference to FIG.

After the recognition function is performed, the initial position of the face to be tracked can be grasped in the input image. In the tracking step, the face portion is continuously tracked by using the initial position. The tracking method used at this time is the center point tracking method.

Since the infrared image is relatively easy to extract the face part from the background than the general image, the center point tracking method is appropriate and the detailed process of the infrared point image is explained.

First, a window setting step 301 is performed to set a window having a predetermined size including a face of a skeptic at an initial position where the registered skeeter is present.

Generally, the center point tracking is performed on the entire image. However, in the present invention, since the initial position can be known through the recognition function, the tracking function is limited to the vicinity of the initial position. That is, a window that is a window of a size including a face of a person can be set to perform the tracking function only in this window.

Here, the size of the window may be a size that includes the size of the face in the image but does not include the face of another person excluded from the process of image processing of the person in the conference.

After the window setting step 300 is performed, a binarization step (step 301) of binarizing the pixel values in the set window and a threshold value setting step of setting a threshold value Thres for tracking the center point of the matched spermator 302).

Since the face part can be easily extracted by setting the threshold value in the window, the binarization value searching step is performed by setting the threshold value as shown in the following equation (6).

[Formula 6]

I (x, y) = 1 if I (x, y)> Thres

If the binarization value is retrieved from the above equation (6) and the value of the image is '1', it is possible to track the delegator. In other words, when the motor is driven so that the camera moves to the portion indicated by '1' by the initial recognition and the singer moves, the portion indicated by '1' moves together, and the movement information is transmitted to the motor again to move the camera (304).

Therefore, since the window is moved with the camera, continuous tracking becomes possible.

On the other hand, when two or more faces are photographed in the sudden movement of the target face or the window during the movement of the camera for tracking, the recognition target face must be recognized again through recognition function again by activating the recognition function.

That is, if it is impossible to track the delegator during the camera movement and tracking step 304, the process proceeds to the delegator image processing steps 202 to 206 to re-recognize and re-track the delegator 305 and 306 .

8, the apparatus for tracking a skeptic of a video conference system includes an image storage 600, a recognition unit 700, a control unit 500, an initial position determination unit 800, a tracking unit 900, And a motor unit 420.

The image storage unit 600 A / D-converts the image signals of the respective models and conference persons corresponding to images of one or more meeting persons during a video conference and stores the converted image signals. An A / D (Analog / Digital) converter 510 for A / D-converting each of the model image signals and the delegator image signals corresponding to images of one or more delegates at a time, And a memory 620 for storing the image signal output from the A / D converter 610 and outputting the image signal to the recognition unit 700. [

The recognition unit 700 extracts a closed curve corresponding to the face of each model and the delegate using the extracted boundary pixels of the model output from the image storage unit 600 and the delegator image signal, A boundary pixel extracting unit 710 for extracting boundary pixels from a model and a conference image signal stored in the image storing unit 600, A boundary direction extracting unit 720 for extracting a boundary direction and comparing a brightness value of each pixel of the model and conference video signals with an average of brightness values of neighboring pixels and adding a vector to the boundary direction, A boundary line of the model and the delegator image signal is shifted along the boundary direction extracted from the boundary pixel having a large brightness value according to the boundary direction extracted from the boundary 720, Output to consist of a closed curve extraction unit 730 to recognize a face model and meeting chair.

The control unit 500 includes a CPU and controls the operation of the image storage unit 600 and the recognition unit 700. The control unit 500 matches the outline of the face of the delegator with the model recognized by the recognition unit 700, To control the tracking of the person.

The initial position determining unit 800 sets a window of a predetermined size at the initial position where the matching conference is performed according to the control of the controller 500. [

The tracking unit 900 receives the output of the initial positioning unit 800 under the control of the control unit 500 and tracks the center point tracking of the registered meeting point.

The motor unit 400 includes a motor 410 for moving the camera 300 and a motor driver 420 for driving the motor 420 to move the camera according to the control of the controller 500 do.

The operation of the apparatus for tracking a skeptic of the video conferencing system according to the present invention will now be described.

First, a signal from the infrared camera 300 is stored in a memory to generate an input image. A signal input from an infrared camera, for example, an NTSC signal, is converted into a digital image I (x, y) having a value within a specified range on a two-dimensional matrix through an A / D converter 610. This image is stored in a memory (620) and used as an input image.

After the digital image is stored in the memory 620, the function of storing the signal in the memory 620 in the A / D converter 610 is turned off until the operation of the closed curve extracting unit 730 is completed in the control unit 500 So that the input image does not change.

When the closed curve extraction of the closed curve extracting unit 730 is completed, the A / D converter 610 is turned on to receive a new input image again.

The boundary pixels are extracted from the image signal I (x, y) output from the memory 620 by differentiation in the boundary pixel extracting section 710. That is, by the above-mentioned equations (1) and (H ² + v ² ) ^1/2 ) is calculated again and the set threshold value for determining the boundary line (h ^2, v ² ) is calculated again after horizontal and vertical differentiations are performed to obtain differential values And finds a boundary pixel.

In addition, the boundary direction extracting unit 720 calculates the boundary direction by arctangent values aran (h / v) of the vertical and horizontal differential values of the extracted boundary pixels, .

The closed curve extraction unit 800 starts from a boundary pixel belonging to the upper 1/10 or more of brightness values and moves along the extracted (x, y, z) direction. When the movement trajectory forms a closed curve, it is extracted with the face of the delegate. The process of extracting the closed curve is performed as many times as the number of the attendees of the conference. The closed curve thus extracted becomes the face of the conference person.

The controller 500 performs the above process on the video signal of the model in advance, and constructs the face of the model having the face of one or more meeting persons photographed by the camera in advance as a database.

On the other hand, the face of the skeeter is extracted through the above process for the infrared image signal of the singer inputted through the current camera. That is, the conference video signal is stored in the memory 620 through the A / D converter 610, and then transmitted to the conference 610 through the boundary pixel extracting unit 710, the boundary direction extracting unit 720, The face of the person is extracted.

The extracted face of the skeeter is stored in the memory. At this time, the controller 500 extracts the same singer as the one selected from the plurality of models through the matching process, .

As described above, the method and apparatus for tracking a skeiner of a video conference system according to the present invention has a function of extracting a closed curve by using boundary pixels and directions and tracking a singer after recognizing the scepter, .

Claims (9)

A model processing step (100 to 105) for extracting a closed curve corresponding to each model face using the extracted boundary pixels by inputting a model video signal corresponding to one or more conference images during a video conference and recognizing the model face, ; A model selection step (201) of selecting one of the recognized one or more model image signals; A conference person image processing step (202 to 206) for recognizing a conference person's face by extracting a closed curve corresponding to the conference person's face using the extracted boundary pixels inputted through the camera; A matching step (207) of matching a closed curve of the selected model face with a closed curve of the recognized sister face; A sceinator tracking step (208) for tracking the matched scepter and moving the camera, and And a model changing step (210) of advancing to a matching step (207) when another one of the recognized one or more model video signals is selected. The method of claim 1, wherein the model processing steps (100 to 105) include: a model image signal storage step (100) of A / D-converting a model image signal corresponding to an infrared image of a singer to be recognized at the time of a video conference; A boundary pixel extracting step (101) of extracting a boundary pixel from the stored model video signal; Extracting a boundary direction using the extracted boundary pixels, comparing a brightness value of each pixel of the model video signal with an average of brightness values of neighboring pixels, and adding a vector to the boundary direction; A closed curve extracting step (103, 105) for extracting a closed curve from a boundary pixel having a larger brightness value among the pixels of the model video signal according to the number of the models, while moving along the extracted boundary direction; And a model face storage step (104) for recognizing and storing the interior of the extracted closed curve as a model face. 2. The method of claim 1, wherein the step of processing the conference image comprises steps of: (202) storing a conference person image signal for A / D conversion of a conference person image signal inputted through a camera; A boundary pixel extraction step (203) of extracting boundary pixels from the stored conference image signals; A boundary direction extracting step (204) of extracting a boundary direction using the extracted boundary pixels and comparing a brightness value of each pixel of the conference video signal with an average of brightness values of surrounding pixels and adding a vector to the boundary direction, ; A closed curve extracting step (205) of extracting a closed curve while moving along the extracted boundary direction from a boundary pixel having a larger brightness value among pixels of the conference image signal according to the number of the conference persons, And a delegate face storage step (206) for storing the delegate face as a delegate face. 4. The video conference system according to claim 2 or 3, wherein the boundary pixels are determined by comparing the vertical and horizontal differential values of the stored model video signal with a threshold for discriminating the boundary line. The video conferencing system of claim 4, wherein the boundary direction is calculated as an arctangent value (aran (h / v)) of the differential values in the vertical and horizontal directions of the boundary pixels. The method of claim 1, wherein the step of tracking the delegate (208) comprises: a window setting step (300) of setting a window of a predetermined size including the face of the delegator at an initial position with the matched delegate; A binarization step (301) of binarizing pixel values in the set window; A threshold setting step (302) of setting a threshold for tracking the center point of the matched skeiner; A binarization value search step (303) for searching whether the binarized pixel value is larger than a threshold value set; And a camera movement and tracking step (304) for moving the camera and performing tracking if the binarized value is greater than the threshold value. An image storage means (600) for A / D converting and storing the image signals of the respective models and conference persons corresponding to images of one or more conference persons during a video conference; A recognizing means 700 for recognizing a model face by extracting a closed curve corresponding to a face of each model and a skeeter using the model outputted from the image storing means 600 and the boundary pixels extracted by inputting the sketcher image signal, ); A control unit 500 for controlling the operation of the image storage unit 600 and the recognition unit 700 and for controlling the recognition unit 700 to match the outline of the dancer's face recognized by the recognition unit 700 and to track the dancer, ; An initial positioning means (800) for setting a window of a predetermined size at an initial position in which the matched sketcher is under control of the control means (500); And And a tracking unit (900) for tracking the matched sketch by inputting the output of the initial positioning unit (800) under the control of the control unit (500). [7] The method of claim 7, wherein the image storage means (600) comprises: a control unit (600) for controlling each of the model image signals and the delegator image signals corresponding to one or more images of the delegator A / D (Analog / Digital) converter 510 for converting the analog signal into digital data; And a memory (620) for storing the video signal output from the A / D converter (610) under control of the control means (500) and outputting the video signal to the recognition means (700) Delegator tracking device. 8. The image processing apparatus according to claim 7, wherein the recognizing unit (700) comprises: a boundary pixel extracting unit (710) for extracting boundary pixels from the model and the sketcher image signal stored in the image storing unit (600); The boundary direction extracted by the boundary pixel extracting unit 710 is used to extract the boundary direction, and the average value of the brightness values of the surrounding pixels and the brightness values of the pixels of the model and the conference image signals are compared, A boundary direction extracting unit 720 for adding the boundary direction extracting unit 720; And the boundary direction extracting unit 720 extracts a closed curve while moving along the extracted boundary direction from a boundary pixel having a large brightness value among the pixels of the model and the delegator video signal according to the boundary direction extracted by the boundary direction extracting unit 720, And a closed curve extracting unit (730) for recognizing the face as a face.