KR100457302B1 - Auto tracking and auto zooming method of multi channel by digital image processing - Google Patents

Abstract

The present invention relates to a multi-channel auto tracking and auto zoom method using image processing, and more particularly, to a camera having a pan / tilt / zoom function. The present invention relates to a method of tracking a face of a person to be tracked in real time.

The present invention relates to a multi-channel auto tracking and auto zoom method using image processing, comprising: a first step of synthesizing an image signal input from four cameras having a pan / tilt / zoom function into a single image divided into four; A second step of acquiring frame data from the synthesized single image of the first step; A third step of color converting the RGB image from the frame data of the second step into an HSV image; A fourth step of automatically setting ROI from the HSV image of the third step; A fifth step of determining Low / Hi threshold data from the motion ROI and the candidate center point obtained by the automatic ROI setting of the fourth step; A sixth step of obtaining a center point and an area from the threshold data of the fifth step; A seventh step of determining and controlling tracking and zoom success from the center point and area of the sixth step; And an eighth step of controlling the camera upon successful tracking and zooming of the seventh step.

Unlike regular static cameras, you can track objects while automatically changing the field of view (FOV) of the camera, so you can expect efficient surveillance and storage space savings, and tracking and zooming on multiple channels. It's easy to control and low additional cost because it can be used on one PC, and intelligent tracking and zooming function is possible by auto-correction function in case of tracking and zoom failure, and tracking and zoom module is software type. It is easy to apply to an application system, can be converted to hardware, and can be used as a module that can replace the input of a mouse or keyboard by the movement of a user's face in a haptic game.

Description

Auto tracking and auto zooming method of multi channel by digital image processing}

The present invention relates to a multi-channel auto tracking and auto zoom method using image processing, and more particularly, to a camera having a pan / tilt / zoom function. The present invention relates to a method of tracking a face of a person to be tracked in real time.

With the recent 9.11 terrorism, security interests are increasing in each country. As a representative security product, CCTVs and DVRs (Digital Video Recorders) are being converted into DVRs that can store digital images. The fixed camera is connected and recorded and monitored, but only one area can be monitored because the FOV (Fields Of View) of the fixed camera is limited.

Conventionally, when the tracking and zooming functions are implemented through image processing, only the automatic tracking function is performed, and the zooming function can monitor only a limited area due to the disadvantage that the user needs to operate manually and the fixed camera. There was a problem that can not track the object.

The conventional method of tracking conventionally, as shown in FIG. 1, when an image is acquired from a pan / tilt / zoom camera and the analog image data is transferred to a control PC, the analog image is obtained from a frame grabber embedded in the PC. After converting the data into digital image data and performing the image processing, the tracking was performed automatically by transferring the resulting camera control data back to the original camera. Had to be controlled manually by the PC to enable the operation.

As shown in FIG. 1, the performance of the system depends on the digital image processing method performed by the control PC. The conventional tracking algorithm method uses a simple search algorithm and a block matching algorithm (BMA) algorithm. In some cases, the SNAKE algorithm may be used when using the optical flow algorithm.

The four conventional tracking algorithms are described in detail as follows.

First, in the simple search algorithm, one frame image is divided into four blocks, as shown in FIG. When there is a difference, it is possible to search for the motion in a manner of informing the information by considering the motion as occurring, but there is a problem that cannot be used for tracking.

Second, the Block Matching Algorithm (BMA) algorithm divides the frame of the image sequence into random block units, finds the block most similar to the next frame based on the block that is expected to move, and moves the reference point to the motion vector. As a method of recognizing and estimating a vetor, all pixels in a block must be moved in parallel, and an intensity between image frames must be constant.

The advantage of the BMA algorithm is that the computation time is relatively fast, which is suitable for real-time processing and works well when the image sequence is fast enough. However, there is a problem in that an error occurs when the brightness information of the image changes abruptly or when the pixel value in the block changes.

The BMA algorithm is calculated by the following equation.

Third, in the case of using the optical flow algorithm, as shown in FIG. 3, a method of estimating the acceleration and the motion vector of a moving object by calculating the amount of change in the amount of image light generated by the time difference dt of the image sequence, as shown in FIG. Because the operation is performed on the pixel, a more accurate motion vector can be obtained than the BMA. However, this method also has a problem in that the brightness of all objects must be constant regardless of time, and the surrounding pixels of the image should move only in a similar direction.

The optical flow algorithm is calculated by the following equation.

C (i, j) = 0 for all (i, j)

Fourth, the SNAKE algorithm is a method of matching the deformed model of the image by energy minimization method (Kass, 1987). For this, the balloon model is applied to the basic snake by applying the Kalman filter and applying the force of additional contraction and expansion. Many application techniques, including cohen, are used. This method has the problem of not finding all the contours in the image, other high-dimensional image processing, interaction with other image data in time and space, and the user to designate an initial node of the snake.

The SNAKE algorithm is based on the following equation.

Figure 4 illustrates the process of finding the exact center of the moving object using the snake algorithm.

As described above, the four algorithm methods have the following technical problems.

First, when the simple search algorithm is used, only the movement can be detected, and the pan / tilt / zoom functions cannot be used automatically because only the moving object can be determined.

Second, in case of using BMA (Block Matching Algorithm), there is a possibility of irregular movement in various directions as well as parallel movement when moving between actual blocks, and tends to easily miss an object to be tracked even if the brightness information changes only a little. Only this information could not perform the auto zoom function.

Third, in the case of using the optical flow algorithm, it was also sensitive to the change in the amount of ambient light, so that it was impossible to establish the exact direction and the center point of the moving object.

Fourth, in the case of using the SNAKE algorithm, the center point of the moving object can be determined most accurately, but the computational amount is not suitable for real-time processing, and the user had to initialize the initial snake node.

Accordingly, the present invention is to solve the above-mentioned problems of the prior art, the camera with a built-in pan / tilt / zoom function to acquire the image, the digital image processing to calculate the position and area of the face to be tracked The present invention provides an algorithm suitable for robustness and real-time processing that can overcome the shortcomings of limited FOV and implement tracking and zooming functions through image processing by automatically performing pan / tilt / zoom using the information. There is this.

Figure 1 shows the technical configuration and flow when performing the tracking function by the prior art.

2 shows a method for tracking by a simple search algorithm in the prior art.

Figure 3 shows a tracking method by the conventional Opital Flow algorithm.

4 shows a method for tracking by the conventional SNAKE algorithm.

5 illustrates an algorithm for implementing auto tracking and auto zoom according to the present invention.

6 shows an example of performing a low pass region filtering process using a 3 × 3 mask performed by the present invention.

FIG. 7 illustrates an image that is subtracted to obtain a candidate center point by the image subtraction method performed by the present invention.

Figure 8 shows the Hue Histogram is set to obtain the Low / High threshold value performed by the present invention.

9 illustrates an image obtained by setting a pixel value corresponding to a low / high threshold value performed by the present invention to a blue color.

10A-10D illustrate the process of monitoring the number of pixels within a threshold to confirm tracking success performed by the present invention.

11 shows a dome camera having a pan / tilt / zoom function as an apparatus for implementing the present invention.

12 shows a low cost frame grabber as an apparatus for implementing the present invention.

13 illustrates a four channel multiplexer as an apparatus for implementing the present invention.

14A to 14C illustrate an automatic tracking process for one channel implemented by the present invention.

15A to 15B illustrate an auto zoom process for one channel.

16 shows a successful auto tracking and auto zoom process for four channels.

The object of the present invention is to synthesize the video signal input from four cameras as shown in Figure 5 into four channels, to obtain the frame data, color conversion, automatic ROI setting, Low / Hi Multi-channel using image processing, characterized in that it has a Hue color probability distribution based algorithm comprising the steps of obtaining a threshold value, obtaining a center point and an area, determining tracking success and zoom success, and controlling a camera It is to realize the auto tracking and auto zoom method.

Details of the above object and technical configuration of the present invention and the effects thereof according to the present invention will be more clearly understood by the following detailed description with reference to the drawings showing preferred embodiments of the present invention.

5 illustrates an algorithm including the following steps to perform auto tracking and auto zoom based on Hue color probability distribution.

The algorithm comprises a first step of synthesizing an image signal input from four cameras having a pan / tilt / zoom function into four divided single images, and a second step of acquiring frame data from the synthesized single image of the first stage. A third step of color conversion of the RGB image from the frame data of the second step to the HSV image, the fourth step of automatically setting a region of interest (ROI) from the HSV image of the third step, the automatic of the fourth step A fifth step of determining Low / Hi threshold data from the motion ROI obtained by the ROI setting and the candidate center point, a sixth step of obtaining a center point and an area from the threshold data of the fifth step, and tracking from the center point and the area of the sixth step And a seventh step of determining and controlling whether or not the zoom fails, and an eighth step of controlling the camera when the tracking and zooming of the seventh step are successful.

The algorithm of each step is described in detail below.

In the above algorithm, the first step of synthesizing the video signals input from four cameras having pan / tilt / zoom functions into four divided single images is performed by four analog cameras incorporating NTSC cameras having pan / tilt / zoom functions. Using the Quad Chipset, the acquired data can be synthesized into four images. Unlike conventional static cameras, it has a technical advantage that four analog cameras equipped with NTSC cameras with pan / tilt / zoom functions can move the F.O.V (Fields Of View) of the camera by capturing moving objects as images.

The second step of acquiring frame data from the synthesized single image of the first stage is to use a direct show in a PC program to display one synthesized image by a message or a thread. Acquire. At this time, the acquired image is converted into analog image data into digital image data using a low-cost frame grabber shown in FIG.

The third step of color conversion of the RGB image from the frame data of the second step into the HSV image is performed by converting the bits into fewer bits than the originals (24 bits) in order to shorten the processing time in the color image processing. Normally, the three primary colors are combined into one complex color signal by inputting the RGB (red, green, blue) color signals, which are the basic colors representing the color of the color image, to the cathode ray tube (CRT) display or computer monitor separately. Compared to the method of inputting with a single signal line, the tuner or the image circuit uses a normalized RGB method or YUV type which displays a complex but clear and clear image, but in this algorithm, the HSV (Hue Saturation Value), which is robust to changes in lighting Convert the color model to use.

In the fourth step of automatically setting the ROI from the HSV image of the third step, a low pass spatial filter and an image subtract method can be used to automatically determine the initial position of the moving object. Conventional techniques usually require the user to specify the initial position when applying the tracking algorithm, but this algorithm solves this problem.

The low pass filter removes the noise of the image, usually performs a convolution operation, and usually uses a 3x3 or 5x5 mask.

6 illustrates an example of a method of performing a convolution operation using a 3 × 3 mask.

In the image subtract method, a candidate center point of a moving image may be obtained by differentiating an image passing through a low pass region filter and a subsequent image.

FIG. 7 shows a result of finding a candidate center point of a moving image by dividing an image passing through a low pass region filter and a next image by an image subtraction method, which is shown in white in the figure.

The fifth step of determining the low / hi threshold data from the motion ROI and the candidate center point obtained by the automatic ROI setting of the fourth step is based on the center point when the motion candidate ROI and the candidate center point of the fourth step are determined. Set a hypothetical rectangle of 15, set a Hue Histogram for all the pixels in that rectangle, and then get a Low / Hi threshold.

FIG. 8 illustrates a process of determining a low / hi threshold after setting a virtual rectangle of 15 × 15 with respect to a center point, setting a Hue histogram for all pixels within the rectangle, and using MAX as a hi threshold value for R1 and R2. Each represents a low threshold value, the vertical axis represents a value value (I), and the horizontal axis represents a Hue value (H).

The Hue Histogram is set, and then a Hue value (H) and a Value value (I) are calculated by the following equation for all pixels in a specific method of obtaining a Low / Hi threshold.

The saturation value is calculated by the following equation, and A and B are obtained from the H value.

The sixth step of obtaining the center point and the area from the threshold data of the fifth step is

If the pixel value corresponding to the Low / Hi threshold value of the ROI obtained in the fifth step is set to blue in a computer program, an image as shown in FIG. 9 can be obtained, and the position of the value indicated in blue is used. The center point and the area of the object to be tracked can be obtained.

FIG. 9 illustrates an image displayed by setting a pixel value corresponding to a low / hi threshold of ROI to blue in a computer program when the object to be tracked is a human face.

The seventh step of determining and controlling the tracking and zooming success from the center point and the area of the sixth step is performed to determine and control the final tracking and zooming success.

The process of determining and controlling the final success of the tracking monitors the number of pixels within the threshold and indicates that the tracking fails when the number is less than a certain number. In one embodiment, the algorithm determines that the ROI is smaller than 10 × 10 pixels. Tracking is considered to have failed. If the tracking fails, the initial ROI can be set to the full size of the screen, and the process can be repeated until the correct position is calculated by returning to the process of step 6 again to prevent the tracking failure. Follow the steps. 10A to 10D show a series of processes for determining and controlling the success of final tracking by four channels.

14A to 14C illustrate a series of automatic tracking processes by one channel as an embodiment.

The process of determining and controlling the final success of auto zoom calculates the size of the final ROI and the screen size, and checks whether the size of the ROI is smaller than 10 × 10 pixels or more than 150 × 110 pixels on the screen. If it becomes larger, the camera zooms out, and if it decreases, the camera zooms in to obtain a constant size of the object to be tracked. At this time, if the ROI decreases, a value similar to the Hue value to be tracked will appear, and eventually the zooming function will fail. In this case, if the ROI is forcibly reduced, the ROI will converge to a certain size and execute a successful auto zoom. Done.

15A to 15B illustrate an embodiment of performing auto zoom on one channel. FIG. 15A illustrates zoom in when the size of the ROI is smaller than 10 × 10 pixels. FIG. 15B illustrates the size of the ROI. Is zoom out when is larger than 150 × 110 pixels.

The eighth step of controlling the camera when the tracking and zooming of the seventh step is successful is controlling the four-channel camera by using the center point and the zoom factor of the finally determined object. This is implemented via an RS 485 or RS 422 interface control module.

FIG. 16 illustrates successful implementation of the auto tracking and auto zoom functions for four channels with the implementation of the above algorithm.

One preferred embodiment for implementing the algorithm is a PC-based device that controls multiple cameras by multi-channel tracking and zooming by acquiring multiple camera images from a PC, and multi-channel automatic tracking using the image processing of the present invention. And the specific spec for realizing the auto zoom function is based on the Pentium III 1.2GHz CPU using Windows to synthesize the images acquired from the trackable four-channel multiplexer and four analog cameras shown in Figure 13 into a single image Quad Chipset, 320 × 240 (QVGA) image resolution, tracking speed of about 26 frames / sec (using MS Direct-Show), 1 ~ 8x zoom (depending on camera spec), communicating with camera It consists of interface control module of RS-485 type or RS-422 type.

In more detail, a PC-based device for realizing a multi-channel auto tracking and auto zoom function using the image processing will be described.

The PC-based device divides the video signal input from the four pan / tilt / zoom cameras and the N pan / tilt / zoom cameras into four divided image signals to capture an image of a moving object. An image synthesizer composed of a multi-channel multiplexer for compositing a single image and frame data from the four synthesized single images, image conversion from the frame data, ROI setting from the image conversion, and the ROI setting. An image control unit comprising a control PC capable of determining Low / Hi threshold data from the motion ROI and candidate center points obtained by the controller, obtaining a center point and an area from the threshold data, and determining and controlling tracking and zoom failure from the center point and area. And an interface capable of controlling the camera upon successful tracking and zooming. Consists of a video transmitter such as RS 485 or RS 422 is configured with a control module.

Another preferred embodiment for implementing the above algorithm is a case of making a stand alone processor to operate the tracking and zooming algorithm of the present invention in a camera. As a specific device for realizing this, a camera having a pan / tilt / zoom function, a digital signal processor (DSP) for image processing, and a chip (ROM), a ROM, and a RAM for an open software description (OSD) are required.

In more detail, a multi-channel auto tracking and auto zoom camera using the independent processor-based image processing will be described.

The multi-channel auto tracking and auto zoom camera using the independent processor-based image processing includes an image receiving unit consisting of four pan / tilt / zoom cameras and an image signal input from the four cameras to capture an image of a moving object. Is synthesized into four divided single images, image data is obtained by converting the frame data from the single image, ROI is set from the converted image, and Low / Hi threshold data is obtained from the motion ROI and the candidate center point obtained by the ROI setting. And an image processor for determining a center point and an area from the threshold data, determining and controlling tracking and zoom failure from the center point and area, and controlling a camera when the tracking and zooming are successful.

The image processor implements a multi-channel auto tracking and auto zoom function using image processing by embedding an image processing DSP, an OSD chip, a ROM, and a RAM in the camera.

The DSP for image processing is a microprocessor made of a single IC chip for signal processing by digital operation. The DSP is a dedicated microprocessor that greatly improves the architecture of a general-purpose microprocessor to pursue high speed continuity and compactness. The basic operations are filtering, freer transform, calculation of correlation function, encoding, modulation and demodulation, derivative, integration, adaptive signal processing, and so on. It is used in speech and communication systems such as speech synthesis, speech recognition, speech encoding, compression, high speed modem and echocanceller, and is used for high speed digital control such as image processing or servo motor control. Application is also practical.

The OSD for image processing is a next-generation version of the hypertext generation language (HTML) as a format for describing the features of the software, and is based on the extensible generation language (XML) being standardized by the W3C. Software features that can be defined are the corresponding operating system (OS), version information, the corresponding central processing unit (CPU), and the amount of blank disk required. If you send the files listed in the OSD at the time of software distribution, you can execute the proper installation and processing on the receiving client side.

Other industrially available fields of the present invention are applicable to the user monitoring and tracking function for a PC and an independent processor as a DVR camera when a user camera and an auto zoom function are applied in a video chat or telemedicine in a PC camera. When used as a user tracking module for VR and games, it saves storage space due to the tracking and zooming function when using the DVR, for example, when a clear resolution is required for the face of an intruder who wants to track in a bank vault. .

Therefore, unlike the conventional static camera, since the object can be tracked while automatically changing the FOV of the camera, an efficient monitoring and storage space can be expected, and the automatic tracking and auto zooming function for multiple channels is limited. It is easy to control because it can be used on multiple PCs, and the intelligent tracking and zooming function is not only possible due to the automatic correction function of tracking and zoom failure, but also the tracking and zoom module is software type. It is easy to apply to the system, can be converted to a hardware module, and can be applied as a module that can replace the input of a mouse or keyboard by the movement of the user's face in the haptic game.

Claims (15)

The present invention relates to a multi-channel auto tracking and auto zoom method using image processing. A first step of synthesizing the video signals input from the N cameras having the pan / tilt / zoom function into a single N divided image; A second step of acquiring frame data from the synthesized single image of the first step; A third step of converting an image from the frame data of the second step; A fourth step of setting a region of interest (ROI) from the image conversion of the third step; A fifth step of determining Low / Hi threshold data from the motion ROI and the candidate center point obtained by the fourth ROI setting; A sixth step of obtaining a center point and an area from the threshold data of the fifth step; A seventh step of determining and controlling tracking and zoom failure from the center point and the area of the sixth step; And And an eighth step of controlling the camera upon successful tracking and zooming of the seventh step. The method of claim 1, In the third step, color image conversion is a multi-channel auto tracking and auto zoom method using image processing, characterized in that for converting the image of the RGB 24-bit format to the HSV 24-bit image. The method of claim 1, The multi-channel automatic tracking and auto zooming method using image processing, characterized in that the setting of the ROI is automatic in implementing the fourth step. The method of claim 3, wherein A multi-channel auto tracking and auto zoom method using image processing, characterized by using a low pass region filter to automatically set the ROI. The method of claim 3, wherein A multi-channel auto tracking and auto zoom method using image processing, characterized in that for using the image subtraction method to automatically set the ROI. The method of claim 1, Multi-channel auto tracking and auto zoom using image processing, characterized in that using the Hue Histogram to obtain the Low / Hi threshold in implementing the fifth step Way. The method of claim 1, The multi-channel automatic tracking and automatic zoom method using the image processing, characterized in that determining the failure of the tracking in the implementation of the seventh step by the size of the ROI. The method of claim 7, wherein The multi-channel automatic tracking and automatic zoom method using image processing, characterized in that determining whether the tracking fails by the variable size of the ROI. The method of claim 1, The multi-channel auto tracking and auto zoom method using the image processing, characterized in that determining the zoom failure in the implementation of the seventh step by the size of the ROI. The method of claim 9, The multi-channel auto tracking and auto zoom method using image processing, characterized in that determining whether the zoom failure by a variable size of the ROI. The method of claim 10, And comparing the variable size of the ROI with a size smaller than 10 × 10 pixels or larger than 150 × 110 pixels, and determining that the zoom has failed when the ROI is corresponding to the variable size. An image receiving unit including N pan / tilt / zoom cameras for capturing an image of a moving object; An image synthesizing unit comprising a multi-channel multiplexer for synthesizing image signals input from the N pan / tilt / zoom cameras into N divided single images; Obtain frame data from the N synthesized single images, image transform from the frame data, set ROI from the image transform, determine Low / Hi threshold data from the motion ROI and candidate center point obtained by the ROI setting, and And an image controller configured to obtain a center point and an area from the threshold data, and to determine and control tracking and zoom failure from the center point and area; And Image transmission unit consisting of an interface control module for controlling the camera when the tracking and zooming success Multi-channel auto tracking and auto zoom device using a PC-based image processing, characterized in that consisting of. The method of claim 12, The interface control module of the transmitting unit is a multi-channel automatic tracking and automatic zoom device using a PC-based image processing, characterized in that using the RS 485 or RS 422. Image receiver consisting of N pan / tilt / zoom cameras to capture the image of the moving object and A video signal input from the N cameras is synthesized into N divided single images, image data is obtained by obtaining frame data from the single image, a ROI is set from the converted image, and a motion ROI obtained by the ROI setting. And determining a low / hi threshold data from a candidate center point, obtaining a center point and an area from the threshold data, determining and controlling tracking and zoom failure from the center point and area, and controlling a camera upon successful tracking and zooming. Processing Multi-channel auto tracking and auto zoom camera using an image processing comprising a. The method of claim 14, And the image processor includes an image processing DSP, an OSD chip, a ROM, and a RAM in the camera.