JP2003235035A - Object detecting method, object detecting apparatus employing the same, object tracking method, and object tracking apparatus employing the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object tracking method with high reliability for detecting an image of an object from an input image within an imaging view field acquired from an image pickup device having a zoom lens and tracking the object by template matching processing between the detected image of the object and an template image. <P>SOLUTION: Magnifying or reducing the template image on the basis of a change in a focal distance equalizes the size of an intruded object shown in the template image with the size of the intruded object shown in an input image for carrying out template matching so as to track the object within an imaging view field. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is designed to automatically detect an object that has entered the imaging field of view from a video signal input from an imaging device and automatically track the movement of the detected object. Involved in object tracking methods and devices.

[0002]

2. Description of the Related Art An image monitoring device using an image pickup device such as a television camera (hereinafter referred to as a TV camera) has been widely used. One of the surveillance systems using such a video surveillance device is to detect and track an intruding object such as a human being or a car that comes into the surveillance field of view while watching an image displayed on a monitor. There is a manned monitoring system in place. However, in recent years, instead of such a manned monitoring system, an intruding object is automatically detected from an image input from an image input unit such as a TV camera, and its movement is automatically tracked, so that a predetermined notification is given. There is a growing demand for automatic monitoring systems that can provide alarm measures.

In order to realize such an automatic monitoring system, first, an intruding object in the visual field is detected by a difference method or the like. The difference method compares the input image acquired from the image pickup device with a reference background image created in advance (that is, an image in which an object to be detected is not captured), obtains a difference in brightness value for each pixel, and then calculates the difference value. Is detected as an object. The partial image of the input image corresponding to the position of the intruding object detected in this way is registered as a template image, and the position where the degree of coincidence with the template image is the maximum is detected from among the sequentially input images. This method is widely known as template matching.

Usually, when a target object (for example, an object detected by the difference method) is tracked using template matching, the position of the target object detected by the matching process is detected because the change in the posture of the target object is tracked. The partial image is sequentially updated as a new template image (for example, see Patent Document 1). About this processing,
This will be described with reference to FIG.

FIG. 1 is a diagram for explaining the flow of processing for detecting an object that has entered the visual field of the image pickup apparatus by the difference method and registering the detected object as an initial template image so that it can be used for template matching. Is.
In FIG. 1, S01 is an input image, S02 is a reference background image,
S03 is the difference image between the input image S01 and the reference background image S02, S
04 is the binarized image of the difference image S03, S05 is the difference processing unit,
S06 is a binarization processing unit (Th), S07 is a human-shaped object in the input image S01, S08 is a human-shaped difference image in the difference image S03 corresponding to the human-shaped object S07, and S09 is a human-shaped object. It represents a human-shaped object (human-shaped binary image) in the binary image S04 corresponding to the difference image S08. Further, S10 is a circumscribed rectangle of the detected humanoid object S09, S11 is an extraction processing unit (CL) for extracting a specified area from the input image S01, and S12 is an extracted area (template image). Extracted image, S13 represents a template image.

In FIG. 1, first, an input image S01 of 320 × 240 pixels (320 × 240 pix) is input from the TV camera E01 to the intruding object monitoring device E05. Next, the difference processing unit S05 calculates the difference in the brightness value of each pixel between the input image S01 and the previously created reference background image S02, and sets the calculated difference value as the brightness value of each pixel in the difference image S03. To get. At this time, the humanoid object S07 in the input image S01 is changed to the humanoid difference image S07 in the difference image S03.
Appears as 08. Next, the binarization processing unit S06 sets the brightness value of the pixel of which the brightness value of each pixel of the difference image S03 is less than a predetermined threshold value (for example, less than 20) to “0”, and the brightness value of the pixel of the threshold value or more. Set the value to “255” (1 pixel (1
pix) is treated as 8 bits and explained below) to obtain the binarized image S04. At this time, the input image S0
The humanoid object S07 imaged in 1 is detected as the humanoid object S09 in the binarized image S04 (outline of the object detection processing by the difference method).

In FIG. 1, the circumscribed rectangle S10 of the humanoid object S09 detected by the difference method is further detected. Next, the extraction processing unit S11 extracts the circumscribed rectangle S from the input image S01.
The area represented by 10 is extracted. The extracted image is registered as the template image S13 in the extracted image S12 (outline of the initial template image registration process).

FIG. 2 is a diagram for explaining a conventional example of the flow of processing for detecting at which position in the input image the intruding object registered in the template image by template matching exists. M01 shown in FIG. 2 (a) represents the extracted image obtained by the difference method (corresponding to the extracted image S12 in FIG. 1), and in FIG. 2 (b), M02 is the imaging device E01.
The input image obtained by, M03 represents the template image. M05, which is one of the dotted areas shown in the input image M02, represents the position of the template image M03, and M06, which is one of the dotted areas shown in the input image M02, represents the search range for template matching. Also, dx, dy
Represents the width of the search range (dx is the horizontal direction, dy is the vertical direction), and is set according to the amount of apparent movement of the target object to be tracked (the amount of movement on the image). For example, dx
Set it to = 50 pix and dy = 15 pix. The template matching is a process of searching for a portion having the highest degree of coincidence with the template image M03 within the search range M06 of the input image M02. As the degree of coincidence, for example, an index called a normalized correlation obtained by the following equation (1) can be used.

[0009]

[Equation 1]

In the equation (1), f () represents the input image,
g () represents a template image. Also, (x, y) represents the coordinates in the search area M06 of the input image (called the matching target area), and (u, v) is the coordinates of the upper left (top left) in the figure of the template image M03 (the template image). Represents the coordinates at the time of registration). However, for all images, the upper left (top left) in the figure is the origin (0,0). W represents the width (horizontal length) of the template image, and H represents the height (vertical length) of the template image. In Equation (1), the bars (￣) on top of the mathematical formulas represent the matching target area in the input image of the mathematical formulas below and the average brightness value of the template image, respectively. It is represented by 3).

[0011]

[Equation 2]

[Equation 3]

The normalized correlation r (x, y, u, v) is the input image f
The brightness value distribution of the area of width W and height H (eg, M05 in Fig. 2 (b)) with the position (x, y) at the position (x, y) as the upper left (top left) coordinate, and the template image position of g (u, v) (u, v)
It represents the degree of coincidence with the brightness value distribution of the area of width W and height H (for example, M03 in FIG. 2 (a)) where is the coordinate of the upper left (top left). Input image f () and template image
If the brightness values of the corresponding pixels of g () are all the same, the degree of coincidence is 1.0. Template matching is a process of detecting a portion of the input image f () that has the highest degree of matching with the template image g (). That is, with reference to the position (u, v) of the template image g (), (x, y)
Is changed to u-dx ≤ x <u + dx, v-dy ≤ y <v + dy, and the normalized correlation r (x, y, u, v) expressed by equation (1) becomes the maximum ( Find x, y).

In addition to the normalized correlation, the input image f ()
Width W with the position (x, y) at the upper left (top left) coordinate
Area of height H and the position (u,
The average value of the absolute values of the differences in the brightness values of the pixels at the respective corresponding positions in the area of width W and height H with v) as the upper left coordinate may be used as the degree of coincidence. In this case, if the brightness values of each pixel of the input image f () and the template image g () are all the same value, the degree of coincidence becomes 0, and the input image
The larger the difference between the brightness value of each pixel of f () and the template image g (), the higher the value (the lower the degree of coincidence). Figure 2
In the example of (b), the humanoid object to be tracked exists at the position of M04, and the region M
07 is detected. This area M07 is a template image
Since the humanoid object captured in M03 and the humanoid object in the input image M02 are detected at the same position, the video monitoring device may determine that the humanoid object to be detected exists in the area M07. it can. That is, it can be seen that the intruding object has moved from the position of the region M05 to the position of the region M07. In this case, the amount of movement of the intruding object can be represented by, for example, an arrow M08 connecting the center of the region M05 with the center of the region M07.

Next, an example in which the template matching described in FIG. 2 is applied to an input image that is sequentially input to track an intruding object in the visual field will be described with reference to FIG. FIG. 3 is a diagram for explaining the operation of the object tracking process that sequentially performs the conventional template matching method. In FIG.
T01a, T02a, T03a, and T04a are times t0-1 and
Extracted images at t0, t0 + 1, and t0 + 2, T01c,
T02c, T03c, and T04c are times t0-1, t0, and t0, respectively.
Template images obtained at +1 and t0 + 2, T01b
, T02b, T03b, and T04b are times t0 and t0 + 1, respectively.
, T0 + 2, t0 + 3 respectively, T05 is a template matching processing unit (MT), and T06 is a template image updating unit (UD). Here, time t0 + n means that the time is n frames away from time t0, and the processing time for one frame is, for example, 100
It is msec.

The matching processing unit T05 compares the template image in the extracted image with the input image, detects the portion of the input image that has the highest degree of coincidence with the template image,
As a result, the positions T01e, T02e, T03e, and T04e of the intruding object to be tracked at times t0, t0 + 1, t0 + 2, and t0 + 3 are obtained (template matching processing). The template image update unit T05 uses the image of the portion with the highest degree of matching detected by the matching processing unit T04 as the position image of the new intruding object, and replaces the extracted image and the template image by using the position image. Update.

The template matching process and the template image updating process will be described with reference to FIG.
The description will be made in order of t0-1, t0, t0 + 1, t0 + 2, t0 + 3.
First, the template image T obtained at time t0-1
Template matching processing is performed using 01c and the input image T01b obtained at time t0. Here, in the first processing frame of the tracking processing, the template image T01c becomes the partial image S13 of the input image S01 corresponding to the position of the circumscribed rectangle of the humanoid object S09 detected by the difference method.
The template matching processing unit T05 is as shown in FIG.
, The position image T01e of the intruding object T01d is detected by the template matching described using (b). The template image updating unit T06 uses the input image T01b having the new position image (template) T01e as the extraction image, updates the extraction image from T01a to T02a, and based on the position image T01e of the intruding object, the template image T01c.
Update from T02c to This process is also performed at time t
By performing at 0, t0 + 1, t0 + 2, and t0 + 3, respectively, the intruding object in the visual field is located at the position images T01e, T02e, T03e,
It can be seen that they moved in the order shown in T04e.

[0017]

[Patent Document 1] Japanese Patent Application No. 2001-184525

[0018]

The above-mentioned conventional template matching is a process of detecting a portion of the input image having the highest degree of coincidence with the template image. Therefore, the intruding object and the input image shown in the template image are detected. It is necessary that the size of the intruding object to be detected therein is almost equal. Therefore, if the zoom lens of the image pickup apparatus is controlled to change the zoom magnification, the size of the intruding object shown in the template image and the size of the intruding object to be detected in the input image are different, and accurate position detection of the intruding object cannot be performed. There is a problem. In addition, whether or not the intruding object can be detected is
It is determined whether or not the maximum degree of matching obtained by template matching is equal to or greater than a predetermined value. When the normalized correlation is used, the predetermined value is, for example, 0.
It is 5. Further, when the zoom magnification of the imaging device changes, the size of the intruding object shown in the template image and the size of the intruding object to be detected in the input image differ, and the degree of coincidence decreases. If the degree of coincidence falls below a predetermined value, there is a problem that it is determined that the intruding object cannot be detected.

As described above, in the aforementioned object tracking method using template matching of the prior art, if the zoom lens is controlled to change the zoom magnification, the position of the intruding object cannot be accurately detected, and further, the intruding object's position cannot be detected. It has the drawback that it cannot be detected. It is an object of the present invention to provide an object tracking method and apparatus that can eliminate the above-mentioned drawbacks of the prior art. Another object of the present invention is to provide a highly reliable object tracking method and apparatus for accurately detecting and tracking an object.

[0020]

In order to achieve the above object, according to one aspect of the present invention, an object which detects an object by comparing an input image acquired from an image pickup device having a zoom mechanism with a template image. In the detection method, a step of recording a first image within the visual field range of the image pickup device as the template image, a step of recording a changed magnification of the zoom mechanism, and a second step for detecting from the image pickup device. Image of the template image, the step of changing the size of any one of the template image or the second image based on the changed magnification of the zoom mechanism, the template image and the second image. The method is characterized by including a step of performing template matching with an image and detecting an object.

The step of recording the changed magnification of the zoom mechanism of the object detecting method of the present invention includes the step of recording the first focal length of the zoom mechanism when the template image is captured, and Recording the second focal length of the zoom mechanism when the second image is captured. The step of recording as the template image of the object detection method of the present invention includes the step of setting an area having a predetermined size so as to include the object, and the zoom mechanism so that the object is included in the area. And a step of controlling

The step of performing the template matching in the object detecting method of the present invention is a step of detecting the position of the partial image in which the degree of coincidence between the template image and the second image is maximum. Further, the object detection method of the present invention further comprises a step of manually operating the zoom mechanism and a step of displaying the input image on the screen of the display device, and the operator views the input image on the screen. While operating the zoom mechanism. Further, the zoom mechanism of the object detection method of the present invention calculates the elapsed time from the predetermined reference time after the start of zoom control. The object detection method of the present invention further updates a new template image as the template image based on the result of template matching.

The object detecting apparatus of the present invention is an object detecting apparatus for detecting an object by comparing an input image acquired from an image pickup apparatus having a zoom mechanism with a template image. Image as a template image recording unit, a magnification recording unit that records the changed magnification of the zoom mechanism, an acquisition unit that acquires a second image from the imaging device, and a zoom mechanism of the zoom mechanism. A control unit that changes the size of any one of the template image or the second image based on the changed magnification, and an image processing unit that performs template matching between the template image and the second image. It is equipped with.

The magnification recording section of the object detecting apparatus of the present invention includes a first focal length recording section for recording the first focal length of the zoom mechanism when the template image is captured, and the second focal length recording section. And a second focal length recording unit that records the second focal length of the zoom mechanism when the image is captured. Further, the template image of the object detection device of the present invention is an area having a predetermined size including the object, and the control unit controls the zoom mechanism so that the object is included in the area. Is. Further, the object detecting device of the present invention is characterized in that the zoom mechanism is at least one of a zoom lens and an electronic zoom. Further, the object detection device of the present invention comprises means for manually changing the zoom mechanism and means for displaying the input image on the screen of the display device, and the operator can see the input image on the screen while viewing the input image. This is to operate the zoom mechanism.

The object inspection / tracking method of the present invention detects an object by comparing an input image acquired from an image pickup device having a zoom mechanism with a template image, and controls the zoom mechanism to track the object. In the tracking method, recording a first image within the visual field range of the image pickup device as the template image, and acquiring a second image for detection from the image pickup device,
Recording the changed magnification of the zoom mechanism, changing the size of any one of the template image or the second image based on the changed magnification of the zoom mechanism, and the template The object tracking process includes the steps of performing template matching between the image and the second image, and controlling the moving mechanism of the imaging device according to the result of the template matching.

Further, the step of recording the changed magnification of the zoom mechanism of the object tracking method of the present invention includes the first step of the zoom mechanism when the template image is captured.
And a step of recording the second focal length of the zoom mechanism when the second image is captured. The step of recording as the template image of the object tracking method of the present invention includes the step of setting an area having a predetermined size so as to include the object, and the zooming so that the object is included in the area. Controlling the mechanism. Further, the step of performing the template matching of the object tracking method of the present invention is characterized by detecting the position of the partial image where the degree of coincidence between the template image and the second image is the maximum. is there. Further, the step of controlling the moving mechanism of the imaging device of the object tracking method of the present invention further comprises the step of controlling the moving mechanism such that the object is located near the center of the visual field range. Is.

Further, the zoom mechanism of the object tracking method of the present invention is characterized in that it is at least one of a zoom lens and an electronic zoom. Also,
The zoom mechanism of the object tracking method of the present invention is a zoom lens, and further comprises a step of manually changing the focal length of the zoom lens and a step of displaying the input image on a screen of a display device. The operator operates the zoom lens while looking at the input image on the screen. Further, the zoom mechanism of the object tracking method of the present invention is characterized in that the focal length of the lens calculates the elapsed time from a predetermined reference time after the start of zoom control. Further, the object tracking method of the present invention further updates a new template image as the template image based on the result of template matching.

The object tracking device of the present invention is an object tracking device for detecting an object by comparing an input image acquired from an imaging device having a zoom mechanism with a template image. Image as a template image, a template image recording unit for acquiring a second image from the imaging device, a magnification recording unit for recording the changed magnification of the zoom mechanism, and a zoom mechanism of the zoom mechanism. Either the template image or the second image based on the modified magnification
A first control unit that changes two sizes, an image processing unit that performs template matching between the template image and the second image, and the imaging for tracking the object according to the result of the template matching. A second control unit for controlling the moving mechanism of the apparatus is provided.

The magnification recording section of the object tracking device of the present invention includes a first focal length recording section for recording the first focal length of the zoom mechanism when the template image is captured, and the second focal length recording section. And a second focal length recording unit that records the second focal length of the zoom mechanism when the image is captured. Further, the template image of the object tracking device of the present invention is an area having a predetermined size including the object, and the first control unit is configured to include the first image in the area so that the first image is included in the area. It controls the zoom mechanism. Further, the zoom mechanism of the object tracking device of the present invention is characterized in that it is at least one of a zoom lens and an electronic zoom. Further, the zoom mechanism of the object tracking device of the present invention comprises means for manually changing the zoom mechanism and means for displaying the input image on the screen of the display device, and the operator inputs the input image on the screen. The user operates the zoom mechanism while watching.

[0030]

BEST MODE FOR CARRYING OUT THE INVENTION According to the object tracking method of the present invention, when the size of an intruding object shown in a conventional template image and the size of the intruding object to be detected in the input image are different, accurate position detection of the intruding object is performed. In order to solve the problem that the image cannot be captured, and the problem that the degree of matching between the template image and the input image decreases, the template image is enlarged or reduced based on the change in the focal length of the zoom lens. And the size of the intruding object in the input image are made equal, and template matching is executed.

That is, according to the present invention, the focal length of the zoom lens of the image pickup apparatus at the time of acquiring the input image obtained by extracting the registered template image and the image pickup at the time of acquiring the input image to be the template matching target. The magnification of the template image is calculated according to the ratio to the focal length of the zoom lens of the device, and the template image is enlarged or reduced to perform the template matching processing, thereby tracking the object within the visual field range of the imaging device. As a result, it is possible to stably track the object even while changing the zoom magnification of the zoom lens of the image pickup apparatus.

With reference to FIG. 6, the structure of the video monitoring apparatus according to the embodiment of the present invention will be described. FIG. 6 is a block diagram showing the configuration of an embodiment of the object tracking device of the present invention. E01 is an image pickup device (for example, a TV camera), E02 is a zoom lens, E03 is an electric swivel base (for example, a camera platform), E04 is an operation unit, and E04a is the first button attached to the operation unit E04. , E04b is a second button attached to the operation unit E04, and E05 is an intruding object monitoring device (image processing unit). In the intruder monitoring system E05, E05a is the image input I / F (I / F: Interface), E05b is the platform control I / F,
E05c is the lens control I / F, E05d is the operation input I / F, E05e
Is the image memory, E05f is the image output I / F, E05g is the alarm output I / F, and E05h is the CPU (CPU: Central Processing Uni
t), E05i is program memory, E05j is work memory, E05k is data bus, E06 is output monitor, E07 is warning light, E05 is image input I / F E05a, pan head control I / F E05b
, Lens control I / F E05c, operation input I / F E05d, image memory E05e, image output I / F E05f, alarm output I / F E05g
, CPU E05h, Program memory E05i, Work memory
These are E05j and the data bus E05k. Image memory E05e, work memory E05j and program memory
The E05i may be composed of a single memory.

In FIG. 6, the TV camera E01 is connected to the image input I / F E05a, the zoom lens E02 is connected to the lens control I / F E05c, and the camera platform E03 is connected to the camera platform control I / FE05b. , E04 is the operation input I / FE
The output monitor E06 is connected to 05d and the image output I / F E05f
Warning light E07 is connected to the alarm output I / F E05g. Also, image input I / F E05a, pan head control I / F
E05b, lens control I / F E05c, input I / F E05d, image memory E05e, image output I / F E05f, alarm output I / F E05g
, CPU E05h, program memory E05i, and work memory E05j are coupled to a data bus E05k.

In FIG. 6, a TV camera E01 is mounted on a camera platform E03 and has a zoom lens E02. The TV camera E01 captures an image of the surveillance target (surveillance field of view range), converts it into a video signal, and outputs it to the image input I / F E05a of the object tracking device E05. The image input I / F E05a converts the captured video signal into image data in a format that can be processed by the object tracking device E05 and outputs it to the image memory E05e via the data bus E05k, and the image memory E05e is input. Accumulate images.

The CPU E05h reads the image stored in the image memory E05e according to the program stored in the program memory E05i in advance, and reads it out from the work memory.
Performs analysis processing of the image read in E05j. CPU E05h
Depending on the analysis processing result, the data bus E05k controls the zoom lens E02 via the lens control I / F E05c and the camera platform E03 via the camera platform control I / F E05b.
Control the TV camera E01 to change the field of view, turn on the warning light E07 via the alarm output I / F E05g, and display on the monitor E06 via the image output I / F E05f, for example, the intruder detection result. Display an image.

The image memory E05e also includes a template image holding unit for storing the registered template image. The embodiment described below will be described assuming that the hardware configuration of the object tracking device described in FIG. 6 is used.

An embodiment of the present invention will be described with reference to FIG. FIG. 5 is a flow chart for explaining the processing process of one embodiment of the present invention. First, the initial visual field movement step
201, zoom lens E02 and camera platform E03
Move to the specified initial position, and set the TV camera E01 to capture the initial visual field range (initial monitoring visual field range).
The initial field of view is, for example, a field of view of the entrance and exit of a specific building that is to be monitored intensively.

The following steps 202 to 206 are
The process of detecting an intruding object by the difference method described in FIG. 1 and registering a template image is shown. Hereinafter, description will be given with reference to FIG. First, image input step 202
Then, from the camera E01, for example, width 320 pix, height 24
Get the input image S01 of 0 pix and 8 bit / pix. In the difference processing step 203, the difference in luminance value for each pixel between the input image S01 obtained in the image input step 202 and the reference background image S02 previously recorded in the image memory E05e is calculated, and the difference value is calculated. A difference image S03 that is the brightness value of each corresponding pixel is obtained.

Next, in the binarization processing step 204, threshold processing is performed on each pixel of the difference image S03 obtained in the difference processing step 203, and when the pixel value is less than the threshold value (for example, 20). Obtains a binarized image S04 with a pixel value of "0" and a pixel value of "255" when it is equal to or greater than the threshold value. Next, in the object presence determination step 205, the binarized image S0 obtained in the binarization processing step 204
From among the four, for example, a well-known labeling method is used to detect a cluster S09 of pixels having a pixel value of "255" to determine whether or not an object exists. When it is determined that the object exists, the process branches to the template image registration step 206. On the other hand, when it is determined that the object does not exist, the process branches to the image input step 202,
The next latest (currently photographed) input image is obtained and the processing of steps 203 to 205 is performed.

In the template image registration step 206, from the image input step 202 to the object presence determination step
The circumscribing rectangle S10 of the binarized object S09 detected by the processes up to 205 is calculated, the area corresponding to the circumscribing rectangle S10 is cut out from the input image S01, and this is registered in the image memory E05e as the initial template image S13. To do.

Steps 207 to 216 are a process of tracking the intruding object S07 detected by the difference method by template matching. In the image input step 207, as in the image input step 202,
From camera E01, for example, width 320 pix, height 240 pix
, 8 bit / pix latest (currently captured) input image (
Get M02). Next, the zoom control determination step 208
Then, if the size of the template image M03 of the intruding object being tracked is not within the predetermined range, it is determined that the zoom lens needs to be controlled.
Branch to 09. If it is within the predetermined range, the process branches to the zoom magnification calculation step.

Here, the predetermined range is set depending on how much the apparent size of the intruding object occupies the input image. For example, the apparent height of an intruding object (length in the vertical (y) direction: for example, H in FIG. 2 (b)) is the height of the entire input image (for example, H0 in FIG. 2 (b)). 1/3
To be within the range of 1/2. This is because the intruding object can be viewed vertically, and if the height of the intruding object is captured in the screen, the entire intruding object can be displayed in the image (screen). Therefore,
In this example, the apparent height of the intruding object is 80pix ~ 120
It is set within the range of pix.

Note that the predetermined range is not limited to such a range, and the distance from the position where the intruding object exists to the upper, lower, left, and right ends of the input screen is such that only the intruding object per processing frame of the input screen appears. The distance may be set to be equal to or greater than the moving distance of. This is the template matching process
Since it is the process of finding the image of the target intruding object from the input image, if the target intruding object goes out of the input image in the next processing frame in which the template is updated, the intruding object by template matching is theoretically used. This is because the detection cannot be performed. Therefore, for example, if the intruding object moves only in the horizontal direction, the apparent height of the intruding object should be set within the range of 1/1 of the total height of the input image (that is, Apparent height of intruding object: Overall height of input image ≈ 1: 1).

Next, in the zoom lens control step 209, in order to appropriately adjust the size of the intruding object appearing in the input image, if the size of the template image is smaller than the above predetermined range, the lens control I / Zoom in on the zoom lens E02 via F E05c (increasing the focal length). Also, if the size of the template image is larger than the specified range, the lens control I / F E05c
Zoom out the zoom lens E02 (shorten the focal length). By doing so, it is possible to control the zoom lens E02 so that the apparent size of the intruding object falls within a predetermined range, that is, the above-mentioned set value. Such zoom lens control can be realized, for example, by performing feedback control of the zoom lens E02 so that the above set value is achieved based on the apparent size of the intruding object.

Next, in the zoom magnification calculation step 210,
When the extracted image M01 of the registered template image M03 is acquired (when the template image registered in step 206 is input, that is, when the image is input in step 207, or the template image updated in step 215 described later is displayed. At the time of input, that is, step 207
Focal length of the zoom lens (at the time of image input)
And the ratio of the focal length fi of the zoom lens at the time of acquiring the input image M02 (at the time of image input in step 207 after one processing frame of template image input) to calculate the zoom magnification. Here, one processing frame refers to the template matching processing based on the input image in step 202 or step 207. The zoom factor is, for example, a focal length of ft = 12 mm and fi = 1
If it is 5 mm, then 15/12 = 1.25.

Further, template enlarging / reducing step
In 211, based on the zoom magnification obtained in the zoom magnification calculation step 210, for example, a known affine transformation (affin
The template image is enlarged (zoom magnification is greater than 1.0) or reduced (zoom magnification is less than 1.0) by the method of (e transform). By this process, for example, the size of the template image before scaling can be reduced
If the zoom magnification is 1.25 and the height is 0 and the height is 50, the size of the template image after enlargement / reduction is 30 × 1.25 horizontal.
= 38 pix, height 50 x 1.25 = 63 pix (both are rounded). Here, instead of enlarging / reducing the template image, enlarging / reducing the input image may be performed electronically or optically.

In the template matching step 212, the partial image having the highest degree of coincidence with the template image in the input image is searched by the template matching processing described with reference to FIGS. 2 (a) and 2 (b) above, and the position of the intruding object is determined. Detect if it exists Next, the matching degree determination step 213
Then, it is determined whether or not the maximum degree of coincidence obtained by the template matching in step 212 is a predetermined value (for example, 0.5 when the normalized correlation is applied) or more.
If the maximum degree of matching is less than the predetermined value, it is determined that an intruding object has not been detected (a partial image matching the template image was not found in the input image), and the process branches to step 201 of moving the initial visual field. Repeat steps 201 to 212. If the maximum degree of coincidence is greater than or equal to the predetermined value, it is determined that an intruding object is detected, and the process branches to the camera platform control step 214.

In the camera platform control step 214, the camera platform E03 is controlled based on the detected position of the intruding object. This control is, for example, when the position of the intruding object (the center position of the partial image detected in the input image) is located to the right of the image center, the camera platform E03 is panned to the right and left from the image center. When it is located at, the image is panned to the left, when it is located above the image center, it is tilted upward, and when it is located below the image center, it is tilted downward.

Further, in the template updating step 215, the template image updating unit T06 described with reference to FIG. 3 is used.
By performing the same process as above, the area corresponding to the partial image M07 having the highest matching degree with the template image is cut out from the input image M02, and this is extracted as a new template image M03.
As a result, the template image is updated. In the alarm / monitor display step 216, via the alarm output I / F E05g,
For example, an image of the target object is displayed on the monitor E06 or a warning lamp E07 is turned on to notify the presence of the intruding object.

A process of repeatedly executing the processes of steps 207 to 216 to perform template matching while enlarging or reducing the template image and tracking the intruding object will be described with reference to FIG. FIG. 4 is a diagram for explaining the operation of an embodiment of the object tracking process of the present invention. As shown in FIG. 4, first, time T1
The intruding object within the visual field range of the imaging device is tracked using the extracted image Z01a and the template image Z01d based on the input image obtained in -1. Image Z01a, Z02a, Z03a each time T1-1, respectively of the extracted image obtained based on the input image at T1, T1 + 1. Also images Z01d, Z02
d and Z03d are template images obtained based on the respective input images at times T1-1, T1, and T1 + 1, and images Z01c, Z02c, and Z03c are times T1 and T03, respectively.
Input images of T1 + 1 and T1 + 2.

The template image Z01d based on the input image acquired at time T1-1 is processed in the enlarging / reducing processing unit (ZM) Z04 (steps 210 and 211) as follows. That is, the zoom magnification is determined by the ratio of the focal length of the zoom lens when the input image that is the source of the extracted image Z01a is acquired from the TV camera and the focal length of the zoom lens when the input image Z01c is acquired from the TV camera. It is calculated (step 210), the template image Z01d is enlarged or reduced using the calculated zoom magnification, and the extracted image Z01b after the enlargement or reduction processing and the template image Z01e are obtained.

Here, the operation of increasing the focal length of the zoom lens will be described with reference to FIGS. 8 and 9. FIG. 8 is a diagram showing an example of an increase in the focal length of the zoom lens during a zoom-up operation according to the execution of the present invention. FIG. 8 shows how the focal length of the zoom lens increases with respect to the zoom lens control time t during zoom-up control. FIG. 9 is a diagram showing an example of an increase in the focal length of the zoom lens during the zoom-out operation according to the execution of the present invention. FIG. 9 shows how the focal length of the zoom lens decreases with respect to the zoom lens control time t during zoom-out control.

In FIG. 8, for example, zoom-up control is started at time t = 0. In this case, the delay of time td (for example, 0.1 se
c) occurs. After that, for time ta (for example, 0.5 sec
) If the zoom lens control is accelerated, and then the focal length increases by a certain increment Δfmax, and the zoom lens control is stopped at time ts, there is a delay of time td and deceleration takes place during time tb (for example, 0.5 sec). Acceleration / deceleration characteristics 701
Indicates. Here, the increase of the actual focal length is the acceleration / deceleration characteristic.
This is the area of the trapezoid of 701, and the focal length is {(ts − ta + ts + tb) × △ by the zoom-up operation for ts seconds.
fmax} / 2 (ts ≧ ta), {(fmax / ta + △ fmax /
ta) × ts 2} / 2 (ts <ta).

Now, the focal length before the zoom-up operation is 10.
If it is 0 mm, then Δfmax = 2.0 mm / sec and t
= 5.0 sec zoom-up operation brings the focal length to 2
It is increased to 0.0mm, and the zoom ratio of the input image before and after zoom-up operation is 20.0 / 10.0 = 2.0. Similarly, in the case of the zoom-out control of FIG. 8, the focal length is {(ts − ta + ts + tb) × Δ by the zoom-out operation for ts seconds.
fmin} / 2 = 10.0mm (ts ≥ ta), {(△ fmax / t
a + Δfmax / ta) × ts 2} / 2 (ts <ta).

In the enlargement / reduction processing unit Z04, the extracted image Z after enlargement or reduction according to this zoom magnification is executed.
Get 01b and template image Z01e. Next, the template matching processing unit (TM) Z05 (step 212
), Enlarged or reduced template image Z0
By performing template matching on the input image Z01c using 1e, a partial image Z01g of the input image Z01c can be obtained as a matching image. Therefore, the input image Z
Intruders in 01c are determined to be present in Z01f. The enlarging / reducing processing unit Z04, the template matching processing unit (TM) Z05, and the template image updating unit Z06 are respectively the enlarging / reducing process, the template matching process, and the template image updating process stored in advance in the program memory E05i. Is a program that does. Next, template image update section Z06 (step 215)
Then, the extraction image and the template image are updated based on the detected partial image Z01g of the input image Z01c, that is, a new extraction image Z02a and a new template image Z02d are obtained.
After that, the obtained partial image Z01g is used as a new template image Z02d, and enlargement / reduction processing is performed using this, and the template image Z02e and the time after the enlargement / reduction processing are performed.
Perform template matching with the new input image at t1 + 1. Then, a new extracted image Z023a and a template image Z03d are obtained based on the detected partial image Z02g of the input image Z02c. By performing similar processing in sequence, the intruding object Z02f at time t1 + 1 and the intruding object Z03f at time t1 + 2 while controlling the zoom lens E02.
Can be obtained, and thus an intruding object in the input image can be tracked.

As described above, according to the embodiment of the present invention,
Zoom control judgment step 2 based on the size of the intruding object
The zoom lens is controlled by 08 and the zoom lens control step 209, the zoom magnification thereof is calculated by the zoom magnification calculation step 210, and the template enlargement / reduction step 210 enlarges or reduces the template image based on the zoom magnification. Therefore, even if the zoom lens is operated, the position of the template image does not deviate from the target intruding object, and further, the intruding object can be accurately tracked without lowering the matching degree at the time of template matching. In the above embodiment, the image that has been subjected to zoom lens control in steps 208 and 209 is input in step 202 or step 207. Therefore, these steps 208 and 209 are, for example, steps 214 and 21 as in the modification shown in FIG.
May be run between 5. This is the second and the third which will be described later.
The same applies to the embodiment.

The second embodiment of the present invention will be described with reference to FIG. FIG. 10 is a flow chart for explaining the processing process of the second embodiment of the present invention. FIG. 10 shows the zoom control step 2 in the process described in FIG.
The zoom operation step 208 ′ is executed instead of 08, and the zoom lens control step 209 ′ is executed instead of the zoom lens control step 209. Therefore, other steps 201 to 20
7 and steps 210 to 216 are the same as those in the first embodiment, the description of those processes will be omitted.

When the processing from step 201 to step 207 is performed as described in FIG. 5, the process proceeds to zoom operation determination step 208 '. Zoom operation determination step 208 ′
Now, let's check if the operator has operated the operation unit E04.
Judgment is based on the operation input from the operation unit E04, the output via the I / F E05d. When the operator operates the operation unit E04, the process branches to the zoom lens control step 209 '. If the operator is not operating the operation unit, the process branches to zoom magnification calculation step 210.

In the zoom lens control step 209 ', when the operator performs a zoom-up operation using the operation unit E04, the focal length of the zoom lens E02 is increased by a predetermined value (for example, 1 mm). Here, the zoom-up operation is executed by, for example, the operator pressing the first button E04a of the operation unit E04. As an example, the operation unit
The focal length of the zoom lens E02 may be increased by a predetermined value each time the operator presses the first button E04a of E04 once. In addition, when the operator performs zoom-out operation using the operation unit E04, the zoom lens E02
Decrease the focal length of the lens by a predetermined value (eg 1 mm).
Here, the zoom-out operation is executed by, for example, the operator pressing the second button E04b of the operation unit E04. As an example, the focal length of the zoom lens E02 may be decreased by a predetermined value each time the operator presses the second button E04b of the operation unit E04 once.

When the operator performs the zoom-up operation and the zoom-out operation, in the above embodiment, the operation unit
When the first button and the second button of E04 are pressed at the same time, for example, the focal length of the zoom lens E02 may not be changed. By doing so, the operator can operate the zoom lens E02. Alternatively, when the operator simultaneously performs the zoom-up operation and the zoom-out operation, one of the zoom-up operation and the zoom-out operation may be executed. After that, the processing from step 210 onward is executed. By doing so, while operating the zoom lens E02, at time t1 + 1, the intruding object Z02f and time t1 +
In 2 the intruding object Z03f and the intruding object in the input image can be traced.

As described above, according to the second embodiment of the present invention, the zoom lens is controlled by the zoom operation determining step 208 'and the zoom lens controlling step 209' based on the zoom operation of the operator, and the zoom magnification is changed. It is calculated in the zoom magnification calculation step 210, and the template image is enlarged or reduced in the template enlargement / reduction step 211 based on the zoom magnification. Therefore,
Operate the zoom lens while looking at the input screen displayed on the output monitor E06 so that the position of the template image does not deviate from the target intruding object, and further reduce the degree of matching during template matching. You can accurately track the intruding object so that there is no.

A third embodiment of the present invention will be described with reference to FIG. FIG. 11 is a flowchart illustrating a processing process according to an embodiment of the present invention. The embodiment of FIG.
In the second embodiment of the present invention described with reference to FIG. 10, the zoom lens is automatically controlled based on the size of the intruding object. That is, when the operator does not perform the zoom operation, the zoom lens is automatically controlled so that the apparent size of the intruder can be displayed in a predetermined size. FIG. 11 is obtained by adding a zoom control determination step 301 and a zoom lens control step 302 to the second embodiment shown in FIG. The processing from step 201 to step 207, step 208 'to step 209', and step 210 to step 215 are the same as those in the second embodiment described above, and therefore the description of these processing is omitted.

In FIG. 11, zoom control determination step
301 is executed when the operator has not performed a zoom operation in the zoom operation determination step 208 '. In zoom control determination step 301, if the size of the template image M03 of the intruding object being tracked is not within the predetermined range, it is determined that the zoom lens needs to be controlled, and the zoom lens control step 302 Branch to. On the other hand, when the size of the template image M03 is within the above-mentioned predetermined range, the process proceeds to the zoom magnification calculation step 210.

According to the above-described third embodiment of the present invention, the zoom operation determining step 20 is performed based on the operation of the operator.
8 ', zoom lens control step 209' controls the zoom lens, if the operator is not operating,
The zoom lens is controlled by the zoom control determination step 301 and the zoom lens control step 302, the zoom magnification thereof is calculated by the zoom magnification calculation step 210, and the template enlargement / reduction step 211 enlarges or reduces the template image based on the zoom magnification. . Therefore, even if the zoom lens is operated, the position of the template image does not deviate from the target intruding object, and further, the intruding object can be accurately tracked without lowering the matching degree at the time of template matching.

[0065]

According to the conventional technique using the template matching, when the zoom lens is controlled, the size of the template image and the size of the intruding target object are different, and there is a guarantee that the matching is accurate. There was a problem that it was not possible to do stable tracking.
However, as described above, according to the present invention, the zoom image is magnified by calculating the zoom magnification based on the ratio of the focal length at the time of acquiring the template image and the focal length at the time of acquiring the input image. The target intruding object can be accurately tracked even while controlling or operating.
Therefore, according to the present invention, it is possible to remove the conventional limitation of the zoom lens control of the image pickup apparatus, to stably track the intruding object, and to broaden the range of application of the image monitoring apparatus.

[Brief description of drawings]

FIG. 1 is a diagram for explaining an operation of an object detection process by a conventional difference method.

FIG. 2 is a diagram for explaining an operation of object detection processing by a conventional template matching method.

FIG. 3 is a diagram illustrating an operation of object tracking processing that sequentially performs a conventional template matching method.

FIG. 4 is a diagram for explaining the operation of an embodiment of the object tracking process of the present invention.

FIG. 5 is a flowchart illustrating a processing process according to an embodiment of the present invention.

FIG. 6 is a block diagram showing the configuration of an embodiment of the object tracking device of the present invention.

7 is a flowchart showing a modification of the processing process shown in FIG.

FIG. 8 is a diagram illustrating an example of an operation of increasing the focal length of the zoom lens according to the embodiment of the present invention.

FIG. 9 is a diagram illustrating an example of an operation of increasing the focal length of the zoom lens according to the embodiment of the present invention.

FIG. 10 is a flowchart illustrating a processing process according to a second embodiment of the present invention.

FIG. 11 is a flowchart illustrating a processing process according to a third embodiment of the present invention.

[Explanation of symbols]

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 5/232 H04N 5/232 CF term (reference) 5B057 BA17 CA08 CA12 CA16 CE12 CH18 DA06 DB02 DB09 DC32 5C022 AA01 AB63 AB66 AC27 AC74 5C054 CG02 CG06 EA01 FC12 HA18 HA31 5L096 AA06 BA02 CA04 EA43 FA02 GA13 HA03 JA09

Claims (8)

[Claims] 1. An object detection method for detecting an object by comparing an input image acquired from an image pickup apparatus having a zoom mechanism with a template image, wherein a first image within a visual field range of the image pickup apparatus is used as the template image. Based on the changed magnification of the zoom mechanism, a step of recording, a step of recording the changed magnification of the zoom mechanism, a step of acquiring a second image to be detected from the imaging device, An object characterized by detecting an object, comprising the steps of changing the size of one of the template image or the second image and the step of performing template matching between the template image and the second image. Detection method. 2. The object detection method according to claim 1, wherein
The step of recording the changed magnification of the zoom mechanism includes the step of recording the first focal length of the zoom mechanism when the template image is captured, and the step of recording the second focal point when the second image is captured. Recording a second focal length of the zoom mechanism. 3. The object detection method according to claim 1, wherein the step of recording as the template image includes the step of setting an area having a predetermined size so as to include the object, and the object in the area. And a step of controlling the zoom mechanism so as to be included therein. 4. The object detection method according to claim 1, wherein
The object detecting method, wherein the step of performing template matching is a step of detecting a position of a partial image in which the degree of coincidence between the template image and the second image is maximum. 5. The object detection method according to claim 1, wherein
A step of manually operating the zoom mechanism, and a step of displaying the input image on the screen of the display device,
An object detecting method, wherein an operator operates the zoom mechanism while looking at the input image on the screen. 6. An object detection device for detecting an object by comparing an input image acquired from an image pickup device having a zoom mechanism with a template image, wherein a first image within a visual field range of the image pickup device is used as the template image. Based on the template image recording unit for recording, a magnification recording unit for recording the changed magnification of the zoom mechanism, an acquisition unit for acquiring a second image from the imaging device, and a changed magnification of the zoom mechanism. A control unit for changing the size of any one of the template image or the second image, and an image processing unit for performing template matching between the template image and the second image. Object detection device. 7. An object tracking method for detecting an object by comparing an input image acquired from an imaging device having a zoom mechanism with a template image, and tracking the object by controlling the zoom mechanism, wherein a field of view of the imaging device is used. Recording a first image within the range as the template image; acquiring a second image for detection from the image pickup device; recording a changed magnification of the zoom mechanism; Changing the size of any one of the template image or the second image based on the changed magnification of the zoom mechanism, and performing template matching between the template image and the second image. Controlling the moving mechanism of the imaging device according to the result of the template matching. An object tracking method comprising: tracking the object. 8. An object tracking device for detecting an object by comparing an input image acquired from an image pickup device having a zoom mechanism with a template image, wherein a first image within a visual field range of the image pickup device is used as the template image. Based on the template image recording unit for recording, an acquisition unit for acquiring the second image from the imaging device, a magnification recording unit for recording the changed magnification of the zoom mechanism, and the changed magnification of the zoom mechanism. A first control unit that changes the size of one of the template image and the second image; an image processing unit that performs template matching between the template image and the second image; A moving mechanism of the imaging device for tracking the object according to the result of
An object tracking device, comprising: