JP4512690B2 - Monitoring system and method by image processing - Google Patents

Description

  The present invention relates to a monitoring system and method based on image processing in which monitoring is performed by a camera mounted on a moving body and a moving object is detected from captured images.

  Conventional monitoring systems based on image processing compare the two images of the input image and the reference image, detect different areas, and if different areas exceed a predetermined area, there are people, vehicles, and other moving objects As a result, a warning or the like was issued (see Patent Document 1). However, since the monitoring system according to the prior art uses a fixed camera, there is a problem that it cannot cover a wide range.

In order to cover a wide monitoring range, patrol by a moving body is required. However, with a camera mounted on a moving body that continues to move, the background looks different from frame to frame, so it is difficult to detect moving objects.
Further, since the position of the fixed camera is known in advance by the intruder, there is a high possibility that the fixed camera will be destroyed when necessary.
JP-A-8-320934

  In order to solve such problems and cover a wide monitoring range, the present invention performs monitoring with a camera mounted on a moving body that continues to move, detects a moving object from the captured image, and moves the moving body. It is intended to enable patrol by.

  The monitoring system using image processing according to the present invention uses a plurality of cameras attached to a moving body at intervals, and adjusts the direction of the camera and the timing of the shutter, so that the rear camera is positioned at the point where the front camera releases the shutter. Is configured to release the shutter when the camera comes, and equivalently, means for obtaining a plurality of images with the same time difference from the same point, an image alignment unit for aligning these images, and a position A difference calculation unit that takes a difference between the combined images and a motion detection unit that detects a moving object in the image based on the difference output from the difference calculation unit.

  In addition, the monitoring method by image processing of the present invention uses a plurality of cameras attached to a moving body at intervals, and by adjusting the camera direction and shutter timing, at the point where the front camera releases the shutter, The camera was configured to release the shutter when the rear camera arrived. Equivalently, multiple images with the same time difference from the same point were obtained, and the multiple images were aligned and aligned. The difference between the images is taken, and a moving object in the image is detected based on the difference.

  According to the present invention, it is possible to monitor a wide area where a fixed camera cannot be installed by patrol by a moving body loaded with cameras. In addition, it is inevitable that the fixed camera is actually unusable due to destruction by an intruder, but patrol by a moving body is a means to counter this.

  FIG. 1 is a diagram illustrating a state in which a plurality of cameras (illustrated as two) are attached to a moving body (vehicle). As shown in the figure, using multiple cameras attached to a moving body, adjusting the camera direction and shutter timing so that the shutter is released when the rear camera arrives at the point where the front camera releases the shutter. it can. Thereby, it is possible to obtain a plurality of images equivalently having the same time difference from the same point. This detects moving objects on the ground.

  FIG. 2 is a diagram illustrating a monitoring system using image processing that embodies the present invention. As shown in FIG. 1, with two cameras mounted in parallel, the delay of the shutter timing of the second camera is changed according to the moving body speed, equivalently the time difference from the same position point Get an image. In this case, data is taken only when the moving body is traveling straight ahead. The reason is that the direction of the moving body changes on the curve, so the direction of the camera changes even if the second camera comes to the same position as the first. Although the mechanism is complicated, this problem can be avoided if the direction of the camera is linked to the axle direction.

The delay of the second shutter timing is determined by the following equation. L: Interval between two cameras (m), V: Speed of moving body (km / h), T: Delay of shutter timing (msec)
T = (3600 × L) / V

In this way, the image capturing timing control unit obtains two time difference images in which the shutter timing is adjusted according to the speed of the moving body whose speed has been detected. In the image alignment unit, in order not to depend on the difference in brightness by individual cameras,
1) Monochrome the original image (for the two original images, find the average of the respective brightness and adjust the brightness based on the average to keep the brightness the same).

2) Thinning a monochrome image. For this purpose, for example, a Laplacian operator can be used. Laplacian operator: As shown in FIG. 3, the output pixel value (B) for the target pixel (A4) at the center of the input pixel is set to the value (A4) of the center pixel and the pixel values (A0) in the neighborhood of the input image. , A1, A2, A3, A5, A6, A7, A8), a total of nine pixel values is obtained by a Laplacian operator (formula in the figure). This is the sum of the second derivative (difference of difference) for each of the four diagonal directions when considering the gray value of the image as a function for two dimensions in length and breadth (that is, f (x, y)). Equivalent to. In addition, it does not depend on the density value itself, and takes a large value when the density change is large.

3) Perform alignment by correlation. Originally, the two images should be at exactly the same angle if the cameras are exactly at the same position, but a slight shift occurs due to the shaking of the moving object, so about 3% of this image (this value) Is determined by the accuracy with which the two cameras are located at the same position.) Within the range, the position is displaced vertically and horizontally to obtain the maximum value by correlation. At this time, the correlation is obtained by using only the image of the portion where the relative speed with respect to the ground surface is zero, such as the separation band, the guardrail, the white line, and the neighboring building, among the two frames taken in succession. A portion surrounded by a thick line in FIG. 4 is a portion with little designated movement for alignment when the camera is directed obliquely forward. This is possible when moving in a known location, but when the background is not known in advance, a long line segment as shown in FIG. 5 is used for correlation.

4) Let the image after alignment be P1 and P2.
Next, the difference calculation unit (FIG. 2) calculates a difference between two continuous monochrome images (P1, P2).
The moving part detection unit detects a moving part based on a threshold value from a difference image between two continuous monochrome images (P1, P2).

FIG. 6 is a diagram illustrating detection of a moving part.
An image obtained by taking the absolute value of the difference between P1 and P2 is represented by | P1-P2 |
When the binarization operator is B (P, T): P is a monochrome image, T is a threshold,
B (| P1-P2 |, T)
Thus, a change in the contour of the moving direction of the object moving with respect to the ground surface is detected. (In actual data, the expansion and contraction operations are performed by the expansion operator and the contraction operator in order to remove noise.)

  FIG. 7 is a diagram illustrating an image of each stage in the case where the timing is taken in accordance with the present invention. FIG. 8 shows a comparative example taken simultaneously by two cameras for comparison with the present invention. 7 and 8, the middle left figure: the image of the front camera, the lower left figure: the front camera image made into a line drawing, the middle right figure: the image from the rear camera, the lower right figure: the camera image after the line drawing, the upper right figure. : A moving object detected from the difference image, upper left figure: display of the speedometer of the moving body.

In FIG. 7, the moving object is clearly detected, whereas in the comparative example of FIG. 8, the moving object is not successfully extracted and detected from the background from the difference image.
As described above, the case where two cameras are used has been described as an example. However, if three or more cameras are used, it is possible to accurately extract a moving region. Specifically, as shown in FIG. 9, three consecutive monochrome images obtained from three cameras are represented by P1, P2, P3,
Signed difference images are represented by (P2-P1), (P2-P3),
If the logical sum image is P = (B (P2−P1), T) OR (B (P2−P3), T),
The moving portion detected by P is detected by overlapping the moving object image in P2 (FIG. 9). (In the detection of the moving part from the two images shown in FIG. 6, the moving object image in P2 only partially overlaps.)

Expand (P, n): P is a binary image, n is the number of expansions, (expansion is the expansion of the black portion of the binary image by one pixel in eight directions)
Shrink operator, Shrink (P, n): P is a binary image, n is the number of times of contraction (contrast is to expand the white portion of the binary image by 8 pixels by 8 directions)
By performing Expand (Shrink (Expand (P, n), 2n), n), the noise of the detected motion part and the noise of the background part can be eliminated (FIG. 9).

Further, when a camera with a fixed number of frames per second is used, a time difference image from the same position point can be obtained equivalently by changing the interval between the two cameras according to the speed. The camera interval is
L: Interval between two cameras (m), V: Speed of moving body (km / h), F: Number of frames per second (1 / sec)
It is determined by L = V / (3.6 · F).
Also in this case, if three or more cameras are used, the outline of the moving object can be extracted with high accuracy.

  FIG. 10 is a diagram illustrating installation of a camera different from that in FIG. As shown in the figure, a camera that rotates horizontally in front of and behind the moving body (vehicle) is installed, and the entire circumference of 360 degrees is photographed at high speed, and the rear camera follows the position and angle photographed by the front camera. In addition, by adjusting the rotation delay and the shutter delay, it is possible to detect a moving object by positioning / difference of the entire surrounding image. The shutter delay is synchronized with the rotation delay.

The phase shift of the rear camera is as follows: L: Distance between two cameras (m), V: Speed of moving body (km / h), ω: Angular speed of camera rotation (degrees / sec), D: Rear camera If the phase of the rotation angle (degrees),
D = (ω · 3.6 · L) / V

The above-mentioned horizontal rotation of the camera can be replaced with an all-around photographing using a rotating prism (or rotating mirror) and an image rotator as shown in FIG. In this case, only the way of installing the camera is different, and the control of the rotation angle is the same as the horizontal rotation method described above. Furthermore, it can be replaced with omnidirectional imaging using a conical mirror or a spherical mirror (or hyperboloidal mirror) as shown in FIG. In this case, since all directions are photographed at one time, it is not necessary to control the angle, and only the shutter timing is required.
Further, when an infrared camera is used as the camera, it becomes easy to detect a car, a person or an animal whose engine has just stopped.

  FIG. 13 is a diagram illustrating the attachment of still another camera to the moving body. As shown in the figure, the 1F, 2F,. . . . .・ The roof, the sky, the roof on the opposite side. . . . .・ Images that can be monitored from the first floor of the building to the rooftop can be obtained by helical scanning and high-speed shooting like 2F, 1F, and the ground. By installing this camera in front of and behind the moving body, it is possible to detect a moving object in a building street. Similar to the example described above with reference to FIG. 10, the rear camera adjusts the rotation delay and the shutter delay so that the position and angle taken by the front camera are traced, and the position and difference of the entire surrounding image are adjusted. It can detect moving objects. The shutter delay is synchronized with the rotation delay.

The phase shift of the rear camera is as follows: L: Distance between two cameras (m), V: Speed of moving body (km / h), ω: Angular speed of camera rotation (degrees / sec), D: Rear camera If the phase of the rotation angle (degrees),
D = (ω · 3.6 · L) / V

The method shown in the figure is effective for monitoring city patrols and guards for people with armored vehicles in the guerrilla area, which must monitor the left and right sides and the rooftop. As in the example of FIG. 10, a rotating prism or a conical mirror can be used. In addition, when an infrared camera is used, it becomes easier to detect a car or a person / animal whose engine has just stopped.
Further, instead of a camera capable of high-speed shooting, a disk in which a plurality of cameras are attached as shown in FIG. 14 can be used. The same applies to the example shown in FIG.

It is a figure which illustrates installation of a plurality of cameras to a moving body. FIG. 2 is a diagram illustrating a monitoring system using image processing that embodies the present invention. FIG. 3 is a diagram showing a Laplacian operator used for thinning. It is a figure which shows the data of 2 sheets from the same position where imaging | photography time differs. The area within the bold line is the position to be aligned by density correlation. It is a figure which shows extracting a long line segment from two data from the same position where imaging | photography time differs, and performing alignment by it. It is a figure which shows detecting a moving part from the difference of two data from the same position where imaging | photography time differs. It is a figure which shows the image taken when the back camera moved to the position and the image of the front camera based on this invention. Middle left figure: Front camera image, Lower left figure: Front camera image made into line drawing, Right middle figure: Rear camera image, Lower right figure: Camera image made after line drawing, Upper right figure: Moving object detected from difference image. It is a figure which shows the comparative example of the image image | photographed with the front camera and the back camera at the same time. Middle left figure: Front camera image, Lower left figure: Front camera image made into line drawing, Right middle figure: Rear camera image, Lower right figure: Camera image after line drawing, Upper right figure: Moving object is detected well from the difference image Not. It is a figure which shows detecting a motion part from the difference of the 3 sheets of data from the same position where imaging | photography time differs. It is a figure which illustrates the camera which rotates horizontally before and behind a moving body. It is a figure which illustrates all the circumference photography by a rotating prism. It is a figure which illustrates all the circumference photography by a conical mirror and a spherical mirror. It is a figure which shows the example of installation of the camera which rotates like a windmill which made the rotating shaft the advancing direction. The camera scans the left and right cityscapes in a helical manner and shoots at high speed. It is a figure which shows the example which attaches a some camera to a turntable instead of the camera which can be image | photographed at high speed. In this case, it can be taken at a frame rate three times that of each camera within a certain time.

Claims (7)

Using multiple cameras attached to a moving object at intervals, adjusting the camera direction and shutter timing so that the shutter is released when the rear camera arrives at the point where the front camera releases the shutter Configured, and equivalently, means for obtaining a plurality of images with a time difference of the same angle from the same point;
An image alignment unit for aligning these multiple images;
A difference calculation unit that takes a difference between the images that have been aligned;
Based on the difference output from the difference calculation unit, a moving object detection unit that detects a moving object in the image;
A monitoring system using image processing.   2. The structure according to claim 1, wherein when a camera with a fixed number of frames per second is used, a time difference image from the same position point is equivalently obtained by changing an interval between the plurality of cameras according to the moving body speed. Monitoring system using image processing.   The plurality of cameras are installed so as to rotate horizontally, perform high-speed shooting of the entire periphery, and adjust the rotation delay and shutter delay so that the rear camera traces the position and angle captured by the front camera. The monitoring system by image processing according to claim 1. The horizontal rotation of the plurality of cameras, the monitoring system by the image processing according to claim 3 carried out by virtually rotate the prisms.   The plurality of cameras are installed with the rotation axis as the direction of travel, perform high-speed shooting of the entire circumference, and the rear camera follows the position and angle taken by the front camera, so that the rotation delay and shutter delay The monitoring system by image processing according to claim 1, wherein the adjustment is performed.   The plurality of cameras are each configured by attaching a plurality of cameras to a disk, and a plurality of cameras attached to each disk are used to perform high-speed shooting of the surroundings, and the subsequent camera sets the position and angle captured by the front camera. The monitoring system by image processing according to claim 1, wherein the rotation delay and the shutter delay are adjusted so as to trace. Using multiple cameras attached to a moving object at intervals, adjusting the camera direction and shutter timing so that the shutter is released when the rear camera arrives at the point where the front camera releases the shutter Configured and equivalently obtained multiple images with the same angle time difference from the same point,
Align these multiple images,
Take the difference of the aligned images,
Based on this difference, detect moving objects in the image,
A monitoring method using image processing.