JP5962297B2 - Camera field of view detection device - Google Patents

Description

  The present invention relates to a camera field-of-view variation detection device, and more particularly to a technique for detecting a variation in field of view of a fixed-point camera used in a fixed-point monitoring system.

  Fixed point cameras are widely used in security cameras and street-mounted cameras. The role of this fixed point camera is to continuously photograph the same field of view for a certain period at the same place, and is used for the purpose of continuously monitoring a specific location. Therefore, a fixed-point camera installed at a predetermined location must originally maintain its field of view for a certain period.

  However, if an unexpected situation occurs, the orientation of the fixed point camera changes, and as a result, the shooting field of view changes. For example, when an animal such as a bird, a dog, or a cat is placed on the camera, or when a child collides with the camera, the direction of the camera changes. In addition, the orientation of the camera changes due to natural phenomena such as strong winds, heavy rains, and earthquakes, and the influence of vibration of passing vehicles. Alternatively, in the case of a camera having a function of adjusting the direction by remote operation, the direction may change due to an operator's erroneous operation.

  Since the fixed point camera is installed for the purpose of continuously photographing within a specific photographing field for a certain period, it is not preferable that the photographing field fluctuates due to an unexpected situation as in the above example. For example, when a fixed point camera is installed as a monitoring camera for monitoring a specific place in real time, if the field of view is deviated from the original monitoring place, a normal function as the monitoring camera cannot be achieved. In addition, when the processing for automatically analyzing the traffic volume is performed based on the road surface image captured by the fixed point camera, the correct analysis cannot be performed if the photographing field of view deviates from the road surface.

  In general, as a technique for performing camera shake correction, a method using various sensors has been proposed as a method for detecting a change in the field of view of a camera. For example, the following Patent Document 1 discloses a technique for detecting camera shake using an angular velocity sensor built in a camera, and Patent Document 2 discloses a method for detecting camera shake using an acceleration sensor instead of the angular velocity sensor. ing. Patent Document 3 discloses a technique for detecting a change in photographing field of view by incorporating an angular velocity sensor into a surveillance camera. On the other hand, Patent Document 4 discloses a technique for detecting a scene change by analyzing a captured image obtained in units of frames.

JP-A-6-148730 JP 2006-098533 A JP 2000-285328 A Japanese Patent No. 4698754

  As described above, a method using an angular velocity sensor, an acceleration sensor, or the like is generally used as a method for detecting a change in the field of view of the camera. However, the method using these sensors is suitable for being incorporated in a digital camera in order to perform camera shake correction, but is not necessarily suitable for being used for detection of photographing field variation of a fixed point camera.

  In general, in a fixed point monitoring system, a captured image of a fixed point camera installed in a remote place is captured and processed in units of frames. Therefore, if a method of incorporating a sensor in the camera is adopted, the detection signal of the sensor is sent to the system together with the captured image. It is necessary to secure a transmission path for transmission up to. In the fixed point monitoring system, the captured image is not necessarily displayed on the monitor in real time, and there are many cases where the recorded captured image is analyzed by performing various image processing later. In such a case, it is necessary to record the variation information detected by the sensor in some form in the captured image.

  On the other hand, there has been proposed a technique for detecting that some variation has occurred in the photographing field of view by analyzing the photographed image itself without using a sensor. For example, the scene change detection method disclosed in Patent Document 4 described above can detect that a video scene has changed by analyzing frame images that are continuously given in time series. However, this method is originally intended to be used for cutting a CM portion from a recorded TV program, and it is highly likely that a false detection will occur if it is used for detecting a shooting field variation of a fixed point camera. .

  SUMMARY OF THE INVENTION An object of the present invention is to provide a photographing visual field fluctuation detection device capable of accurately detecting a change in camera direction by analyzing a photographed image itself of the camera.

(1) According to a first aspect of the present invention, there is provided a camera field-of-view variation detecting device for detecting a camera field-of-view variation.
An image input unit for inputting an image captured by the camera as a frame-unit original image continuously given in time series;
Foreground indicating an extracted foreground region by obtaining an image having an average characteristic of a plurality of original images input in the past as a background image, extracting a foreground region different from the background image from the newly input original image A foreground region extraction unit for generating an image;
A foreground image storage unit that stores the foreground image generated for each original image;
A difference area extraction unit that extracts a difference area for two sets of foreground images stored in the foreground image storage unit and generates a difference image indicating the extracted difference area;
A block dividing unit that divides each generated difference image into a plurality of blocks;
For each individual block obtained, an occupancy ratio calculation unit that calculates the occupancy ratio that each difference area occupies,
A block counting unit that recognizes a block whose occupation ratio is equal to or greater than the first threshold as a sudden change block, and counts the number of sudden change blocks for each difference image;
A variation determination unit that determines that a variation has occurred in the field of view when a difference image is obtained in which the number of suddenly changing blocks is equal to or greater than a second threshold;
A determination result notification unit for notifying the determination result by the variation determination unit;
Is provided.

(2) According to a second aspect of the present invention, in the photographing field fluctuation detection device for a camera according to the first aspect described above,
The foreground area extraction unit
An original image storage unit for sequentially storing the input original images;
Based on a plurality of original images input in the past, an average image creation unit that sequentially creates an average image having average characteristics of these original images;
An average image storage unit for sequentially storing the created average images;
The average image stored in the average image storage unit is used as a background image, compared with the original image stored in the original image storage unit, and the different parts are recognized as the foreground region. An image comparison unit that creates a foreground image as a binary image that distinguishes the area;
Is provided.

(3) According to a third aspect of the present invention, in the photographing field fluctuation detection device for a camera according to the second aspect described above,
The image input unit inputs a color-unit color original image continuously given in time series as pixel aggregate data having pixel values of the three primary colors,
When the average image creation unit receives the i-th original image P (i) (i = 1, 2,...), The past original image including the original image P (i) A weighted average value of pixel values for each color of pixels at corresponding positions is calculated, and an i-th average image A (i) composed of a collection of pixels having the average value is created as a background image.
When the (i + 1) -th original image P (i + 1) is input, the image comparison unit compares the original image P (i + 1) with the i-th average image A (i), The (i + 1) th foreground image F (i + 1) is created.

(4) According to a fourth aspect of the present invention, in the photographing field fluctuation detection device for a camera according to the third aspect described above,
The average image creation unit
When the first original image P (1) is input, the original image P (1) is stored in the average image storage unit as the first average image A (1) as it is,
Thereafter, every time the i-th original image P (i) is input (i = 2, 3,...), The i-th average image A (i) is
a (i) = (1-w) .a (i-1) + w.p (i)
(However, a (i) is the average image A (i)
A pixel value of a predetermined color of a pixel at a predetermined position,
a (i-1) is the average image A (i-1)
A pixel value of a predetermined color of a pixel at a predetermined position,
p (i) is the original image P (i)
A pixel value of a predetermined color of a pixel at a predetermined position,
w is a parameter indicating a predetermined weight (w <1))
It is made using the following arithmetic expression.

(5) According to a fifth aspect of the present invention, in the above-described camera visual field fluctuation detection device according to the third or fourth aspect,
The image comparison unit
A pixel value reading unit that reads a pixel value of a pixel at a predetermined position of one image to be compared as a reference pixel value, and reads a pixel value of the pixel at the predetermined position of the other image to be compared as a comparison pixel value;
A similarity determination unit that determines whether or not the comparison pixel value is within a predetermined similarity range set for the reference pixel value;
When the similarity determination unit determines that the pixel value of the pixel at the predetermined position constituting the foreground image is within the similar range, the pixel value indicating the background area is determined to be outside the similarity range. Is a pixel value writing unit for writing the pixel value indicating the foreground region to the foreground image storage unit,
Is provided.

(6) According to a sixth aspect of the present invention, in the photographing field fluctuation detection device for a camera according to the fifth aspect described above,
In the three-dimensional coordinate system in which the similarity determination unit takes each pixel value of the three primary colors on each coordinate axis, the reference point Q located at the coordinate corresponding to the reference pixel value and the comparison point q located at the coordinate corresponding to the comparison pixel value And the positional relationship between the solid of a predetermined size centered on the reference point Q and the comparison point q is determined, and if it can be determined that the comparison point q is located outside the solid, it is determined that it is outside the similar range, When it can be determined that the position is located in the position, it is determined that it is within the similar range.

(7) According to a seventh aspect of the present invention, in the photographing field fluctuation detection device for a camera according to the first to sixth aspects described above,
The foreground image storage unit includes a first frame memory for storing the foreground image of the odd-numbered frame, and a second frame memory for storing the foreground image of the even-numbered frame;
The foreground area extracting unit performs a process of alternately rewriting the first frame memory and the second frame memory with the newly generated foreground image.

(8) According to an eighth aspect of the present invention, in the photographing visual field fluctuation detection device for a camera according to the first to seventh aspects described above,
The foreground area extraction unit generates a foreground image as a binary image composed of a collection of pixels having one of the first pixel value indicating the foreground area and the second pixel value indicating the background area. And
The difference area extraction unit reads out a pixel at a predetermined position of one foreground image and a pixel at the predetermined position of the other foreground image, and determines whether the pixel values of both are the same or non-identical. The pixel value indicating the difference area is obtained when the non-identical determination is obtained for the pixel at the predetermined position, and the pixel value indicating the outside of the difference area is obtained when the determination is the same. The difference image is generated by giving each of them.

(9) According to a ninth aspect of the present invention, in the photographing field fluctuation detection device for a camera according to the first to eighth aspects described above,
The image input unit inputs an original image with a rectangular outline,
The foreground area extraction unit generates a foreground image having a rectangular outline,
The difference area extraction unit generates a difference image having a rectangular outline,
The block dividing unit divides the difference image having a rectangular outline into N × M rectangular blocks by dividing the difference image vertically into N and horizontally into M.

  (10) According to a tenth aspect of the present invention, the camera field-of-view variation detecting device according to the first to ninth aspects described above is configured by incorporating a dedicated program into a computer.

  (11) According to an eleventh aspect of the present invention, the imaging field variation detection device for a camera according to the first to ninth aspects described above is configured by a semiconductor integrated circuit.

  (12) According to a twelfth aspect of the present invention, there is provided a photographic visual field fluctuation detection device for a camera according to the first to ninth aspects described above, and a frame in which photographic images within a predetermined visual field are continuously given in time series. A fixed point monitoring system is constituted by a fixed point camera that outputs a unit original image, and a change in the field of view of the fixed point camera is detected by a change detection device.

(13) According to a thirteenth aspect of the present invention, in the fixed point monitoring system according to the twelfth aspect described above,
When the variation determination unit in the variation detection apparatus determines that the imaging field of view has changed, the determination result notification unit in the variation detection apparatus issues a variation alarm to the operator.

(14) According to a fourteenth aspect of the present invention, in the fixed point monitoring system according to the twelfth or thirteenth aspect described above,
Further provided is an image recording device for recording images taken by a fixed point camera,
When the fluctuation determination unit in the fluctuation detection apparatus determines that fluctuation has occurred in the shooting field of view, the determination result notification unit in the fluctuation detection apparatus causes the image recording apparatus to stop recording the shot image. A recording stop signal is given.

(15) According to a fifteenth aspect of the present invention, there is provided a photographing field fluctuation detecting device for a camera according to the first to ninth aspects described above, and a frame in which photographed images within a predetermined field of view are continuously given in time series. An image processing system is configured by a fixed-point camera that outputs a unit original image and an image processing device that performs predetermined image processing on the original image sequentially given in time series from the fixed-point camera,
When the image captured by the fixed point camera is given to the variation detection device, and the variation determination unit in the variation detection device determines that the variation has occurred in the field of view, the determination result notification unit in the variation detection device performs image processing. A processing stop signal for stopping image processing is given to the apparatus.

(16) According to a sixteenth aspect of the present invention, there is provided a photographing field fluctuation detecting device for a camera according to the first to ninth aspects described above, an image recording device for recording a plurality of original images in time series, and the image. An image processing system is configured by an image processing device that sequentially reads out original images recorded in a recording device along a time series and performs predetermined image processing on the read original images,
In the image recording device, a captured image in a predetermined field of view captured by a fixed point camera is recorded as a frame-unit original image that is continuously given in time series,
The original image read out from the image recording device by the image processing device is also given to the variation detection device, and when the variation determination unit in the variation detection device determines that a variation has occurred in the shooting field of view, The determination result notification unit gives a processing stop signal for stopping the image processing to the image processing apparatus.

(17) According to a seventeenth aspect of the present invention, in the camera field-of-view variation detection method for detecting a variation in the field of view of the camera,
An original image input stage for inputting an image captured by the camera as an original image in units of frames continuously given in time series,
An image having an average characteristic of a plurality of original images input in the past is obtained as a background image, and the i-th (i = 1, 2,...) Newly input original image P (i) Extracting a foreground area different from the background image and sequentially generating a foreground image F (i) indicating the extracted foreground area;
A difference area is extracted and extracted for the generated i-th (i = 2, 3,...) Foreground image F (i) and (i−1) -th foreground image F (i−1). A differential region extraction stage for sequentially generating a differential image D (i) showing the differential region,
A block division step of dividing each generated difference image D (i) into a plurality of blocks;
For each individual block obtained, an occupancy ratio calculation stage for calculating an occupancy ratio occupied by the difference area,
A block counting step of recognizing a block having an occupation ratio equal to or higher than the first threshold as a sudden change block and counting the number of sudden change blocks for each difference image;
A change detection stage for detecting that a change has occurred in the field of view when a difference image is obtained in which the number of sudden change blocks is equal to or greater than a second threshold;
Is to be executed by a computer.

  According to the camera field-of-view variation detection device of the present invention, a foreground region is extracted from an original image in units of frames continuously given in time series, and a difference image indicating the temporal difference region is obtained. Then, the difference image is divided into blocks, the occupation ratio of the difference area is calculated for each block, the number of suddenly changing blocks whose occupation ratio is equal to or greater than the first threshold is counted, and the count value is equal to or greater than the second threshold. If it becomes, it is determined that the photographing field of view has changed.

  If such a method is adopted, it is determined that the photographing field of view has fluctuated when the difference area is present in a certain proportion of blocks and distributed over a certain number of blocks. For this reason, it is possible to accurately detect the change in the photographing field based on the change in the orientation of the camera by analyzing the photographed image itself of the camera.

  If such a photographic field fluctuation detection device is incorporated in a fixed point monitoring system, it is possible to notify an operator that a variation has occurred in the photographic field of view. On the other hand, it is possible to stop recording and prevent unnecessary recording. Furthermore, if it is incorporated in a system that performs some kind of image processing on a photographed image, the image processing apparatus is stopped when the photographing field of view changes, and inappropriate image processing is prevented from continuing. It becomes possible.

It is a top view which shows the original images P10-P40 arranged in time series, and the foreground images F10-F40 corresponding to this. It is a figure which shows the basic principle of the method of producing the foreground image F (i + 1) corresponding to the original image P (i + 1). It is a figure which shows an example of the production method of average image A (i) used as a background image in this invention. It is a figure which shows the pixel value transition process of 1 pixel on the average image produced based on the method shown in FIG. It is a figure which shows the pixel value determination process of the foreground image F (i) in this invention. FIG. 3 is a three-dimensional color space diagram showing a basic principle of a pixel similarity determination method. It is a three-dimensional color space figure which shows the 1st example of the specific method of the similarity determination method of a pixel. It is a three-dimensional color space figure which shows the 2nd example of the specific method of the similarity determination method of a pixel. It is a three-dimensional color space figure which shows the 3rd example of the specific method of the similarity determination method of a pixel. FIG. 6 is a three-dimensional color space diagram illustrating a pixel similarity determination method using a spheroid model. It is sectional drawing which shows the advantage of the spheroid model with respect to a cylinder model. It is a top view which shows the process which produces | generates the difference image D40 based on two sets of foreground images F30 and F40. It is a top view which shows the process which produces | generates the difference image D31 based on two sets of foreground images F30 and F31. FIG. 14 is an enlarged plan view of a difference image D31 (i-th difference image D (i)) shown in the lower part of FIG. It is a top view which shows the state which divided | segmented the difference image D (i) shown in FIG. 14 into 20 blocks. It is a top view which shows the occupation ratio of the difference area | region calculated about 20 blocks shown in FIG. 15 (When the 1st threshold value Th1 = 20%, the block which becomes more than threshold value Th1 is displayed with a thick frame.). FIG. 16 is a plan view showing the occupancy ratio of the difference area calculated for the 20 blocks shown in FIG. 15 (when the first threshold value Th1 is 15%, blocks that are equal to or greater than the threshold value Th1 are displayed in a thick frame). It is a top view which shows each original image and foreground image corresponding to this when a fluctuation | variation arises in a picked-up visual field between the original images P30 and P31. FIG. 19 is an enlarged plan view of a difference image D31 (i-th difference image D (i)) created based on the foreground images F30 and F31 shown in FIG. It is a top view which shows the state which divided | segmented the difference image D (i) shown in FIG. 19 into 20 blocks. FIG. 21 is a plan view showing the occupancy ratio of the difference area calculated for the 20 blocks shown in FIG. 20 (when the first threshold Th1 = 20%, blocks that are equal to or greater than the threshold Th1 are displayed in a thick frame). FIG. 21 is a plan view showing the occupancy ratio of the difference area calculated for the 20 blocks shown in FIG. 20 (when the first threshold Th1 = 50%, blocks that are equal to or greater than the threshold Th1 are displayed in a thick frame). It is a flowchart which shows the procedure of the imaging | photography visual field fluctuation | variation detection method of the camera which concerns on this invention. It is a block diagram which shows the structure of the fluctuation | variation detection apparatus 100 which concerns on this invention. FIG. 25 is a block diagram illustrating a detailed configuration example of a foreground area extraction unit 120 illustrated in FIG. 24. It is a block diagram which shows the structural example of the fixed point monitoring system which concerns on this invention. It is a block diagram which shows another structural example of the fixed point monitoring system which concerns on this invention. It is a block diagram which shows the structural example of the image processing system which concerns on this invention. It is a block diagram which shows another structural example of the image processing system which concerns on this invention.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic principle of moving object detection >>
The photographing visual field fluctuation detection device according to the present invention has a function of detecting that the orientation of the fixed point camera has fluctuated as a fluctuation of the photographing visual field by analyzing a photographed image itself of the camera. The basic principle is that a moving object is detected from an image captured by a fixed-point camera, and it is determined that the field of view has changed based on the amount of change and the distribution of the moving object. Therefore, here, the basic principle of detecting a moving object from a captured image will be described.

  Now, as shown on the left side of FIG. 1, it is assumed that captured images P10, P20, P30, and P40 in units of frames are given in this order. The illustrated example shows a photographed image by a fixed-point video camera installed on the street, and shows a state in which one vehicle (moving object) passes from right to left. Note that in the case of a general video camera, captured images are output continuously at a cycle of about 30 frames per second, so in practice, a large number of frame images can be obtained at finer time intervals. However, for convenience of explanation, it is assumed that four frame images P10, P20, P30, and P40 are given in order as shown in the figure.

  Looking at these four images, the image P10 is only the background of the city, but in the image P20, since a part of the vehicle has entered the right side, the image is a part of the vehicle overlapped on the background. Similarly, the images P30 and P40 are images in which vehicles overlap on the background. Since the background is stationary, it is common for the images P10 to P40. However, since the vehicle is a moving object, the images P10 to P40 are different. Here, on each image, the area constituting the background is referred to as the background area b, and the area constituting the vehicle (moving object) is referred to as the foreground area f.

  The detection of the moving object described here is to distinguish the background area b and the foreground area f from each of the given images P10 to P40. The images F10 to F40 shown on the right side of FIG. 1 are images showing the distinction between the background region b and the foreground region f for the images P10 to P40 shown on the left side. The hatched area in the figure is the foreground area f, and the white area is the background area b. Since it is only necessary to identify which region, these images F10 to F40 are given as binary images. For example, if the pixel value “0” is given to the pixels belonging to the background region b and the pixel value “1” is given to the pixels belonging to the foreground region f, the images F10 to F40 are 1-bit pixels. It can be obtained in the form of image data consisting of a collection of pixels having values.

  According to the moving object detection process described here, a corresponding binary image can be created for each given frame image. In other words, the essence of this moving object detection process can be said to be a process of creating a binary image indicating the distinction between the background region b and the foreground region f for each frame-by-frame captured image continuously given. . Therefore, hereinafter, for convenience, captured images continuously given in time series are referred to as “original images”, and binary images created based on these “original images” are referred to as “foreground images”.

  In the example illustrated in FIG. 1, an example in which foreground images F10 to F40 are created for each of the original images P10 to P40 is illustrated. The foreground image F10 is entirely composed of a background area b10 (white area), which indicates that there is no moving object on the original image 10. On the other hand, the foreground images F20, F30, and F40 are respectively composed of foreground regions f20, f30, and f40 (hatched regions) and the remaining background regions b20, b30, and b40 (white regions). ing. Here, the foreground areas f20, f30, and f40 indicate areas occupied by the moving object.

  Thus, the foreground area f on the foreground image F indicates the area occupied by the moving object on the original image P. By using this foreground image F, the shape and size of the moving object can be grasped. It is possible to perform processing for zooming in on the moving object part and confirming the vehicle number or specifying a person. In addition, it is possible to track a moving object on the screen while zooming up, or to grasp a moving path.

  Of course, it is difficult to create such a foreground image F from only one original image P. For example, when only the illustrated original image P20 is given as a still image, the human brain can recognize the presence of the vehicle from various information, but only from this one still image P20, the computer can recognize the vehicle. In order to recognize the existence and generate the illustrated foreground image F20, processing based on a complicated algorithm is required. Therefore, a general moving object detection algorithm employs a method of determining a difference area between the original image and the background image as a moving object area (foreground area f).

  For example, if the original image P10 is designated as a background image in advance, the difference area is recognized as the foreground area f20 by comparing the original image P20 when the original image P20 is input, and the foreground image F20 as shown in the figure is obtained. Can be created. As a method for recognizing the difference area, when the newly input original image P20 and the background image P10 are compared on a pixel basis, and the pixel values of both the pixels are the same or similar, the pixel determines the background area b20. If the pixel is a constituent pixel and the pixels are not similar, a method may be adopted in which the pixel is a pixel constituting the foreground region f20 constituting the moving object.

  As the background image, it is possible to use a still image that has been captured in a state where there is no moving object in advance (for example, the state shown in the original image P10). However, due to factors such as weather and time, the lighting environment is constantly changing. It is inappropriate to continue using a still image taken at a specific time point as a background image as it is. Therefore, usually, a method is employed in which an average image having an average characteristic of these original images is created based on a plurality of original images input in the past, and this average image is used as a background image. In the example described here, the average image obtained by such a method is used as the background image. Since the average image is sequentially updated each time a new image is input, it always functions as an optimum background image at that time.

  FIG. 2 is a diagram showing the basic principle of detecting a moving object by such a method. Here, the images P (1), P (2),..., P (i-2), P (i-1), and P (i) shown in the upper part of the figure are given continuously in time series. A frame-unit original image is shown, and a state in which images for i frames from the first original image P (1) to the i-th original image P (i) have been given is shown. . On the other hand, the image P (i + 1) shown on the left in the middle of the figure is the (i + 1) th original image newly given, and the image A (i) shown on the right of the middle of the figure is the past shown in the upper part of the figure. Is an average image having an average characteristic of the original images P (1) to P (i) for i frames given to.

  The foreground image F (i + 1) for the newly input original image P (i + 1) is created by comparing the original image P (i + 1) with the average image A (i). The lower part of FIG. 2 shows the foreground image F (i + 1) created in this way. Specifically, the pixel value of each pixel on the original image P (i + 1) is compared with the pixel value of the corresponding pixel on the average image A (i). By giving a pixel value (for example, “0”) indicating b (i + 1), and for a pixel determined to be out of the similar range, a pixel value (for example, “1”) indicating the foreground region f (i + 1) is given. A foreground image F (i + 1) consisting of a binary image may be created.

  Subsequently, when the (i + 2) th original image P (i + 2) is input, an average image A () having an average characteristic up to the (i + 1) th original image P (i + 1) input in the past. i + 1) is newly created, and the foreground image F (i + 2) is created by comparing the original image P (i + 2) and the average image A (i + 1). In this way, since the average image is sequentially updated, even if the lighting environment changes due to factors such as weather and time, it can function as a background image suitable for each time point.

  Note that when creating an average image, it is preferable to perform weighting along a time series, and obtain a weighted average with the latest original image information weighted. For example, if an average is obtained by giving a weight of 1 for an original image input in the past 10 minutes and a weight of 0 for an original image input before that, an average image for the original image obtained in the latest 10 minutes is always obtained. It can be obtained and can cope with a change in the lighting environment. Of course, the weights can be set finely so that the latest original image always has a higher weight.

  After all, in the case of the example shown in FIG. 2, when creating the average image, when the i-th original image P (i) is input (i = 1, 2,...), For the past original image including the image P (i), the weighted average value of the pixel values for each color of the pixels at the corresponding positions is calculated, and the i-th pixel composed of a collection of pixels having the average value. An average image A (i) may be created. When the (i + 1) th foreground image F (i + 1) is created for the (i + 1) th original image P (i + 1), the original image P (i + 1) and the i-th average image A What is necessary is just to compare (i).

  However, if a moving image is input at a general frame rate of 30 frames per second, even if it is an original image for the past 10 minutes, in order to store all the image data, the image data buffer Considerable storage capacity is required. Therefore, practically, when the i-th original image P (i) is given, the (i−1) -th average image A (i−1) and the i-th original image P ( Based on the two images i), a method of creating the i-th average image A (i) is adopted. FIG. 3 is a diagram illustrating an example of an average image creation method based on such a method.

First, when the first original image P (1) is given, the original image P (1) is processed as it is as the first average image A (1). Then, every time the i-th original image P (i) shown in the upper right of FIG. 3 is inputted (i = 2, 3,...), The (i−) shown in the upper left of FIG. 1) The i-th average image A (i) shown in the middle part of FIG. 3 is shown in the lower part of FIG.
a (i) = (1-w) .a (i-1) + w.p (i) Formula (1)
It may be created using the following arithmetic expression. Here, a (i) is a pixel value of a predetermined color of a pixel at a predetermined position of the average image A (i), and a (i-1) is a pixel value of the pixel at the predetermined position of the average image A (i-1). The pixel value of the predetermined color, p (i) is the pixel value of the predetermined color of the pixel at the predetermined position of the original image P (i), and w is a parameter (w <1) indicating a predetermined weight.

  In short, the pixel value a (i−1) of the (i−1) -th average image A (i−1) calculated for the original image up to the original image P (i−1) given immediately before, For the pixel value p (i) of the i-th original image P (i) that is newly given, an average value considering the weight w is calculated, and the calculated value is used as the i-th average image A (i). The process of setting the pixel value a (i) is performed. Thus, every time a new original image P (i) is input, the average image A (i) is updated, and only the immediately preceding average image A (i-1) is used for the update. By doing so, it is not necessary to store and hold past original image data for several frames, so that it is possible to greatly save the capacity of the buffer necessary for processing.

  As shown in FIG. 3, as the value of the weight w is increased, the pixel value of the latest original image P (i) greatly affects the pixel value of the average image A (i). FIG. 4 is a diagram showing a pixel value transition process of one pixel on the average image created based on the method shown in FIG. In this example, a specific example in which the weight w = 0.01 is shown.

  For example, if the pixel value of the pixel at the specific position of the first (i = 1) original image P (1) is p (1) = 100, the specific of the first average image A (1) is specified. The pixel value a (1) of the pixel at the position is a (1) = 100 as it is. Subsequently, assuming that the pixel value of the pixel at the specific position of the second original image P (2) is p (2) = 100, the pixel at the specific position of the second average image A (2). The pixel value a (2) is calculated as a weighted average “0.99 × 100 + 0.01 × 100” between a (1) and p (2), so that a (2) = 100.

  In the illustrated example, since the pixel value at the specific position of the original image remains 100 from p (1) to p (4), the pixel value at the specific position of the average image is also a (1) to a Maintain 100 until (4). However, in the fifth original image P (5), the pixel value a (5) of the fifth average image A (5) is changed in the lower part of FIG. As shown in the equation, a (5) = 101 is obtained by calculating the weighted average “0.99 × 100 + 0.01 × 200” of a (4) and p (5). Further, since the pixel value p (6) = p (7) = 200 is maintained in the sixth original image P (6) and the seventh original image P (7), the lower part of FIG. As shown in the equation, in the sixth average image A (6) and the seventh average image A (7), the pixel value a (6) = 102 and the pixel value a (7) = 103 gradually increase. doing.

  As in this example, when the weight w is set to a value of about 0.01, even if the pixel value of the original image changes abruptly, the pixel value of the average image does not change immediately, and the pixel value of the original image It will approach you slowly. Therefore, as in the example shown in FIG. 1, even if a moving image through which the vehicle passes is given, the transient pixel value change does not cause a large change in the pixel value of the average image, and the average image It can serve as an image. On the other hand, if the lighting environment changes, such as when the time changes from daytime to dusk, the change persists across many frame images, so the average image pixel value also changes following the change. become. Therefore, an average image suitable for a daytime background image is obtained during the daytime, and an average image suitable for a sunset background image is obtained at dusk.

  The foreground image in the present invention is created by comparing individual original images using the average image created by such a method as a background image. FIG. 5 is a diagram showing a pixel value determination process of the foreground image created in this way. The foreground image F (i) for the i-th original image P (i) is a binary image composed of a collection of pixels having pixel values determined by the following method. That is, the pixel value f (i) of the pixel at the specific position of the foreground image F (i) is equal to the pixel value p (i) of the pixel at the specific position of the original image P (i) and the average image A (i−1). ) Is compared with the pixel value a (i-1) of the pixel at the specific position, and if both pixel values are within the similar range, a pixel value indicating that the pixel is in the background region b (for example, If both pixel values are outside the similar range, a pixel value (for example, “1”) indicating that the pixel is in the foreground region f is given.

  Actually, since the original image is a color image having pixel values of three primary colors (for example, in the case of the RGB color system, pixel values of three colors R, G, and B), the original image P (i ) And the pixel value a (i) of each pixel constituting the average image A (i) are each composed of three independent values for each color. It will be. Then, the calculation of Expression (1) is executed independently for each color. For example, the pixel value a (i) of a pixel at a specific position on the average image A (i) is composed of three sets of pixel values, an R color pixel value, a G color pixel value, and a B color pixel value. The R pixel value is a value obtained as a weighted average of the R pixel values of the past original image.

  Therefore, the comparison between the pixel value p (i) and the pixel value a (i−1) in FIG. 5 is not a simple comparison between two values, but a comparison between three sets of pixel values and three sets of pixel values. become. Naturally, the pixel similarity determination is processing for determining whether or not the three sets of pixel values are similar to each other. Specific processing for determining the similarity of pixels will be described in detail in Section 2.

<<< §2. Pixel similarity determination method >>>
In the present invention, foreground image F needs to be created by extracting foreground region f constituting the moving object from original image P (input image). As a method for this, in the method described in §1, the original image P and the average image A (background image) are compared in pixel units, and if both pixels are similar, the pixels in the background region b are dissimilar. In the case of, it is determined that the pixel is in the foreground area f. In this way, in the method of determining similarity on a pixel-by-pixel basis, pixel similarity determination is a factor that affects the detection accuracy of a moving object. Judgment is very important.

  For example, in §1, when the image P10 in FIG. 1 is an average image (background image) and the image P20 is an original image (input image), a comparison between the two results in a foreground image such as the image F20 being obtained. Stated. Here, the foreground region f20 of the foreground image F20 is a region in which the pixels are determined to be outside the similar range as a result of comparing both the images P10 and P20, and the background image b20 is a region in which the pixels are determined to be in the similar range It is. In this case, a correct foreground image cannot be obtained unless the similar range is appropriately set.

  That is, in FIG. 1, the exact same image is drawn in the background part, but in an actual landscape, the lighting environment due to sunlight changes due to the movement of the clouds, and the position and orientation of the leaves of the trees change slightly due to the wind. . Therefore, if the criteria for judging whether or not the pixels are similar is strict, an erroneous determination that the pixel is actually in the background portion but outside the similar range occurs, and the pixel in the foreground region f is recognized. Conversely, if the pixel similarity criterion is relaxed, an erroneous determination that the pixel is in the background area b occurs because an erroneous determination that the pixel is in the similar range occurs even though it is actually the foreground part.

  In addition, in the case of a color image, as described above, the similarity of pixels needs to be performed based on a comparison between three pixel values of one pixel and three pixel values of the other pixel. It is possible to set a similar similarity criterion. The determination accuracy also varies depending on what kind of similarity determination criterion is adopted. Usually, such pixel comparison of color images is performed by plotting coordinate points corresponding to both pixel values in a three-dimensional color space and examining the spatial positional relationship between the two coordinate points. Hereinafter, there will be described several methods for determining similarity in the color space when each pixel has an RGB color system pixel value.

  FIG. 6 is a three-dimensional color space diagram showing the simplest method for determining similarity of pixels. As illustrated, coordinate points A and P are plotted in an RGB three-dimensional orthogonal coordinate system in which the pixel values of the three primary colors R, G, and B are taken as coordinate axes. Here, the coordinate point A (Ra, Ga, Ba) is a point obtained by plotting pixel values (Ra, Ga, Ba) of pixels on the average image A indicating the background, and the coordinate point P (Rp, Gp, Bp). ) Is a point in which pixel values (Rp, Gp, Bp) of pixels on the original image P to be determined are plotted.

  Here, if the distance between the two points A and P in the three-dimensional color space is δ, the distance δ is an index indicating the similarity between the two pixels. If δ = 0, both pixels have exactly the same pixel value, and the overall similarity of the three pixel values decreases as δ increases. Therefore, if a predetermined threshold value δth is set in advance and it is determined that δ <δth is within the similar range and δ ≧ δth is out of the similar range, a temporary determination criterion can be set. .

  Alternatively, starting from the origin O, a vector Va going to the coordinate point A and a vector Vp going to the coordinate point P are defined, and an angle θ formed by both vectors Va and Vp is used as an index indicating the similarity between both pixels. You can also. When θ = 0, the two points A and P are not necessarily the same point, but are points on the same straight line passing through the origin O. Therefore, the hues of both pixels determined by at least the blend ratio of the three primary colors R, G, and B are used. Will be similar. Eventually, if θ is small, the similarity of hue will increase. Therefore, like the criterion using index δ, a predetermined threshold θth is set in advance, and if θ <θth, If it is determined that θ ≧ θth is outside the similar range, a temporary determination criterion can be set.

  However, when such a simple similarity determination method based on the distance δ and the angle θ is used for a moving object detection process in a general live-action image, it may be difficult to obtain a satisfactory detection result. Are known. This means that, with only simple parameter values such as distance δ and angle θ, appropriate judgment criteria corresponding to the actual shooting environment, such as changes in the lighting environment due to sunlight and changes in light and darkness due to wind, etc. should be set. This is because it is difficult.

  As a specific method for dealing with such a problem, a similarity determination method using a cylindrical model has been proposed. FIG. 7 is a three-dimensional color space diagram showing the basic principle of the method using this cylindrical model. In FIG. 7, as in FIG. 6, the pixel values (Ra, Ga, Ba) of the pixels on the average image A indicating the background are plotted as the coordinate point A, and the pixels on the original image P to be determined are plotted. Pixel values (Rp, Gp, Bp) are plotted as coordinate points P. Here again, the vectors Va and Vp are vectors from the origin O toward the points A and P.

  In this model, a cylinder C having a length L with a predetermined radius around a coordinate point A is defined as shown in the figure. In other words, the cylinder C is a cylinder arranged such that the reference axis Z defined on the vector Va is the central axis and the center point is at the coordinate point A. Then, the space area inside the cylinder C is defined as a similar range of pixels constituting the average image A indicating the background. Therefore, as shown in the example in the figure, a pixel having a pixel value corresponding to the coordinate point P is determined to be outside the similar range because it is located outside the cylinder C, and recognized as a pixel constituting the foreground region f. become. On the contrary, if the coordinate point P is located inside the cylinder C, it is determined to be within the similar range and is recognized as a pixel constituting the background region b.

  A feature of this cylindrical model is that a cylindrical region having a reference axis Z facing the direction of the vector Va as a longitudinal direction is set. The radius of the cylinder C functions as a parameter for determining the hue similarity range, and the length L functions as a parameter for determining the brightness similarity range. As described above, in an actual scene, the pixel value of the background region b varies with time due to changes in the lighting environment due to sunlight, changes in brightness such as trees due to wind, and the like. , It is considered to have a greater effect on brightness than hue. Therefore, as in this model, it makes sense to set the similarity range based on the cylinder C having the reference axis Z as the longitudinal direction.

  On the other hand, JP 2011-076311 A also proposes a model that uses a spheroid instead of the column C of the column model shown in FIG. FIG. 8 is a three-dimensional color space diagram showing a spheroid model in which the cylinder C shown in FIG. 7 is replaced with a spheroid Ea. Here, the spheroid Ea indicating the similar range is an ellipse having the coordinate point A indicating the pixel value of the average image A as the center point and the reference axis Za defined on the vector Va as the major axis direction. It is a rotating body obtained by rotating using Za as a rotation axis. The size of the spheroid Ea is defined by the length La1 in the major axis direction and the length La2 in the minor axis direction.

  Even when this spheroid model is used, the similarity determination method is the same as the above-described determination method using the cylindrical model, and the pixels plotted in the inner area of the spheroid Ea are coordinate points. A pixel having a pixel value corresponding to A is determined as a similar pixel. Specifically, in the case of the illustrated example, a pixel having a pixel value corresponding to the coordinate point P is determined to be outside the similar range because it is located outside the spheroid Ea, and the pixels constituting the foreground region f Certified. On the contrary, if the coordinate point P is located inside the spheroid Ea, it is determined to be within the similar range and is recognized as a pixel constituting the background region b.

  On the other hand, FIG. 9 is a three-dimensional color space diagram showing variations of the spheroid model shown in FIG. As shown in the figure, a spheroid Ep indicating a similar range in this model has a coordinate point P indicating a pixel value of the original image P to be determined as a center point, and a reference axis Zp defined on the vector Vp as a major axis It is a rotating body obtained by rotating an ellipse as a direction around a reference axis Zp as a rotation axis. The size of the spheroid Ep is defined by the length Lp1 in the major axis direction and the length Lp2 in the minor axis direction.

  In the case of the model shown in FIG. 8, a spheroid Ea indicating a similar range is defined on the basis of the coordinate point A on which the pixel values (Ra, Ga, Ba) of the average image A indicating the background are plotted, and is a comparison target. While it has been determined whether the coordinate point P on which the pixel values (Rp, Gp, Bp) of the original image P are plotted is inside or outside the spheroid Ea, the model shown in FIG. In this case, a spheroid Ep indicating a similar range is defined with reference to the coordinate point P on which the pixel value (Rp, Gp, Bp) of the original image P to be compared is plotted, and shows the background. It is determined whether the coordinate point A on which the pixel values (Ra, Ga, Ba) of the average image A are plotted is inside or outside the spheroid Ep.

  In short, the former takes a method of determining whether or not the pixel of the newly input original image P is in the similar range of the pixels of the past average image A indicating the background, whereas the latter is A method of determining whether or not the pixels of the past average image A indicating the background are within the similar range of the pixels of the newly input original image P is taken. After all, the only difference between the two is in which of the two pixels to be compared is used as a reference to define the spheroid.

  Therefore, in the following description, of the two images to be compared, the pixel value of one image is expressed as a reference pixel value (R, G, B), and the pixel value of the other image is expressed as a comparison pixel value (r , G, b). Also, the coordinate point Q (R, G, B) on which the reference pixel value (R, G, B) is plotted in the three-dimensional color space is called the reference point Q, and the comparison pixel value (r, g, b) is plotted. The coordinate point q (r, g, b) is referred to as a comparison point q. The similarity determination is made based on the positional relationship between the spheroid defined around the reference point Q and the comparison point q.

  FIG. 10 is a three-dimensional color space diagram illustrating a spheroid model according to such general notation. If the vector V starting from the origin O and going to the reference point Q is a vector V, the reference axis Z is an axis on the vector V. The spheroid E is a rotator obtained by rotating an ellipse having the reference point Q as the center point and the reference axis Z defined on the vector V as the major axis direction with the reference axis Z as the rotation axis. It is. Here, if the comparison point q (r, g, b) is located inside the spheroid E, both pixels are determined to be within the similar range, and if located outside the spheroid E, both pixels are similar. Determined to be out of range.

  The model shown in FIG. 8 uses the pixel value of the pixel a (i-1) on the average image A (i-1) as the reference pixel value (R, G, B) when performing the similarity determination shown in FIG. The reference point Q (R, G, B) is plotted, and the pixel value of the pixel p (i) on the original image P (i) is set as the comparison pixel value (r, g, b). , B) are plotted. On the other hand, the model shown in FIG. 9 uses the pixel value of the pixel p (i) on the original image P (i) as the reference pixel value (R, G, B) when performing the similarity determination shown in FIG. ) Is plotted as a reference point Q (R, G, B), and the pixel value of the pixel a (i−1) on the average image A (i−1) is set as a comparison pixel value (r, g, b). q (r, g, b) is plotted.

  According to Japanese Patent Application Laid-Open No. 2011-076311, the spheroid model shown in FIG. 10 has the following two advantages compared to the cylindrical model shown in FIG. The first advantage is that, when used in the pixel similarity determination process in the moving object detection method described in §1, it is possible to detect a moving object more accurately without the influence of illumination fluctuations due to time or weather. It is a point to become.

  FIG. 11 is a cross-sectional view showing the advantages of the spheroid model over the cylindrical model. In the example shown in the figure, when the cross section of the cylinder C and the cross section of the spheroid E are compared, all of them are figures having the reference point Q as the center and the direction of the reference axis Z as the longitudinal direction, but the figure is hatched. It can be seen that the area is determined to be the inner area in the cylinder C, whereas it is determined to be the outer area in the spheroid E. Therefore, when the comparison point q is located in the hatched area, it is determined that the comparison point q is within the similar range on the one hand and is outside the similar range on the other hand. Of course, which judgment result is correct cannot be generally determined, but at least as long as it is used for detection processing of a moving object using a real image, the latter judgment result may be a correct result. high.

  That is, in the case of a cylindrical model, when the comparison point q is located in the hatched area in the figure, the pixel to be determined is similar to the background image, and thus is a pixel in the background area b. Is relatively disproportionate to the background image and should have been a pixel in the foreground region f. As shown in FIG. 11, when compared with cross-sectional shapes, the cross section of the spheroid E is an ellipse made of a curved surface, whereas the cross section of the cylinder C is a rectangle made of a straight line. An ellipse whose cross section is a curved surface is more suitable as the boundary line for the negative determination.

  In this way, at least when used for detection processing of a moving object based on a live-action image, the first advantage is obtained that the detection accuracy of the spheroid model is improved compared to the cylindrical model. On the other hand, the second advantage obtained by adopting the spheroid model is that the calculation burden can be reduced. This is because the size of the spheroid E can be set to a value proportional to the distance Δ between the origin O of the three-dimensional coordinate system and the reference point Q.

  Specifically, as shown in FIG. 10, a value α = m · Δ obtained by multiplying the distance Δ by a predetermined parameter m (where m <1) is a major axis radius, and a predetermined parameter n (where n An ellipse having a minor axis radius β = n · Δ obtained by multiplying <m) is set so that the major axis overlaps the reference axis Z connecting the origin O and the reference point Q, and the reference point Q is centered. The spheroid E may be defined by arranging it to be a point and rotating it around the reference axis Z.

  When the size of the spheroid E is determined under such conditions, the size of the spheroid E increases as the reference point Q moves away from the origin O, and the similarity range increases accordingly. This means that the higher the lightness, that is, the highlight portion of the image, the wider the similar range. However, in practice, there is no problem even if such a similar range is set. On the other hand, the calculation burden for determining whether or not it is within the similar range, that is, whether the inside of the spheroid E is inside or outside is set such that the long axis radius is α = m · Δ and the short axis radius is β = n · Δ. (That is, by setting the size of the ellipse to a value proportional to the distance Δ), it can be greatly reduced. Details thereof are described in Japanese Patent Application Laid-Open No. 2011-076311, and the description thereof is omitted here.

  As described above, the example using the cylindrical model and the example using the spheroid have been described as the similarity determination method of the pixels at the positions corresponding to each other with respect to the original image composed of the color image and the background image (average image). In practicing the present invention, the pixel similarity determination method for a color image is not necessarily limited to the method described above. For example, a solid having an arbitrary shape may be used instead of a cylinder or a spheroid.

  When a three-dimensional model having an arbitrary shape is adopted, in a three-dimensional coordinate system in which each pixel value of the three primary colors is taken as each coordinate axis, the reference point Q located at the coordinate corresponding to the reference pixel value and the coordinate corresponding to the comparison pixel value are used. The comparison point q is taken, the positional relationship between the comparison point q and a solid with a predetermined size centered on the reference point Q is examined, and if it can be determined that the comparison point q is located outside the solid, it is out of the similar range. If it can be determined that it is located inside, it may be determined that it is within the similar range.

<<< §3. Basic Principle of Method for Detecting Field-of-View Variation According to the Present Invention >>
Next, the basic principle of detecting the change in the photographing field of view by performing the photographing field change detection process according to the present invention for the example shown in FIG. 1 will be described. As described in §1, FIG. 1 shows a photographed image by a fixed point video camera installed on the street, and shows a state in which one vehicle (moving object) passes from right to left. . When such four frame images P10, P20, P30, and P40 are given as original images, moving objects are detected by the method described in §1 and §2, and foreground images F10, F20, F30, and F40 are respectively detected. Is created.

  In the present invention, by analyzing the foreground images F10, F20, F30, and F40 thus created, it is detected that a change in the field of view, that is, a change in the direction of the camera has occurred. For this purpose, a difference area is extracted from the foreground images of the two original images taken with a time difference, and a difference image indicating the extracted difference area is created.

  FIG. 12 is a plan view showing a process of generating a difference image D40 based on two sets of foreground images F30 and F40. Here, the foreground images F30 and F40 shown in the upper part of FIG. 12 are the same as the foreground images F30 and F40 shown in FIG. 1, and are obtained by comparing the original images P30 and P40 with the background image (average image), respectively. It is what was done. The foreground area f30 on the foreground image F30 is an area corresponding to the vehicle, and the foreground area f40 on the foreground image F40 shows a state in which the vehicle has moved leftward. For convenience of explanation, the foreground areas f30 and f40 are shown by changing the direction of hatching with diagonal lines.

  The middle part of FIG. 12 is obtained by superimposing the foreground images F30 and F40 on the same plane, and the mutual positional relationship between the foreground regions f30 and f40 is clearly shown. That is, the area where double hatching (mesh hatching in which diagonal lines of different directions intersect) is shown in the figure indicates the foreground area common to the foreground images F30 and F40, and the area where single hatching is applied is A foreground area that exists only in the foreground image is shown. The difference area mentioned here means an area where this single hatching is applied.

  In other words, in the present invention, the difference area extracted from the pair of foreground images F30 and F40 is a portion where a difference is generated between the foreground areas f30 and f40 when the foreground images F30 and F40 are overlapped (geometrical). Exclusive OR part). The image shown in the lower part of FIG. 12 is a difference image D40 that shows the difference region d40 extracted from the foreground images F30 and F40 in this way. Here, the difference area d40 is indicated by hatching with dots.

  In short, when each of the foreground images F30 and F40 is configured by a binary image in which the pixel value of the pixel in the foreground region f is “1” and the pixel value of the pixel in the background region b is “0”, the difference image D40 Is an image obtained by performing a geometric exclusive OR operation on the foreground images F30 and F40.

  Incidentally, FIG. 1 illustrates a case where four frame images P10, P20, P30, and P40 are sequentially given as original images for convenience of explanation, and FIG. 12 corresponds to the original images P30 and P40 among them. For the foreground images F30 and F40 created in this way, the process for obtaining the difference image D40 is shown. However, as described above, in the case of a general video camera, captured images are output continuously at a period of about 30 frames per second, so in practice, a large number of frame images are obtained at finer time intervals. Will be. Therefore, it is assumed here that an original image P31 in which the moving object (vehicle) has moved slightly to the left is obtained as the next frame of the original image P30, and is created corresponding to the original images P30 and P31. Consider a process for obtaining a difference image D31 for the foreground images F30 and F31.

  FIG. 13 is a plan view showing a process of generating a difference image D31 based on two sets of foreground images F30 and F31. The foreground image F30 shown in the upper part of FIG. 13 is the same as the foreground image F30 shown in the upper part of FIG. 12, but the foreground image F31 is an image obtained before the foreground image F40. The foreground region f31 indicating the position of is moved slightly to the left compared to the foreground region f30.

  The middle part of FIG. 13 is obtained by superimposing foreground images F30 and F31 on the same plane, and the mutual positional relationship between foreground regions f30 and f31 is clearly shown. The image shown in the lower part of FIG. 13 is a difference image D31 showing the difference area d31 extracted from both foreground images F30 and F31. Again, the difference area d31 is shown hatched with dots.

  Eventually, the difference area d indicates the amount of change in the foreground area f (area indicating the moving object) for the pair of foreground images F to be compared. The larger the area, the larger the amount of change. become. In other words, the faster the moving speed of the moving object, the larger the area of the difference area d.

  However, the area of the difference region d also depends on the time difference between the pair of foreground images to be compared. For example, the area of the difference area d40 in the difference image D40 shown in the lower part of FIG. 12 (hatched portion by dots) is larger than the area of the difference area d31 in the difference image D31 shown in the lower part of FIG. This is because the time difference at the time of photographing the original images P30 and P40 is set longer than the time difference at the time of photographing the original images P30 and P31. Therefore, in order to evaluate the speed of the moving object based on the area of the difference region d, it is necessary to determine in advance a time difference between a pair of foreground images to be compared.

  For example, if the fixed-point camera is a video camera that outputs captured images continuously at a cycle of 30 frames per second, the series of captured images are given as a large number of frame images with a time interval of 1/30 seconds. Therefore, if all of these images are captured as original images to generate a foreground image, and a difference image is generated between each foreground image and the immediately preceding foreground image, a pair to be compared. The time difference between the foreground images is 1/30 seconds. Of course, it is also possible to perform operations such as taking captured images obtained from the video camera as original images every other frame, in which case the time difference between the pair of foreground images to be compared is 1/15 seconds. become.

  As described above, the time difference between the pair of foreground images used when obtaining the difference image D can be arbitrarily set, but the setting of this time difference is a parameter that affects the area of the difference region d. In performing variation detection under the same determination condition, it is preferable to maintain the time difference at a constant value. In the following description, a captured image given at a rate of 30 frames per second is directly taken as an original image, 30 foreground images are created per second, and a pair of foreground images adjacent to each other on the time axis (1/30 A case where the difference image D is obtained for each of the foreground images having a time difference of seconds is taken as an example.

  For example, in the example shown in FIG. 13, the foreground images F30 and F31 are images created for the original images P30 and P31 having a time difference of 1/30 seconds, and the foreground region f31 is the foreground region f30. The position after 1/30 second of the moving object (vehicle) indicated by is shown. The difference image D31 is an image indicating a region where the moving object (vehicle) fluctuates in 1/30 seconds as the difference region d31. Similarly, a foreground image F32 is created for the original image P32, a difference image D32 is created for the foreground images F31 and F32, a foreground image F33 is created for the original image P33, and a difference image for the foreground images F32 and F33. D33 is created, and so on.

  The basic principle of the present invention is that a moving object is detected from a captured image of a fixed point camera, and it is determined that the photographing field of view has fluctuated based on the fluctuation amount and fluctuation distribution of the moving object. More specifically, two conditions are imposed to determine that a change has occurred in the field of view. The first condition is that the amount of fluctuation of the moving object is greater than or equal to a predetermined reference, and the second condition is that the fluctuation of the moving object is distributed over the entire photographing screen.

  The reason for imposing the first condition is to prevent erroneous determination that the photographing field of view fluctuates due to the influence of various fluctuation components. FIG. 1 shows an example in which a moving object (vehicle) is detected in an ideal situation where the background image does not change at all. Foreground regions f20, f30, and f40 included in the foreground images F20, F30, and F40 are shown in FIG. The vehicle contour is accurately detected. However, in the actual landscape, a fine foreground area is formed in the part that should originally be the background area due to the influence of various fluctuation components such as changes in the lighting environment due to the movement of clouds and changes in the leaves of trees due to the wind. . If the first condition that the amount of fluctuation of the moving object is greater than or equal to a predetermined reference is imposed, it is possible to prevent a fine foreground region due to such a fluctuation component from contributing to the determination.

  Here, the fluctuation amount of the moving object necessary for the first condition determination can be grasped as the area of the difference region d in the difference image D. For example, in the example shown in FIG. 13, the area of the difference region d31 indicates the amount of change of the moving object. Therefore, in the present invention, in order to determine that the photographing field of view has changed, the first condition that the area of the difference region d31 is equal to or larger than a predetermined reference is imposed.

  However, if the first condition is only satisfied, it cannot be determined that the field of view has changed (the direction of the fixed point camera has changed). In fact, in the case of the example shown here, even if the area of the difference area d31 shown in FIG. 13 is equal to or larger than a predetermined reference, the cause is that the vehicle has passed through the imaging field of view at a certain speed, and the imaging field of view varies. This is not because of

  Therefore, in the present invention, a second condition is set that the movement of the moving object is distributed over the entire shooting screen. When a moving object such as a person or a vehicle passes through the photographing field, the movement of the moving object is detected as a difference area d31 as shown in FIG. On the other hand, when the orientation of the fixed-point camera changes, not only people and vehicles but all subjects including the background in the field of view will move, so to speak, people, vehicles, houses, roads, trees, sky Are all detected as moving objects. Therefore, in the difference image obtained in that case, the difference area will be distributed over the entire screen.

  The second condition is imposed based on such an idea, and can be determined from the viewpoint of whether or not the difference area is widely distributed in the entire difference image. For example, in the case of the difference image D31 shown in FIG. 13, the difference area d31 is mainly distributed only in the lower right part of the screen, and therefore cannot be determined as being distributed throughout. Therefore, the second condition is not satisfied.

  As described above, in the present invention, in order to determine whether or not the shooting field of view has changed, that is, whether or not the orientation of the fixed-point camera has changed, the above two conditions need to be determined. Therefore, in the present invention, a method is used in which the obtained difference image D is divided into a plurality of blocks, and the occupation ratio occupied by the difference area is calculated for each block. Then, a block whose occupation ratio is equal to or greater than the first threshold is recognized as a sudden change block, the number of sudden change blocks is counted for each individual difference image, and a difference image whose number of sudden change blocks is equal to or greater than the second threshold is obtained. When it is obtained, it is detected that the photographing field of view has changed.

  I will explain this technique with a concrete example. FIG. 14 is an enlarged plan view of the difference image D31 shown in the lower part of FIG. Here, for the sake of general explanation, the image shown in FIG. 14 is referred to as a difference image D (i) in the sense of the i-th difference image, and the difference area included therein is referred to as a difference area d (i ). In the present invention, the difference image D (i) is divided into a plurality of blocks. FIG. 15 is a plan view showing an example in which the difference image D (i) is divided into 20 blocks B1 to B20 by dividing the difference image D (i) into 4 parts vertically and 5 parts horizontally.

  For the 20 blocks B1 to B20 obtained in this way, the occupation ratio H occupied by the difference area (in the example of the figure, a hatched portion by dots) is calculated. For example, in the example shown in FIG. 15, the blocks B1 to B11 and B16 do not include any difference area, so the occupation ratio H for these blocks is H = 0%. On the other hand, for the block B12, since the minute portion in the lower right corner is the difference area, a result of H = 2% is obtained. Similarly, H = 17% for the block B13 and H = 3 for the block B14. %, H = 12% for the block B15, and so on.

  FIG. 16 is a plan view showing the occupation ratio H of the difference area calculated for each of the 20 blocks B1 to B20 shown in FIG. Here, the first threshold value is set as Th1 = 20%, and blocks that are equal to or greater than the threshold value Th1 are displayed in a thick frame. As shown in the figure, the lower right block B20 (H = 21%) is the only block whose occupation ratio H is equal to or greater than the first threshold value. In the present invention, a block whose occupation ratio H is equal to or greater than the first threshold value Th1 is called a sudden change block. Within this sudden change block, it can be estimated that the position of the subject is changing rapidly. Of course, it is only the value of the occupation ratio H for the block whether such a sudden change of the subject is caused by the passing of a vehicle or the like or the change of the camera direction. Can not be judged from.

  Therefore, when the number of blocks recognized as sudden change blocks among a plurality of blocks constituting one difference image is counted, and a difference image in which the number of sudden change blocks is equal to or greater than the second threshold Th2 is obtained. Therefore, it is determined that the photographing field of view has changed. For example, when a value of Th2 = 10 (a value of 50% for a total of 20 blocks) is set as the second threshold Th2, if the number of suddenly changing blocks is 10 or more, the field of view changes. Can be determined to have occurred. In the example shown in FIG. 16, since the sudden change block indicated by the thick frame is only the block B20, it does not reach the second threshold Th2, and it is not determined that the photographing field of view has changed.

  Of course, since the criterion for determining whether or not the block is a sudden change block is given as the first threshold Th1, if the setting of Th1 is changed, the number of sudden change blocks also changes. For example, FIG. 17 is a plan view showing an example in which a sudden change block is displayed with a thick frame when the first threshold is relaxed to Th1 = 15%. In this case, the blocks B13, B17, and B20 are recognized as suddenly changing blocks, and the number of suddenly changing blocks is 3. However, since the second threshold value Th2 is less than 10, it is in the field of view. It is not determined that a change has occurred.

  Actually, the example shown in FIG. 1 is an example showing a photographed image obtained when photographing a state in which the vehicle passes while maintaining the same photographing field of view with no change in the direction of the camera. Is a correct decision.

  In contrast, consider the case shown in FIG. This case is exactly the same as the case shown in FIG. 1 until the time when the original image P30 is shot. However, the direction of the camera changes immediately before the original image P31 is shot and the shooting field of view changes. is there. For example, in a case where a cat jumps on the camera immediately after the frame image corresponding to the original image P30 is shot and the camera direction suddenly changes downward, the original image P31 like this example is obtained. become. While the field of view of the original images P10 to P30 is the same, the field of view of the image after the original image P31 is completely different.

  Comparing the original images P30 and P31, of course, the point that the passing vehicle is traveling to the left is different, but since the entire field of view is moving downward, the sky that was visible in the upper part disappears, New subjects such as flower beds and dogs appear in the lower part. As a result, most of the foreground image F31 obtained for the original image P31 is the foreground region f31 as shown in the lower left of FIG. This is because the entire subject including the background is moved upward when the shooting field of view of the camera is used as a reference. That is, when the foreground region is extracted by performing the similarity determination of the pixel at the corresponding position on the image by the method described in §2, most of the pixels are determined to be dissimilar and are included in the foreground region. .

  FIG. 19 is an enlarged plan view of the difference image D31 created based on the foreground images F30 and F31 shown in FIG. Here, in order to explain in general terms, the image shown in FIG. 19 is referred to as a difference image D (i) in the sense of the i-th difference image, and the difference area included therein is referred to as a difference area d (i ). Again, as in the example described above, this difference image D (i) is divided into a total of 20 blocks. FIG. 20 is a plan view showing a state in which such a division is performed on the difference image D (i) shown in FIG. Here, for each block, the occupation ratio H occupied by the difference area is calculated, and the authorization process for the sudden change block is performed.

  FIG. 21 is a plan view showing the occupancy ratio H of the difference area calculated for each block shown in FIG. Here, the first threshold is set as Th1 = 20%, and the sudden change block in which the occupation ratio H is equal to or greater than the threshold Th1 is displayed with a thick frame. As a result, 17 blocks out of all 20 blocks are recognized as sudden change blocks. Therefore, similarly to the above-described example, when a value of Th2 = 10 is set as the second threshold value Th2, the number of suddenly changing blocks is equal to or greater than the second threshold value Th2, and thus the shooting field of view varies. Judgment is made.

  FIG. 22 is a plan view showing an example in which the first threshold is set to Th1 = 50% and the condition of the sudden change block is made stricter. Even if conditions are set strictly in this way, 16 blocks indicated by a thick frame among all 20 blocks are recognized as suddenly changing blocks, and it is determined that the shooting field of view has also changed. . In fact, the case shown in FIG. 18 is a case in which the orientation of the camera has changed immediately before the original image P31 is shot and the shooting field of view has changed, so the above determination is correct.

<<< §4. Procedure of the photographing visual field fluctuation detection method according to the present invention >>
Next, the procedure of the variation detection method for detecting the variation in the field of view of the camera (the variation in the direction of the camera) based on the basic principle described in Section 3 will be described with reference to the flowchart of FIG. This procedure is actually executed as image processing by a computer.

  First, in step S1, necessary initial settings are made. Specifically, the number of divisions and the division method of the difference image D, the setting of the first threshold Th1 and the second threshold Th2, the setting of the weight w shown in FIG. 3, the setting of the parameters m and n shown in FIG. 10 (rotation) (Ellipsoid setting) is performed. In addition, after setting the parameter i indicating the number of the original image (captured frame image) to the initial value 1, the initial original image P (1) is input, and this is set as the initial average image A (1) as it is. Then, an initial difference image D (1) between the two is obtained, and subsequently, a process of updating to the parameter i = 2 is performed. In this case, since the initial original image P (1) and the initial average image A (1) are the same image, the entire area of the initial difference image D (1) is the background area b. Subsequent processing after step S2 is performed sequentially after setting the parameter to i = 2.

  First, in step S2, the i-th original image P (i) is input, and in the subsequent step S3, the foreground region f (i) is extracted from the original image P (i), and the foreground region f (i) is extracted. A foreground image F (i) is generated as a binary image that is distinguished from other regions (background region b (i)). For example, when the parameter i = 2, the second original image P (2) is input, and the foreground image F (2) is compared by comparing the original image P (2) with the average image A (1). Will be generated. Here, the specific update method of the average image A (i) and the specific generation method of the foreground image F (i) are as described with reference to FIGS.

  Next, in step S4, a difference area d (i) is obtained by taking a difference (geometric exclusive OR) between the foreground image F (i) and the previous foreground image F (i-1). Are extracted, and a difference image D (i) is obtained. This difference image D (i) is an image showing a difference area d (i) as shown in FIGS. 14 and 19, for example. In step S5, as shown in FIGS. 15 and 20, the difference image D (i) is divided into a plurality of blocks.

  Subsequently, in step S6, the parameter j indicating the block number is set to an initial value 1, the parameter k indicating the number of suddenly changing blocks is set to an initial value 0, and the processing from step S7 is executed. First, in step S7, the occupation ratio H of the difference area d (i) in the j-th block is calculated. In step S8, it is determined whether or not the occupation ratio H is equal to or greater than the first threshold value Th1. If the determination is affirmative, the parameter k is increased by 1 in step S9. This means that when the occupation ratio H is equal to or greater than the first threshold Th1, the block is recognized as a sudden change block, and the count value of the sudden change block is increased by one. When the occupation ratio H is less than the first threshold Th1, the value of the parameter k is not increased.

  The processes in steps S7 to S9 are repeatedly executed for all blocks in the i-th difference image D (i). That is, the process proceeds from step S10 to step S11 until the process for all blocks is completed, the parameter j is updated by 1, and the process from step S7 is continued.

  Thus, when the processing for all the blocks is completed, the process proceeds from step S10 to step S12, and it is determined whether or not the value of the parameter k is equal to or greater than the second threshold Th2, and if affirmative, in step S13 Variation detection is performed. That is, when the number of suddenly changing blocks in the i-th difference image D (i) is equal to or greater than the second threshold Th2, the fluctuation detection in step S13 is performed, which is the i-th original image. Immediately before P (i) is photographed, it means that the direction of the camera has changed and that the photographing field of view has been detected to be changed.

  Such processing in steps S2 to S13 is repeatedly executed for the original image P (i) given continuously in time series. That is, until it is determined in step S14 that the process is completed, the process proceeds from step S14 to step S15, the parameter i is updated by 1, and the process from step S2 is continued. The determination of the completion of the process in step S14 can be made unconditionally to complete the process when, for example, the change detection in step S13 is performed. In this case, when the change is detected, the process shown in the flowchart is finished. It is also preferable to complete the processing even when there are no new input original images. In this case, even if no change is detected, when there is no original image to be newly processed (when provision of the captured image from the fixed point camera is interrupted), the processing shown in this flowchart ends.

  In short, the camera field of view variation detection method according to the present invention includes an original image input step of inputting an image captured by the camera as an original image in units of frames continuously given in time series, and a plurality of images input in the past. An image having an average characteristic of the original image is obtained as a background image, and the foreground that is different from the background image for the newly input i-th (i = 1, 2,...) Original image P (i). A foreground region extraction stage for extracting a region and sequentially generating a foreground image F (i) indicating the extracted foreground region, and the generated i th (i = 2, 3,...) Foreground image F (i ) And the (i−1) -th foreground image F (i−1), and a difference region extraction stage that sequentially generates a difference image D (i) indicating the extracted difference region is generated. A block that divides each difference image D (i) into a plurality of blocks. A block division stage, an occupancy ratio calculation stage for calculating an occupancy ratio for each obtained individual block, and a block whose occupancy ratio is equal to or greater than a first threshold is recognized as a sudden change block, A block counting stage for counting the number of suddenly changing blocks for each difference image, and detecting that a change has occurred in the field of view when a difference image having the number of suddenly changing blocks equal to or greater than a second threshold is obtained. The change detection stage may be executed by the computer.

  As described in §3, in the present invention, in order to determine that “the field of view has changed”, the first condition that “the moving amount of the moving object is greater than or equal to a predetermined reference” and “the moving object The second condition is that “the variation of the image is distributed over the entire photographing screen”. In the procedure shown in FIG. 23, in order to determine whether or not the first condition is satisfied, a process of recognizing a block whose difference area occupation ratio H is equal to or greater than the first threshold Th1 as a sudden change block (step S8). ) And a process of determining whether the number of sudden change blocks is equal to or greater than the second threshold Th2 (step S12) is performed to determine whether the second condition is satisfied. become.

  Thus, the first threshold value Th1 is a parameter that affects the first condition, and the second threshold value Th2 is a parameter that affects the second condition. The determination result depends on the result. Therefore, what values are set for the two parameters Th1 and Th2 is very important for improving the accuracy of final determination. The number of block divisions for the difference image is also a parameter that determines the accuracy of the determination.

  However, in reality, whether the fixed-point camera is installed outdoors or indoors, the background in the original shooting field of view is a simple scenery like a beach, a complex scenery like a city, or a moving object that crosses the shooting field of view The number of block divisions, parameter Th1, based on various factors such as whether the speed of the vehicle is about a pedestrian or a vehicle that is traveling at high speed, and the size of the moving object on the screen The optimum value of Th2 varies. Therefore, in practice, it is preferable to appropriately set the number of block divisions and the optimum values of the parameters Th1 and Th2 in consideration of the installation environment and application of each fixed point camera.

  As a general theory, the smaller the total number of block divisions in step S5, the higher the sensitivity for fluctuation detection, and the higher the number, the lower the sensitivity for fluctuation detection. For this reason, if the number of divisions is too small, even if there is a change in only a small part of the area, it will be detected as a change in the shooting field of view, and the passing of a vehicle or the like may be erroneously detected as a change in the shooting field of view. Increases nature. On the other hand, if the number of divisions is increased too much, if a background such as a white wall is included in the background, a detection omission in which the change detection in step S13 is not performed although the shooting visual field change actually occurs. It is more likely to occur.

  On the other hand, as the first threshold Th1 used in the determination in step S8 is set smaller, the detection sensitivity related to the fluctuation amount becomes higher. Therefore, a minute vibration or the like generated when the vehicle passes is erroneously detected as a photographing visual field fluctuation. There is a high possibility that On the other hand, the larger the setting is, the lower the detection sensitivity related to the amount of fluctuation. Therefore, if the direction of the camera does not change to some extent, detection as a photographing field fluctuation is not performed.

  Further, the second threshold Th2 used in the determination in step S12 is set to a smaller value, so that the criterion for the fluctuation distribution becomes lower. Therefore, when a large truck passes, this is erroneously detected as a photographing visual field fluctuation. There is a high possibility that it will end. On the other hand, the larger the setting, the higher the criterion for the fluctuation distribution, so that the detection as the photographing visual field fluctuation is not performed unless the fluctuation distributed over the entire screen is detected.

  According to a general experiment conducted by the inventor of the present application, an image taken with a color video camera having a resolution of horizontal 320 pixels × vertical 240 pixels is used as the original image P (i) input in step S2. A total of 32 blocks are formed by dividing the block in S5 into 8 equal parts in the horizontal direction and 4 equal parts in the vertical direction. The first threshold value Th1 = 50% and the second threshold value Th2 = 10 (all As a result of the setting of 30% of the number of blocks), it was possible to accurately detect photographing field fluctuations for fixed-point cameras in various installation environments. Of course, the specific setting condition is an example experimentally performed by the inventor of the present application, and the setting condition for carrying out the present invention is not limited to the specific setting condition.

<<< §5. Configuration of a photographing visual field fluctuation detection device according to the present invention >>>
Here, the configuration of the photographing visual field fluctuation detection apparatus 100 having a function of executing the fluctuation detection method described in §4 will be described with reference to the block diagram of FIG. The fluctuation detection device 100 is a camera installed for the purpose of continuously taking a picture of the same field of view at a same place for a certain period of time. It is a device for detecting that the orientation has changed due to unforeseen factors.

  As shown in the figure, the fluctuation detection device 100 includes an image input unit 110, a foreground region extraction unit 120, a foreground image storage unit 130, a difference region extraction unit 140, a block division unit 150, an occupation ratio calculation unit 160, a block counting unit 170, The variation determination unit 180 and the determination result notification unit 190 are configured. Hereinafter, the functions of these components will be described in order.

  The image input unit 110 is a component for inputting an image captured by the camera as a frame-unit original image P that is continuously given in time series. For example, when inputting an image from a digital video camera that captures images at a rate of 30 frames per second, a process of sequentially taking frame images that are continuously given at a period of 1/30 seconds as digital data is performed. Here, the i-th frame image is called an original image P (i), and the following description will be given.

  As described above, the fluctuation detection process according to the present invention is executed as an image process by a computer. Therefore, when an image from an analog camera is input, a process of converting the image into a digital image is required at the time of input. When the image captured by the camera is a color image, the original image P (i) is given as pixel aggregate data having pixel values of the three primary colors. Note that the image input unit 110 does not necessarily need to capture an image captured by the camera in real time, and may capture a recorded image that has already been captured as the original image P. In this case, it is not possible to detect in real time the shooting field fluctuation of the camera that has taken the image, but by analyzing the recorded video that has already been shot, the processing to identify when the shooting field fluctuation has occurred is performed. Is possible.

  The foreground area extraction unit 120 extracts a foreground area f (i) from the original image P (i) input in this way, and sequentially generates a foreground image F (i) indicating the extracted foreground area f (i). Do. The foreground image F (i) is composed of a binary image that distinguishes the foreground region f (i) from the other region (background region b (i)). For example, in the example shown in FIG. 1, foreground images F10 to F40 are generated for the original images P10 to P40, respectively.

  In the foreground area f (i) extraction process, an image having an average characteristic of a plurality of original images input in the past is obtained as a background image, and an area different from the background image is determined for the newly input original image. Done by asking. A specific configuration example of the foreground area extraction unit 120 will be described later.

  The foreground image storage unit 130 has a function of sequentially storing the foreground image F (i) generated by the foreground region extraction unit 120. However, since the foreground image F (i) only needs to be temporarily stored until the processing for extracting the difference area is performed, the foreground image storage unit 130 generates the foreground image F (i). It is not necessary to constitute a device (memory) having a storage capacity for storing all of the above. That is, the foreground image storage unit 130 only needs to have a function of storing at least two sets of latest foreground images among the foreground images F (i) generated for the individual original images P (i).

  Therefore, in practice, as shown in the figure, the foreground image storage unit 130 includes a first frame memory 131 that stores the foreground image of the odd-numbered frame and a second frame memory that stores the foreground image of the even-numbered frame. 132, and the foreground region extraction unit 120 may perform processing of alternately rewriting the first frame memory 131 and the second frame memory 132 with the newly generated foreground image.

  Specifically, when the first foreground image F (1) is generated, it is stored in the first frame memory 131, and when the second foreground image F (2) is generated, it is stored in the first frame memory 131. When the third foreground image F (3) is generated in the second frame memory 132 and stored in the first frame memory 131 (F (1) may be rewritten), the second foreground image F (3) is generated. When the fourth foreground image F (4) is generated, it is stored in the second frame memory 132 (F (2) may be rewritten). Good. Then, the foreground image storage unit 130 always stores two sets of the latest foreground images F (i−1) and F (i) adjacent on the time axis.

  The difference area extraction unit 140 extracts the difference area d (i) for the two sets of latest foreground images F (i-1) and F (i) stored in the foreground image storage unit 130, and extracts the difference area A process of sequentially generating a difference image D (i) indicating d (i) is performed. For example, the example shown in FIG. 13 is an example in which a difference image D31 is obtained for two sets of foreground images F30 and F31. As described above, the process for obtaining the difference image can be performed as an operation for obtaining a geometric exclusive OR for two sets of foreground images F (i−1) and F (i).

  The foreground image F generated by the foreground area extraction unit 120 is either a first pixel value (eg, “1”) indicating the foreground area f or a second pixel value (eg, “0”) indicating the background area b. It is given as a binary image composed of a collection of pixels having one of the pixel values. Therefore, the difference area extraction unit 140 reads out a pixel at a predetermined position of one foreground image F (i-1) and a pixel at the predetermined position of the other foreground image F (i), and whether the pixel values of both are the same. A pixel value indicating the difference area d (i) is obtained when a determination is made that the pixels at the predetermined position forming the difference image D (i) are not the same by performing a process for determining whether they are not the same. (For example, “1”), when it is determined that they are the same, a pixel value (for example, “0”) indicating the outside of the difference area is given to each of the difference images D (i) composed of binary images. Should be generated.

  The block dividing unit 150 performs processing for dividing the individual difference images D (i) thus generated into a plurality of blocks. FIG. 15 and FIG. 20 show examples in which the difference image D (i) is divided into 20 blocks in total by dividing the difference image D (i) into 4 parts vertically and 5 parts horizontally. As in the example shown here, if the image input unit 110 inputs an original image P (i) having a rectangular outline, the foreground image F (i) generated by the foreground region extraction unit 120 is also rectangular. The image has an outline, and the difference image D (i) generated by the difference area extraction unit 140 is also an image having a rectangular outline. In the case of such a general example, the block division unit 150 divides the difference image D (i) having the rectangular outline vertically into N and horizontally into M, thereby obtaining N × M rectangles. What is necessary is just to divide | segment into the block of a shape.

  Of course, the original image P handled in the present invention is not necessarily an image having a rectangular outline, and may be, for example, a circular image taken with an omnidirectional camera incorporating a fisheye lens or the like. Also, the block division method does not necessarily need to be divided vertically into N and horizontally into M, for example, it may be divided into concentric circles or radial shapes, or may be divided into regular hexagonal blocks. However, when a general photographed image having a rectangular outline is input as an original image, if it is divided vertically into N and horizontally into M as in the above example, the calculation burden for obtaining the coordinate position of each pixel is calculated. Is reduced.

  The occupation ratio calculation unit 160 performs a process of calculating the occupation ratio H occupied by the difference area d (i) for each obtained block. For example, when the difference area extraction unit 140 generates a difference image D (i) in which the pixel value indicating the difference area d (i) is “1” and the pixel value indicating the outside of the difference area is “0”, For each block, pixels having a pixel value “1” are counted, and a ratio to the total number of pixels in the block may be obtained. As a result, the occupation ratio exemplified as a% value in FIGS. 16 and 21 is calculated.

  Based on the occupation ratio H calculated in this way, the block counting unit 170 recognizes a block whose occupation ratio H is equal to or greater than the first threshold Th1 as a sudden change block, and for each individual difference image D (i), the sudden change block. Count k. In FIG. 16, FIG. 17, FIG. 21, and FIG. 22, blocks indicated by thick frames are suddenly changed blocks, and the block counting unit 170 counts the total number for each difference image D (i).

  When the difference image D (i) in which the number k of sudden change blocks is equal to or greater than the second threshold Th2 is obtained, the fluctuation determination unit 180 obtains the original image P (i) corresponding to the difference image D (i). It is determined that a change has occurred in the field of view at the timing when is taken. This determination result is notified to the outside by the determination result notification unit 190.

  As described above, the variation detection apparatus 100 shown in FIG. 24 sequentially inputs the original images P in units of frames that are continuously given in time series, and the variation is detected based on the difference images D corresponding to the individual original images P. It has a function of executing a process for determining presence or absence and outputting a determination result. In practice, such processing can be realized by incorporating a dedicated program into the computer. The fluctuation detecting apparatus 100 shown in FIG. 24 includes a computer as hardware and software incorporated therein. It can be configured as an organic conjugate. Alternatively, a device that performs the same function can be formed of a semiconductor integrated circuit. In particular, the arithmetic processing performed by each component of the fluctuation detection device 100 shown in FIG. 24 is a processing with a relatively low calculation burden such as addition, multiplication, and comparison calculation. Therefore, even in a semiconductor integrated circuit incorporating a relatively inexpensive CPU. Can be fully configured.

  Note that the number of block divisions when the block dividing unit 150 divides the block, the first threshold Th1 used as a reference for determining whether or not the block is a sudden change block in the block counting unit 170, and the imaging field of view changes in the variation determining unit 180. As described in §4, the second threshold value Th2 used as a criterion for determining whether or not it has occurred is a parameter that affects the determination accuracy. Therefore, an optimum value according to the installation environment of the camera and the like should be set as appropriate. Is preferable. Therefore, it is preferable that the block dividing unit 150, the block counting unit 170, and the fluctuation determining unit 180 are provided with a setting function that can set each parameter value to a desired value by an operator's operation.

  Next, a detailed configuration example of the foreground area extraction unit 120 illustrated in FIG. 24 will be described with reference to FIG. The role of the foreground area extraction unit 120 is to obtain an average image A having an average characteristic of a plurality of original images P input in the past as a background image, and for the newly input original image P, A different foreground area f is extracted, and a foreground image F indicating the extracted foreground area f is generated. In order to perform such processing, the foreground region extraction unit 120 shown in FIG. 25 includes an original image storage unit 121, an average image creation unit 122, an average image storage unit 123, an image comparison unit 124, a parameter setting unit, as illustrated. 125.

  Here, the original image storage unit 121 is a component that sequentially stores the original image P input by the image input unit 110 shown in FIG. On the other hand, the average image creation unit 122 is a component that sequentially creates an average image A having an average characteristic of these original images based on a plurality of original images P input in the past, and the average image storage unit 123. Is a component that sequentially stores the created average image A.

  When the original image P is a color image, the average image creating unit 122 receives the original image P (i) when the i-th original image P (i) is input (i = 1, 2,...). ), The weighted average value of the pixel value for each color of the pixel at the corresponding position is calculated, and the i-th average image A (a set of pixels having the average value) i) is created as a background image and stored in the average image storage unit 123.

  The original image storage unit 121 and the average image storage unit 123 can be configured by a buffer memory. If the average image creating unit 122 sequentially creates the average image A using the weight w as a parameter by the method shown in FIG. 3 and stores it in the average image storage unit 123, the original image storage unit 121 has Since only the latest original image P necessary for processing is always stored, and only the latest average image A necessary for processing is always stored in the average image storage unit 123. Memory capacity can be saved.

Specifically, when the first original image P (1) is input, the average image creating unit 122 uses the original image P (1) as the first average image A (1) as it is, and the average image storage unit 123. After that, every time the i-th original image P (i) is input (i = 2, 3,...), The i-th average image A (i)
a (i) = (1-w) .a (i-1) + w.p (i)
(However, a (i) is the average image A (i)
A pixel value of a predetermined color of a pixel at a predetermined position,
a (i-1) is the average image A (i-1)
A pixel value of the predetermined color of the pixel at the predetermined position;
p (i) is the original image P (i)
A pixel value of the predetermined color of the pixel at the predetermined position;
w is a parameter indicating a predetermined weight (w <1))
It may be created using the following arithmetic expression.

  On the other hand, the image comparison unit 124 compares the average image A stored in the average image storage unit 123 as the background image with the original image P stored in the original image storage unit 121, and compares the difference portion with the foreground region f. This is a component that recognizes the other part as the background area b and generates a foreground image F as a binary image that distinguishes the foreground area f and the background area b. In short, as shown in FIG. 2, when the (i + 1) th original image P (i + 1) is input, the image comparison unit 124 and the original image P (i + 1) and the i-th average image A ( i) and the (i + 1) th foreground image F (i + 1) is generated. Actually, the generated foreground image F is stored in one of the frame memories 131 and 132 in the foreground image storage unit 130 shown in FIG.

  FIG. 25 shows a specific example in which the image comparison unit 124 includes a pixel value reading unit 124A, an similarity determination unit 124B, and a pixel value writing unit 124C. Hereinafter, the functions of these components will be described in order.

  The pixel value reading unit 124 </ b> A may select any one of a pair of images to be compared (that is, the original image P stored in the original image storage unit 121 and the average image A stored in the average image storage unit 123). The pixel value of a pixel at a predetermined position in one image is read as a reference pixel value (R, G, B), and the pixel value of a pixel at a corresponding position in the other image is read as a comparison pixel value (r, g, b). Process.

  As described above, when the model shown in FIG. 8 is adopted, the pixel value of the pixel of the average image A is read as the reference pixel value (R, G, B), and the pixel value of the pixel of the original image P is compared with the comparison pixel value. It will be read out as (r, g, b). On the other hand, when the model shown in FIG. 9 is adopted, on the contrary, the pixel value of the pixel of the original image P is read as the reference pixel value (R, G, B), and the pixel value of the pixel of the average image A is obtained. The comparison pixel value (r, g, b) is read out.

  The similarity determination unit 124B performs a process of determining whether or not the comparison pixel value (r, g, b) is within a predetermined similar range set for the reference pixel value (R, G, B). Specifically, as shown in FIG. 10, in a three-dimensional coordinate system in which the pixel values of the three primary colors are taken on the coordinate axes, a reference point Q (located at a coordinate corresponding to the reference pixel value (R, G, B). R, G, B) and a comparison point q (r, g, b) located at coordinates corresponding to the comparison pixel value (r, g, b), and a spheroid of a predetermined size centered on the reference point Q The positional relationship between E and the comparison point q is examined. When it can be determined that the comparison point q is located outside the spheroid E, it is determined that it is outside the similar range, and when it can be determined that it is located inside, the similar range. The inside is judged. At this time, the size of the spheroid E is determined with reference to the parameters m and n.

  Then, the pixel value writing unit 124 </ b> C responds to the determination result of the similarity determination unit 124 </ b> B as the pixel value of the pixel at a predetermined position (position read by the pixel value reading unit 124 </ b> A) constituting the foreground image F. A value is determined, and a process of writing to one of the frame memories 131 and 132 in the foreground image storage unit 130 shown in FIG. 24 is performed. That is, the pixel value (eg, “0”) indicating the background region b is determined when the similarity determination unit 124B determines that it is within the similar range, and the foreground region when the similarity determination unit 124B determines that it is outside the similarity range. The pixel value (for example, “1”) indicating f is written into the frame memory 131 or 132 in the foreground image storage unit 130, respectively.

  On the other hand, the parameter setting unit 125 uses the parameter w (weight w shown in FIG. 3) used for the average image creation process by the average image creation unit 122 and the parameter m used for the similarity determination process of the similarity determination unit 124B. , N (values for determining the major axis radius α and the minor axis radius β shown in FIG. 10) are set to arbitrary values by user operation input. The user can obtain a detection result with higher accuracy by adjusting the values of these parameters w, m, and n as necessary. However, some or all of these parameters can be fixed values. In an embodiment in which all parameter values are fixed values, the parameter setting unit 125 need not be provided.

<<< §6. Application example of photographing visual field fluctuation detection device according to the present invention >>>
The photographing visual field fluctuation detection device according to the present invention has a function of performing processing for detecting a fluctuation in the photographing visual field of the camera by inputting and analyzing a photographed image by the camera as a frame-unit original image continuously given in time series. It is a device with Here, an embodiment in which this imaging visual field fluctuation detection device is applied to a fixed point monitoring system and an image processing system will be described.

  FIG. 26 is a block diagram showing a configuration example of a fixed point monitoring system according to the present invention. As shown in the figure, this system includes a fixed point camera 10, a monitor device 20, and a fluctuation detection device 100. The fixed-point camera 10 is a camera that outputs a captured image within a predetermined field of view as a frame-unit original image that is continuously given in time series, and is configured by a digital video camera that is used as a general surveillance camera. Can do.

  Of course, this fixed point camera 10 may incorporate a mechanism for controlling the direction. The “fixed-point camera” referred to in the present invention may be any camera that can continuously shoot the same field of view for a certain period of time at the same place, so long as it can be used for the purpose of continuously monitoring a specific location. A camera having a function of adjusting the direction by remote control or the like may be used.

  The monitor device 20 is a device having a function of displaying an image taken by the fixed point camera 10 in real time, and can be configured by, for example, a general liquid crystal display. As described above, the fixed point monitoring system in which the monitor device 20 is combined with the fixed point camera 10 is a system that is very widely used. The fixed point monitoring system shown in FIG. 26 is obtained by adding a fluctuation detecting device 100 to this general system. The fluctuation detection apparatus 100 is an apparatus shown in FIG. 24 and has a function of detecting fluctuations in the field of view of the fixed point camera 10. The frame-by-frame image captured by the fixed point camera 10 is given to the monitor device 20 and also to the variation detection device 100, and the variation in the field of view of the fixed point camera 10 is detected.

  In particular, in the case of the embodiment shown in FIG. 26, when the fluctuation determination unit 180 in the fluctuation detection device 100 determines that a fluctuation has occurred in the shooting field of view, the determination result notification unit 190 in the fluctuation detection device 100 It has a function of issuing a change alarm to the operator. As the fluctuation alarm, for example, an alarm means using a sound that sounds an alarm buzzer or an alarm means using light that lights or blinks a red lamp can be adopted. The operator who is monitoring the monitor device 20 can recognize that an abnormality has occurred in the direction of the fixed point camera 10 by such a variation alarm.

  Of course, if the operator is gazing only at a single monitor screen, it may be possible to immediately recognize the occurrence of an abnormality from the information on the monitor screen when the field of view changes. It is difficult to immediately recognize that an abnormality has occurred in the shooting field of view of one fixed-point camera in an environment where monitoring is performed while displaying captured images from a large number of fixed-point cameras on a large number of monitor screens, such as a security room. It is. Therefore, the fluctuation alarm from the fluctuation detection device 100 is very useful for making the operator recognize the occurrence of an abnormality.

  FIG. 27 is a block diagram showing another configuration example of the fixed point monitoring system according to the present invention. This system further includes an image recording device 30 in addition to the fixed point camera 10, the monitor device 20, and the fluctuation detecting device 100. It is given to the detection device 100. The monitor device 20 is a device having a function of displaying an image photographed by the fixed point camera 10 in real time, and the image recording device 30 is a device having a function of recording a photographed image of the fixed point camera 10. Moreover, the fluctuation | variation detection apparatus 100 is an apparatus shown in FIG. 24, and has the function to detect the fluctuation | variation of the imaging | photography visual field of the fixed point camera 10. FIG.

  In the fixed point monitoring system shown in FIG. 27, when the fluctuation determination unit 180 in the fluctuation detection apparatus 100 determines that a fluctuation has occurred in the photographing field of view, the determination result notification unit 190 in the fluctuation detection apparatus 100 displays an image. The recording apparatus 30 has a function of giving a recording stop signal for stopping recording of the photographed image. The image recording device 30 performs processing (recording processing) for sequentially recording images taken by the fixed point camera 10 in a normal operation state after activation, but when a recording stop signal is given from the fluctuation detection device 100. Therefore, it has a function to cancel the recording process.

  Therefore, for example, when images as shown as original images P10 to P30 in FIG. 18 are taken by the fixed point camera 10, the original images P10 to P30 are recorded by the image recording device 30, but the original image P31 is taken. At that point, the fluctuation detection device 100 detects a change in the field of view and outputs a recording stop signal. Therefore, the recording of subsequent captured images (captured images with an inappropriate field of view) is stopped.

  Of course, as in the embodiment shown in FIG. 26, if the fluctuation detecting device 100 is provided with a function for giving a fluctuation warning to the operator, a fluctuation warning is given to the operator when a recording stop signal is output. Therefore, the operator can recognize that the recording process has been stopped due to the occurrence of an abnormality. If the operator restarts the system after returning the orientation of the fixed point camera 10 to normal, the recording process by the image recording device 30 continues again.

  In a monitoring system used for the purpose of crime prevention, recording is often performed on the image recording device 30 for a long time, and usually, processing for deleting from old image data is performed in order to suppress the recording capacity. As described above, in order to effectively use the limited recording capacity in terms of hardware, it is important to avoid recording useless image data. In the system shown in FIG. 27, when the shooting field of view changes, the recording process is stopped. Therefore, it is possible to prevent a shot image of an inappropriate shooting field of view from being recorded unnecessarily, and limited recording is performed. The capacity can be used effectively.

  Note that if only the captured image is recorded and real-time monitoring is unnecessary, the monitor device 20 in the system shown in FIG. 27 can be omitted.

  FIG. 28 is a block diagram showing a configuration example of an image processing system according to the present invention. As illustrated, this system includes a fixed point camera 10, a monitor device 20, an image processing device 40, and a fluctuation detection device 100. As in the system shown in FIG. 27, the fixed point camera 10 is a camera that outputs a captured image within a predetermined field of view as a frame-unit original image that is continuously given in time series, and the monitor device 20 is a fixed point camera 10. This is a device that has a function of displaying images taken in real time in real time. Moreover, the fluctuation | variation detection apparatus 100 is an apparatus shown in FIG. 24, and has the function to detect the fluctuation | variation of the imaging | photography visual field of the fixed point camera 10. FIG.

  On the other hand, the image processing device 40 is a device that performs predetermined image processing on an original image sequentially given in time series from the fixed point camera 10. As this image processing apparatus 40, for example, a process for recognizing a moving object by a process equivalent to the foreground region extraction process described in §1, a process for recognizing a number when the recognized moving object is a vehicle, When the moving object is a person, a device that performs processing for recognizing a face can be used. The processing result by the image processing device 40 can be displayed on the monitor device 20.

  A feature of this image processing system is that an image captured by the fixed point camera 10 is given to the monitor device 20 and the image processing device 40 and also to the variation detection device 100 to detect variation. When the variation determination unit 180 determines that the imaging field of view has changed, the determination result notification unit 190 in the variation detection device 100 causes the image processing device 40 to stop the image processing. It is a point to give.

  The image processing device 40 is a device that performs image processing based on some algorithm on a captured image on the assumption that the position and orientation of the fixed point camera 10 are fixed and always captures a predetermined imaging field of view. If the given captured image is not an image within the intended field of view, correct processing cannot be performed.

  For example, the fixed point camera 10 installed at the gate of the parking lot captures an image with the vicinity of the vehicle number as the field of view, and the image processing device 40 performs image processing on the captured image to obtain information on the vehicle number. Consider the case where the extraction work is performed. In this case, the image processing apparatus 40 performs the operation of detecting the vehicle number on the premise that the given captured image is an image obtained with the vicinity of the vehicle number as the shooting field of view. Therefore, if the photographing field of view fluctuates due to an unforeseen cause, correct detection work cannot be performed, and an erroneous detection result may be output.

  Even in such a case, in the system shown in FIG. 28, the change in the field of view is detected by the change detection device 100 and a processing stop signal is given to the image processing device 40. It can be prevented that the detection result is canceled and an erroneous detection result is output. As described above, in the image processing system shown in FIG. 28, the image processing device 40 can be stopped when the photographing field of view changes, and inappropriate image processing is prevented from continuing. be able to.

  Subsequently, FIG. 29 shows a modification of the image processing system shown in FIG. The only difference between the image processing system shown in FIG. 28 and the image processing system shown in FIG. 29 is that the fixed point camera 10 in the former is replaced with the image recording device 35 in the latter. That is, the fixed point camera is not included in the image processing system shown in FIG. However, in the image recording device 35, a plurality of original images are recorded in time series. That is, in the image recording device 35, a photographed image within a predetermined field of view photographed by the fixed point camera 10 is recorded as a frame-unit original image that is continuously given in time series.

  The image processing device 40 sequentially reads the original images recorded in the image recording device 35 in chronological order instead of the images given from the fixed point camera 10, and performs predetermined image processing on the read original images. Will do. In short, in the image processing system shown in FIG. 28, an image photographed by the fixed point camera 10 is processed in real time in the image processing device 40, whereas in the image processing system shown in FIG. The captured image is once recorded by the image recording device 35, and the image processing device 40 performs processing on the image recorded in the image recording device 35.

  In the system shown in FIG. 29, the original image read out from the image recording device 35 by the image processing device 40 is also given to the variation detecting device 100, and the variation determining unit 180 in the variation detecting device 100 causes variation in the photographing field of view. When it is determined that the error has occurred, the determination result notification unit 190 in the fluctuation detection device 100 gives a processing stop signal for stopping the image processing to the image processing device 40. Therefore, similarly to the system shown in FIG. 28, the image processing apparatus 40 can be stopped when the photographic field of view changes, and inappropriate image processing can be prevented from continuing. it can.

  In the image processing system shown in FIG. 28 or FIG. 29, the monitor device 20 can be omitted when only image processing is performed and there is no need to perform confirmation on the monitor screen.

  As described above, the fixed point monitoring system and the image processing system are exemplified as application examples of the photographing visual field fluctuation detection device according to the present invention. However, the photographing visual field fluctuation detection device according to the present invention can be used for various other purposes. is there. For example, even if a transmission cable for transmitting captured image data from a fixed point camera or a communication line causes some trouble and the transmission of the captured image is interrupted (on the receiving side, a so-called black image is obtained. The visual field variation is detected by the photographing visual field variation detection device according to the present invention.

  Similarly, even when the transmission cable or communication line is affected by noise and noise is mixed in the transmitted image, the field-of-view variation detection device according to the present invention detects the field variation. Therefore, the photographing visual field fluctuation detection device according to the present invention can be used for general purposes such as abnormality detection of transmission cables and communication lines.

  Even if there is no change in the orientation of the fixed-point camera, for example, a dust bag blown by the wind may cover the lens of the camera, or bird droppings, paint, or other dirt may adhere to the lens. Even in this case, the change in the photographic field of view is detected by the photographic field change detector according to the present invention. As described above, the variation that can be detected by the imaging visual field variation detection device according to the present invention is not necessarily limited to the variation in the orientation of the camera, and part or all of the optical system of the camera is covered with some obstacle. This also includes changes in the field of view taken due to this.

10: Fixed point camera 20: Monitor device 30: Image recording device 35: Image recording device 40: Image processing device 100: Fluctuation detection device 110: Image input unit 120: Foreground region extraction unit 121: Original image storage unit 122: Average image creation Unit 123: Average image storage unit 124: Image comparison unit 124A: Pixel value reading unit 124B: Similarity determination unit 124C: Pixel value writing unit 125: Parameter setting unit 130: Foreground image storage unit 131, 132: Frame memory 140: Difference area extraction unit 150: block division unit 160: occupancy ratio calculation unit 170: block counting unit 180: fluctuation determination unit 190: determination result notification unit A, A (i-1), A (i): average image (background image) )
A, A (Ra, Ga, Ba): Coordinate points a (i−1) and a (i) indicating pixel values of the average image Pixel value of one pixel constituting the average image B: Pixel value of the three primary colors ( Reference pixel value)
B1 to B20: Block b: Pixel values of three primary colors (comparison pixel values)
b, b10 to b40, b (i + 1): background region C: cylinders D31, D40, D (i) of the cylindrical model: difference images d31, d40, d (i): difference regions E, Ea, Ep: spheroids F, F10 to F40, F (i-1), F (i), F (i + 1): Foreground image f, f10 to f40, f (i), f (i + 1): Foreground region G: Three primary color pixel values ( Reference pixel value)
g: Three primary color pixel values (comparison pixel values)
H: Occupancy ratio i: Parameter indicating the image number j: Parameter indicating the block number k: Parameter indicating the number of suddenly changing blocks L: Length La1 of the cylinder C1: Length La2 of the spheroid Ea in the major axis direction: The length Lp1 in the minor axis direction of the spheroid Ea: the length Lp2 in the major axis direction of the spheroid Ep: the length m in the minor axis direction of the spheroid Ep: the major axis radius α of the spheroid E is determined. Parameter n: Parameter defining the minor axis radius β of the spheroid E O: Origin point of RGB coordinate system constituting the three-dimensional color space P: Original image / coordinate points P (1) to P (1) indicating pixel values of the original image i), P (i + 1), P10 to P40: Original image (input image)
P (Rp, Gp, Bp): Coordinate point p (i) indicating the pixel value of the original image Q: Pixel value of one pixel constituting the original image Q: Reference point (with pixel value (R, G, B) Coordinate point)
q: Comparison point (coordinate point with pixel values (r, g, b))
R: Pixel value of three primary colors (reference pixel value)
r: Pixel value of three primary colors (comparison pixel value)
S11 to S15: Steps Th1 and Th2 of the flowchart: Threshold V: Vector Va indicating the reference pixel value: Vector Vp indicating the pixel value of the average image V: Vector indicating the pixel value of the original image w: Parameters Z and Za indicating the weight , Zp: reference axis α: major axis radius of spheroid E: minor axis radius of spheroid E Δ: distance between origin O and reference point Q δ: distance between coordinate points θ: angle

Claims (17)

A fluctuation detection device for detecting fluctuations in the field of view of a camera,
An image input unit for inputting an image captured by the camera as a frame-unit original image continuously given in time series;
An image having an average characteristic of a plurality of original images input in the past is obtained as a background image, a foreground region different from the background image is extracted from the newly input original image, and the extracted foreground region is indicated A foreground region extraction unit for generating a foreground image;
A foreground image storage unit that stores the foreground image generated for each original image;
A difference area extraction unit that extracts a difference area for two sets of foreground images stored in the foreground image storage unit and generates a difference image indicating the extracted difference area;
A block dividing unit that divides each generated difference image into a plurality of blocks;
For each individual block obtained, an occupancy ratio calculation unit that calculates an occupancy ratio occupied by the difference area, and
A block counting unit for recognizing a block in which the occupation ratio is equal to or greater than a first threshold as a sudden change block, and counting the number of sudden change blocks for each individual difference image;
A variation determining unit that determines that a variation has occurred in the field of view when a difference image in which the number of suddenly changing blocks is equal to or greater than a second threshold is obtained;
A determination result notification unit for notifying a determination result by the variation determination unit;
A camera field-of-view variation detection device comprising: The variation detection apparatus according to claim 1,
The foreground area extraction unit
An original image storage unit for sequentially storing the input original images;
Based on a plurality of original images input in the past, an average image creation unit that sequentially creates an average image having average characteristics of these original images;
An average image storage unit for sequentially storing the created average images;
By comparing the average image stored in the average image storage unit as a background image with the original image stored in the original image storage unit and recognizing a different part as the foreground region, An image comparison unit that creates a foreground image as a binary image that distinguishes a background region other than
A camera field-of-view variation detection device comprising: In the fluctuation detection device according to claim 2,
The image input unit inputs a color-unit color original image continuously given in time series as pixel aggregate data having pixel values of the three primary colors,
When the average image creation unit receives the i-th original image P (i) (i = 1, 2,...), The past original image including the original image P (i) A weighted average value of pixel values for each color of pixels at corresponding positions is calculated, and an i-th average image A (i) composed of a collection of pixels having the average value is created as a background image.
When the (i + 1) -th original image P (i + 1) is input, the image comparison unit compares the original image P (i + 1) with the i-th average image A (i), An i + 1) -th foreground image F (i + 1) is generated, and a camera field-of-view variation detector is provided. In the fluctuation detection device according to claim 3,
The average image creation unit
When the first original image P (1) is input, the original image P (1) is stored in the average image storage unit as the first average image A (1) as it is,
Thereafter, every time the i-th original image P (i) is input (i = 2, 3,...), The i-th average image A (i) is
a (i) = (1-w) .a (i-1) + w.p (i)
(However, a (i) is the average image A (i)
A pixel value of a predetermined color of a pixel at a predetermined position,
a (i-1) is the average image A (i-1)
A pixel value of the predetermined color of the pixel at the predetermined position;
p (i) is the original image P (i)
A pixel value of the predetermined color of the pixel at the predetermined position;
w is a parameter indicating a predetermined weight (w <1))
A camera field-of-view variation detection device, characterized by being created using the following equation. In the fluctuation detection device according to claim 3 or 4,
The image comparison unit
A pixel value reading unit that reads a pixel value of a pixel at a predetermined position of one image to be compared as a reference pixel value, and reads a pixel value of the pixel at the predetermined position of the other image to be compared as a comparison pixel value;
A similarity determination unit that determines whether or not the comparison pixel value is within a predetermined similar range set for the reference pixel value;
When the similarity determination unit determines that the pixel value of the pixel at the predetermined position constituting the foreground image is within the similar range, the similarity determination unit determines that the pixel value indicating the background region is out of the similarity range. A pixel value writing unit that writes a pixel value indicating the foreground region to the foreground image storage unit in each case;
A camera field-of-view variation detecting device characterized by comprising: In the fluctuation detection device according to claim 5,
In the three-dimensional coordinate system in which the similarity determination unit takes each pixel value of the three primary colors on each coordinate axis, the reference point Q located at the coordinate corresponding to the reference pixel value and the comparison point q located at the coordinate corresponding to the comparison pixel value And a positional relationship between the comparison point q and a solid having a predetermined size centered on the reference point Q is determined, and if it can be determined that the comparison point q is located outside the solid, it is determined that it is out of the similar range. And a camera field-of-view variation detection device that performs a determination within a similar range when it can be determined that it is located inside. In the fluctuation | variation detection apparatus in any one of Claims 1-6,
The foreground image storage unit includes a first frame memory for storing the foreground image of the odd-numbered frame, and a second frame memory for storing the foreground image of the even-numbered frame;
A camera field-of-view variation detection apparatus, wherein a foreground region extraction unit performs a process of alternately rewriting the first frame memory and the second frame memory with a newly generated foreground image. In the fluctuation | variation detection apparatus in any one of Claims 1-7,
The foreground area extraction unit generates a foreground image as a binary image composed of a collection of pixels having one of the first pixel value indicating the foreground area and the second pixel value indicating the background area. And
The difference area extraction unit reads out a pixel at a predetermined position of one foreground image and a pixel at the predetermined position of the other foreground image, and determines whether the pixel values of both are the same or non-identical. The pixel value indicating the difference area is obtained when the non-identical determination is obtained for the pixel at the predetermined position, and the pixel value indicating the outside of the difference area is obtained when the determination is the same. A camera field-of-view variation detecting device for generating a difference image by giving each of them. In the fluctuation | variation detection apparatus in any one of Claims 1-8,
The image input unit inputs an original image with a rectangular outline,
The foreground area extraction unit generates a foreground image having a rectangular outline,
The difference area extraction unit generates a difference image having a rectangular outline,
The field of view of the camera, wherein the block dividing unit divides the difference image having the rectangular outline into N × M rectangular blocks by dividing the difference image vertically into N and horizontally into M. Fluctuation detection device.   A program for causing a computer to operate as the photographing field fluctuation detection device for a camera according to any one of claims 1 to 9.   A semiconductor integrated circuit which functions as a photographing field fluctuation detection device for a camera according to claim 1.   The fluctuation detection device according to any one of claims 1 to 9, and a fixed point camera that outputs a captured image within a predetermined field of view as a frame-unit original image continuously given in time series, the fixed point A fixed point monitoring system, wherein a change in a field of view of a camera is detected by the change detection device. The fixed point monitoring system according to claim 12,
A fixed point monitoring system, characterized in that, when a fluctuation determination unit in a fluctuation detection device determines that a fluctuation has occurred in the field of view, a determination result notification unit in the fluctuation detection device issues a fluctuation warning to an operator. The fixed point monitoring system according to claim 12 or 13,
It further comprises an image recording device for recording a photographed image of the fixed point camera,
When the change determination unit in the change detection apparatus determines that the change has occurred in the shooting field of view, the determination result notification unit in the change detection apparatus causes the image recording apparatus to stop recording the shot image. A fixed point monitoring system characterized by providing a recording stop signal. The fluctuation detection device according to any one of claims 1 to 9, a fixed-point camera that outputs a captured image within a predetermined field of view as a frame-unit original image that is continuously given in time series, and a time from the fixed-point camera. An image processing device that performs predetermined image processing on original images sequentially given in a series,
When a photographed image by the fixed point camera is given to the variation detection device, and the variation determination unit in the variation detection device determines that variation has occurred in the photographing field of view, a determination result notification unit in the variation detection device Provides a processing stop signal for stopping the image processing to the image processing apparatus. The fluctuation detection device according to any one of claims 1 to 9, an image recording device in which a plurality of original images are recorded in time series, and the original images recorded in the image recording device in order along the time series An image processing apparatus that performs predetermined image processing on the read-out original image,
In the image recording device, a captured image within a predetermined field of view captured by a fixed point camera is recorded as a frame-unit original image that is continuously given in time series,
The original image read from the image recording device by the image processing device is also given to the variation detection device, and when the variation determination unit in the variation detection device determines that a variation has occurred in the shooting field of view, An image processing system, wherein the determination result notifying unit in the fluctuation detecting device gives a processing stop signal for stopping the image processing to the image processing device. A variation detection method for detecting variations in the field of view of a camera,
An original image input stage for inputting an image captured by the camera as an original image in units of frames continuously given in time series,
An image having an average characteristic of a plurality of original images input in the past is obtained as a background image, and the i-th (i = 1, 2,...) Newly input original image P (i) Extracting a foreground area different from the background image and sequentially generating a foreground image F (i) indicating the extracted foreground area;
A difference area is extracted and extracted for the generated i-th (i = 2, 3,...) Foreground image F (i) and (i−1) -th foreground image F (i−1). A differential region extraction stage for sequentially generating a differential image D (i) showing the differential region,
A block division step of dividing each generated difference image D (i) into a plurality of blocks;
For each individual block obtained, an occupancy ratio calculating step for calculating an occupancy ratio occupied by the difference area, and
A block counting step of recognizing a block whose occupancy ratio is equal to or greater than a first threshold as a sudden change block and counting the number of sudden change blocks for each difference image;
A variation detection step of detecting that a variation has occurred in the field of view when a difference image is obtained in which the number of suddenly changing blocks is equal to or greater than a second threshold;
A method for detecting a change in the field of view of a camera, wherein the computer executes the above.

Images