JP4270254B2 - Image signal processing apparatus and image processing method - Google Patents

Description

The present invention relates to an image signal processing apparatus and an image processing method.
More specifically, the present invention relates to a technique suitable for application to a device that detects a suspicious object or a suspicious person from an image obtained by imaging a predetermined monitoring place by a monitoring camera.

2. Description of the Related Art Conventionally, a monitoring system that monitors a predetermined monitoring target by visually recognizing an image captured by a monitoring camera via a predetermined monitoring monitor is known.
In such a monitoring camera, it is difficult for a 24-hour monitor to continue to monitor the monitor as the number of places to be monitored increases. In video recorders connected to surveillance cameras, there is an urgent need to shorten the actual operation time, such as recording only the minimum necessary video.
For this reason, establishment of the technique in which a machine detects a suspicious object or a suspicious person from an input image is calculated | required.
For example, in a bank ATM, the above-described requirements can be satisfied relatively easily if there is a sensor for detecting a human body.
However, such a sensor cannot be used when detecting a suspicious ship or the like from a distant landscape such as the seaside.
JP 2006-14215 A JP-A-10-328226 JP-A-8-297020

Conventionally, in such a situation, a method of performing comparison with the immediately preceding image data as described in Patent Document 1 has been often employed.
When an object enters the video, the brightness of the object portion in the video data changes. Therefore, an object can be detected by detecting a region having a difference in luminance value in the video as a difference region. However, in the case of natural scenery such as the sea, desert, grassland, and sky, water, sand, grass, clouds, etc. other than the object to be detected are also moving. For this reason, there has been a problem that they are erroneously detected as being different from the previous image data.

One method for solving this problem is, for example, the technique described in Patent Document 2.
In Patent Document 2, a difference value between an image taken at the current time and an image taken in the past is created and binarized by threshold processing. At that time, a background image in which the threshold value is changed is created based on the accumulated result of past difference values. In this way, attempts are made to reduce false detection of fluctuations caused by trees and water existing in the background image.
However, with this technique, when the change in the luminance value due to the fluctuation of the tree or the like is large, the case where the threshold value becomes too large can be considered. In such a case, there is a possibility that a detection omission may occur when an important intruder or the like enters.

In particular, in the case of a distant landscape such as the sky including a horizontal line from the seaside, the appearing features differ depending on the location of the image even though the background is the same. For example, when comparing the image data before the sea with the image data near the horizon, the wave splashes occurring on the sea surface appear to be large in front and small in the distance. That is, the sea does not form a uniform pattern as a whole on the captured still image data.
In this way, in the case of distant scenery where the features appearing on the image data differ depending on the location, if processing is performed assuming that all are the same background, the detection performance of suspicious objects is significantly reduced. This is because if the background features are considered to be the same even though they are not uniform, the background features will have a large width.

  The present invention has been made in view of the above points, and from image data obtained by a monitoring camera that monitors a place where a photographed image fluctuates mainly due to a natural phenomenon or the like, such as a sea, a desert, or a grassland. An object of the present invention is to provide an image processing apparatus and an image processing method capable of stably invading a suspicious person or a suspicious object.

  In order to solve the above problems, an image processing apparatus according to the present invention includes an image storage unit that stores input image data, a horizontal region dividing unit that horizontally divides input image data in the image storage unit, and a horizontal region. A feature amount calculation unit that calculates the co-occurrence probability matrix of the horizontal division region obtained from the division unit and the co-occurrence probability matrix obtained by the feature amount calculation unit are compared with a predetermined threshold value to determine whether or not the background is used. And a region determination processing unit including a pixel set determination unit that outputs the presence / absence of a suspicious object or the like based on a determination result obtained by the pixel determination unit.

  According to the present invention, the input image data is divided into a plurality of regions in the horizontal direction. These horizontal division areas are treated as one image data, and a discrimination process between the background and the suspicious object is performed in the area.

  According to the present invention, in a monitoring device that monitors a landscape with a certain fluctuation, such as a sea or a grassland, particularly when monitoring a distant landscape, the characteristics of image data that varies depending on the distance are correctly captured, and the ocean wave It is possible to provide an excellent image processing apparatus and image processing method capable of recognizing a natural phenomenon such as a sky and clouds in the background without erroneous recognition.

Hereinafter, embodiments of the present invention will be described with reference to FIGS.
The embodiments described below are roughly classified into two.
In the first, second, and third embodiments, an input image is subdivided, and a feature amount is calculated for each subdivided small region.
In the fourth, fifth, sixth, and seventh embodiments, a co-occurrence probability matrix is created from an input image, and a background region is specified based on the matrix.
In any of the embodiments, as a superordinate concept, processing is performed by setting a horizontal division area in which the input image is divided into a plurality of parts in the horizontal direction at the beginning of processing, and regarding the horizontal division areas as one input image.

[1. Common concept of embodiment]
FIGS. 1A and 1B are image diagrams illustrating the outline of the operation of the monitoring apparatus according to the embodiment of the present invention.
In FIG. 1A, the input image data 101 includes a sea 102, a sky 103, and a ship 104 as a suspicious object floating in the sea. This sea 102 is a landscape obtained by photographing a distant place, and the actual distance differs greatly between the lower part of the sea part in the input image data 101 and the vicinity of the horizontal line. For this reason, the wave splash on the sea surface appears to be large in the foreground and small in the distance. That is, the sea does not form a uniform pattern as a whole on the captured still image data.
If such input image data 101 is evaluated as a whole, it is difficult to obtain precise results.
Therefore, as shown in FIG. 1B, the input image data 101 is divided into a plurality of parts in the horizontal line direction, and horizontal divided areas 105, 106, 107, 108, 109, and 110 are created.
Whether or not there is a suspicious object is determined for each of the horizontally divided areas of the divided image data.

Block diagrams for realizing this technical idea are shown in FIGS. FIG. 2 is an overall block diagram of the monitoring apparatus, which is a common concept of the embodiment of the present invention. FIG. 2A is a block diagram corresponding to the contents of an actual apparatus, and FIG. 2B is a virtual block diagram focusing on functions.
In FIG. 2A, the monitoring apparatus 201 detects the presence / absence of a suspicious object from image data obtained from the imaging camera 202, and performs a binary signal (alarm output) to notify the presence / absence of a suspicious object or a predetermined process. The applied image signal is output. The imaging camera 202 is a well-known camera that outputs an image signal, which includes, for example, a CCD imaging device. The output image signal is sent to another host device via a large-capacity storage such as an HDD or a communication line.
The monitoring device 201 is mainly composed of a microcomputer. The CPU 203, ROM 204, RAM 205 and bus 206 constituting the microcomputer perform predetermined processing on the image data obtained from the imaging camera 202 and output a predetermined signal or the like through the output interface 207.
The configuration shown in FIG. 2A is common to all the embodiments to be described in detail later.

In FIG. 2B, each unit is a function realized by a software program executed on a microcomputer, except for an image storage unit 212 whose entity is a RAM 205. These software are stored in the ROM 204.
An image signal obtained from the imaging camera 202 is stored in the image storage unit 212 as input still image data.
The horizontal area dividing unit 213 gives horizontal divided areas divided in the horizontal direction to the input still image data in the image storage unit 212.
The region determination processing unit 214 calculates the presence / absence and position, shape, area, etc. of foreign matter for each region of the input still image data in the image storage unit 212 divided in the horizontal direction by the horizontal region dividing unit 213. The result is output to a storage or network as binary alarm or image data.
The configuration in FIG. 2B is a concept common to all the embodiments described in detail later.

FIG. 2C is a flowchart showing the operation of the monitoring device of FIG.
When the monitoring apparatus 201 starts processing (S221), the horizontal area dividing unit 213 divides the input image data 101 into n pieces (n is an integer of 2 or more) (S222). Actually, a relative address range in the RAM 205 in which the input image data 101 is stored is designated by a program.
Next, a counter variable i is initialized to 1 to control loop processing (S223). Then, the region determination processing unit 214 performs region determination processing (such as the above-described determination of the presence or absence of foreign matter) on the i-th horizontal divided region, that is, a part of the horizontally divided input image data (S224).
When the area determination processing of the area determination processing unit 214 is completed, the counter i is incremented (S225), whether or not the counter i exceeds the number n of areas (S226), and the horizontal division area until the counter i exceeds n is reached. An area determination process is performed on
When the area determination process for all horizontal division areas is completed, the process ends (S227).

In the first to seventh embodiments described below, specific processing of the region determination processing unit 214 will be described.
Each embodiment shows good suspicious object detection performance for natural scenery alone without dividing the input image data horizontally.
However, when processing distant scenery as described above, it is not always optimal alone. Therefore, when processing a distant landscape, prior to the various area determination processes described below, areas that are divided horizontally are set for the input image data, and each of these horizontally divided areas is regarded as one image data. The area determination process is performed. Applying such preprocessing contributes to an improvement in suspicious object detection performance in distant scenery.

[2. Feature value]
Prior to the description of the embodiment, a feature amount that is a basis for determination in the first, second, and third embodiments will be described.
The feature amount is an image pattern or texture. It is almost synonymous with what is called “texture” in general drawing software.
In the case of grassland, there are many green shades in a parabolic shape in an oblique direction.
If it is a calm sea surface, there are many white parts such as fine patterns parallel to the horizon that reflects sunlight and white parts such as splashes, based on dark blue.
These natural landscapes do not form a mechanically constant pattern, such as artificially created tiles. However, although it has a random variation, it is recognized that a feature like a certain similar pattern is formed. There is no wave splash on the grassland, and there is no diagonal shading on the calm sea surface.
These natural landscapes move with natural phenomena with time. Therefore, simply comparing a plurality of still images with a time difference as in the prior art, it is misrecognized as a moving object, that is, a suspicious object even though it is essentially a background. .
In the present embodiment, a technique for firmly recognizing the background as the background without being confused by the change on the time axis by capturing the “background texture” with respect to the “moving background” is provided.
Capturing a background texture or background pattern means that it can also capture the characteristics of a foreign object present in the background. This is the calculation of the feature amount.
The input image data is divided into equal fine regions, and the feature amount is calculated for each divided region. Of the calculated feature quantities for each divided area, those indicating values deviating from the background feature quantity are determined to be areas where suspicious objects exist. For this purpose, it is calculated how much the feature quantity of the sample is similar to the feature quantity of the target divided region. This is “distance calculation”. Finally, it is determined whether or not the background is obtained by comparing the obtained distance with a predetermined threshold. This is an extension of the technical idea of calculating the luminance difference between single pixels, which is well known in the art.
The above is a common concept in all the embodiments.
How to obtain the feature amount and how to calculate the distance between the feature amounts will be described later.

[3. First Embodiment]
The first embodiment will now be described. First, an outline of the present embodiment will be described. In the present embodiment, sample image data is given in advance, a background portion is designated manually from the sample image data, and the feature amount is held. After that, when the apparatus is in full operation, it is compared with the feature quantity data of the background sample held for each divided area of the input image data.
FIG. 3 is a block diagram of the monitoring apparatus according to the first embodiment of the present invention. This is based on the content of FIG.
Input still image data obtained from the imaging camera 202 is temporarily stored in the image storage unit 212.
The area dividing unit 313 includes a horizontal area dividing unit 313a and a small area dividing unit 313b. First, after the horizontal area information as a processing unit is given by the horizontal area dividing unit 313a, the small area dividing unit 313b further divides it into rectangular areas of equal shape.
The feature amount calculation unit 314 calculates the feature amount of each divided image data.
On the other hand, the background feature amount storage unit 315 stores the feature amount of the background area specified in advance by the operation of the operator.
The distance calculation unit 316 calculates the distance between the feature amount for each divided region obtained from the feature amount calculation unit 314 and the background feature amount stored in the background feature amount storage unit 315. As a result of the distance calculation by the distance calculation unit 316, it is determined that there is a suspicious object in the divided area determined not to be the background, and an alarm is output.

A display unit 303 and an input unit 304 are connected to the background feature quantity storage unit 315.
The display unit 303 is a known display device such as an LCD.
The input unit 304 is a known pointing device such as a mouse.
The background feature amount storage unit 315 stores a background feature amount as a sample. Prior to the actual operation of this apparatus, input image data as a sample is displayed on the display unit 303 in advance, and a range in which a background is acquired is input by the input unit 304 there. And the feature-value calculation part 314 calculates a feature-value, and preserve | saves this.

4A, 4 </ b> B, and 4 </ b> C are image diagrams illustrating an outline of the operation of the monitoring apparatus according to the first embodiment by exemplifying input image data.
In FIG. 4A, input image data includes image data 401 including a sea 402, a sky 403, and a ship 404 that is a suspicious object. This is stored in the image storage unit 212.
In FIG. 4B, the background feature amount storage unit 315 displays input image data on the display unit 303 by the operation of the operator, and surrounds the background portion with the input unit 304. A range surrounded by a dotted line in FIG. 4A is the background partial region input by the operation of the input unit 304 by the operator. This area is calculated by the feature amount calculation unit 314 and stored in the background feature amount storage unit 315. In FIG. 4A, two background samples B (1) and B (2) are stored in the sky part and the sea part.
In FIG. 4C, the input image data is finely divided by the region dividing unit 313, and the feature amount calculating unit 314 calculates the feature amount for each region. In FIG. 4C, 25 divided areas from S (1) to S (25) are provided.
The distance calculation unit 316 calculates the distance between the background sample and the small divided area. This calculation is performed one by one. That means
Calculate the distance between the background sample B (1) and the subdivision area S (1),
Calculate the distance between the background sample B (2) and the subdivision area S (1),
Calculating the distance between the background sample B (1) and the subdivision S (2);
Calculate the distance between the background sample B (2) and the subdivision area S (2),
In order, and finally calculate the distance between the background sample B (1) and the subdivision S (25),
The distance between the background sample B (2) and the small divided area S (25) is calculated, and the process ends.

The above is the description of calculating the feature amount for the entire screen.
In the present embodiment, the above-described processing is performed after the input image data is divided in advance into a plurality of regions in the horizontal direction. Therefore, the background area must be specified so as to straddle these horizontally divided areas.
FIG. 5 shows a screen displayed on the display unit 303.
A horizontal dividing line 502 is explicitly displayed on the displayed screen. When the user designates the background area, the user sets the background area setting range 503 so that the background designation is not missed in consideration of the horizontal dividing line 502.

FIG. 6 is a flowchart illustrating pre-processing prior to actual operation of the monitoring apparatus according to the first embodiment.
When the processing is started (S601), sample input still image data is taken into the RAM (S602).
Next, the input still image data is displayed on the display unit 303 (S603).
While confirming this visually, the operator operates the input unit 304 to input the address range of the background area (S604).
When the input is confirmed, the designated address range is stored in a RAM or the like (S605).
Then, the pixels in the designated address range are extracted from the sample input still image data, the feature amount is calculated, stored (S606), and the process ends (S607).

7 and 8 are flowcharts showing an actual operation state of the monitoring apparatus according to the first embodiment.
When processing is started (S701), sample input still image data is taken into the RAM (S702).
Next, the horizontal region dividing unit 313a performs horizontal region division, and then the feature amount for each small divided region divided by the small region dividing unit 313b is calculated by the feature amount calculating unit 314 (S703).
The subsequent processing is to calculate the distance between the calculated feature amount for each small region and the background sample in order.
First, the variable i is initialized to 1 (S704). The variable i is a variable that specifies a feature amount for each small area.
Next, the variable j is initialized to 1 (S705). The variable j is a variable that specifies the background sample.
Then, the distance between the background sample B (j) and the divided region feature amount S (i) is calculated (S706).
Next, the distance obtained as a result of the distance calculation is compared with a threshold value separately held in advance (S707). If the distance is equal to or greater than the threshold, the flag variable f (i) is increased to 1 (S708). The flag variable f (i) is a variable provided in the same number as that of the small divided areas. The flag is raised by 1 and the flag is lowered by 0. If the distance is not greater than or equal to the threshold, the flag f (i) is lowered to 0 (S709).

Next, it is verified whether or not the variable j has reached the maximum value (S810). In the case of FIG. 8, since there are two background samples, it is determined whether or not j has reached 2. If j is the maximum value, the calculation of the distances between all background samples and the subdivision area currently being verified is complete, so the variable i is incremented (S811). That is, the process proceeds to the evaluation of the next small divided area. If j is not the maximum value, there is a background sample for which distance calculation still remains, so j is incremented (S812).
After the flag is lowered, the variable i is verified (S813). If it is not the maximum value as a result of the verification, i is incremented (S814). This is because the flag is lowered because the subdivision area is determined to be the background as a result of verification by distance calculation. In the example of FIG. 8, if it is determined that the distance from the sky background sample is empty first, it is not necessary to calculate the distance from the sea background sample next. For this reason, this process is performed in order to skip the distance calculation process with the next background sample.
If the variable i has reached the maximum value, it means that the distance calculation has been completed for all the small divided areas. Therefore, all flag variables f (i) are checked to verify whether there is a variable with a flag raised (S815).
As a result of the verification, if the flag is not raised at all, it can be determined that all the small divided areas are the background, and there is no suspicious object.
On the other hand, as a result of verification, if even one of the flags is raised, it can be determined that the subdivided area is not the background and the suspicious object exists there.
If the presence of a suspicious object is recognized, an alarm is output (S816) and the process is terminated (S817).

In the present embodiment, a background sample is designated manually in advance. For this reason, there is a drawback in that if the background feature value changes, it cannot follow this. For example, in the case of monitoring the sea, if the sea is rough due to the weather or if the imaging camera 202 moves the shooting angle and the arrangement relationship between the sky and the sea changes, a misrecognition will occur.
However, if these features are sufficiently grasped and an appropriate installation is performed for an appropriate monitoring target, the present embodiment can sufficiently exhibit its functions. If the background feature value is a monitoring target that does not change much, the imaging camera 202 is fixed, or the background feature value changes, the background sample may be input again. A method may be used.

  By the way, how to determine the size of the small divided area is slightly different depending on the monitoring target. For example, even if a small divided region that is too small for the background forming a complex pattern is set, it cannot be said that the feature amount can be calculated appropriately. However, when the size of a suspicious object to be detected is small, even if a too small subdivision region is set, the change is less likely to appear in the obtained feature amount. Therefore, it is necessary to change the optimal small divided area according to the monitoring target. For this reason, it is advisable to perform a test operation while changing the subdivision area area on a trial basis in advance and determine the optimum value before actually operating the monitoring device.

Summarize what has been described above.
The monitoring apparatus 301 of this embodiment is mainly composed of software that runs on a microcomputer.
When the input still image data is input to the monitoring apparatus 301, it is stored in the image storage unit 212 (substantially the RAM 205). Next, the horizontal area dividing unit 313a divides the image into horizontal divided areas. And after dividing | segmenting into an equal area shape in the small area division part 313b, the feature-value calculation part 314 produces feature-value data for every small-division area | region. Next, the range of the obtained feature data is set in advance by manual input, the distance from the predetermined value 115 held after the arithmetic processing is calculated, and the obtained “distance” is compared with a predetermined threshold value. . A region having a distance less than the threshold is determined to be the background. Then, the flag indicating the small divided area determined to be the background is lowered.
The above processing is performed for each horizontal division region created by the horizontal region division unit 313a.
Through the above processing, it is possible to discriminate between the background area and the other area areas from the input still image data through the feature amount calculation and the distance calculation.

[4. Feature value calculation method]
Here, a method of calculating the feature amount will be described.
The feature value is obtained by reading image data in a certain range and digitizing features appearing in the image data.
Various features of image data can be considered. Wide range of chromaticity, saturation, and pattern. Therefore, it cannot be a single scalar quantity, and is represented by a matrix composed of many elements.
FIGS. 9A, 9B, and 9C illustrate a color histogram, which is one of feature amount calculation methods that can be commonly used in the first, second, and third embodiments described in the present invention. FIG.
FIG. 9A shows a sample of input image data. In the input image data 901, the sea 902 spreads out as a whole, and a ship 903 that is a suspicious object exists therein. Suspicious objects consist of black pixels.
FIG. 9B shows an enlarged view of the sea portion of the input image data sample shown in FIG. Waves 904 are generated in the sea 902. The seawater 905 portion is a blue pixel, and the wave splash 904 portion is a white pixel.
FIG. 9C shows a matrix of all color components. Each pixel existing in the input image data in FIG. 9A is added to the elements corresponding to the matrix 906 one by one. Then, the accumulated points are concentrated on a white area 907 that expresses splashing water, a blue area 908 that expresses seawater, and a black area 909 that expresses a ship that is a suspicious object. In other words, you should think that a curved surface such as a well-known Gaussian surface is formed at these three locations.
As described above, the color histogram is obtained by counting the color appearance frequency for each pixel in the color image data. You may think that the color particles that make up the image are scattered and collected for the same pigment. For this reason, the pattern concept is lost in the generated data.

FIGS. 10A, 10B, 10C, and 10D are one of feature amount calculation methods that can be commonly used in the first, second, and third embodiments described in the present invention. It is the schematic explaining frequency analysis.
FIG. 10A shows a part 1001 of the input image data. This is the same as that shown in FIG. 9 (b), and shows an enlarged portion of the sea portion of the sample of input image data. FIG. 10B shows a part of the splash by further enlarging the part. Pixels corresponding to wave splash are bright and pixels corresponding to seawater are dark. The luminance of each pixel 1002 is acquired.
FIG. 10C shows an example in which the brightness of each pixel constituting the input image data is plotted in order from the left side to the right side. It can be seen that the portion corresponding to seawater has low luminance, and the portion corresponding to wave splash has high luminance. Then, the frequency analysis is to treat the figure obtained in this way as if it were a signal and analyze the frequency component.
FIG. 10D shows an example of the result of the frequency analysis. The waveform plotted in FIG. 10C is subjected to Fourier analysis, and the horizontal axis represents frequency and the vertical axis represents level. Finally, the data obtained in this way is used as a matrix. That is, it is a matrix composed of frequency components of the waveform of the luminance of the image data.
As described above, the frequency analysis is a Fourier transform for each pixel in the image data in order to analyze the frequency component by regarding the change in luminance as a waveform. For this reason, a shading pattern, in turn, a pattern of a predetermined pattern is expressed as a frequency component.

FIG. 11 is a diagram for explaining the outline of the co-occurrence probability matrix.
In FIG. 11A, P1 and P2 are any two pixels in the image data. The pixel P1 is separated from the pixel P2 by a distance r and an angle θ. Let r and θ be the relative position function δ. δ = (r, θ).
In FIG. 11B, when δ1 = (1,0 °), the pixel P2 indicates the next adjacent pixel with respect to the pixel P1.
In FIG. 11C, when δ1 = (1, 90 °), the pixel P2 indicates the adjacent pixel immediately above the pixel P1.
The co-occurrence probability matrix is a mechanism that adds a combination of the luminances of two pixels separated by a certain relative position function δ to a square matrix having luminance as the number of elements.

FIG. 12 shows simplified image data and a simplified co-occurrence probability matrix for explanation.
In FIG. 12A, each pixel in the image data composed of 4 × 4 16 pixels has four levels of brightness from 0 to 3.
FIG. 12B is a co-occurrence probability matrix that is a 4 × 4 square matrix having four levels of luminance. Here, a procedure for creating a co-occurrence probability matrix from the image data of FIG. 12A in the relationship of relative position function δ = (r, θ) = (1,0) is shown below.
If (x, y) = (0, 0) in FIG. 12A is the pixel P1, the pixel P2 is (x, y) = (1, 0). At this time, the luminance i of the pixel P1 is 0, and the luminance j of the pixel P2 is 0. Therefore, 1 is added to (i, j) = (0,0). Furthermore, since round trip counting is performed, 1 is added to (i, j) = (0, 0).
If (x, y) = (1, 0) in FIG. 12A is the pixel P1, the pixel P2 is (x, y) = (2, 0). At this time, the luminance i of the pixel P1 is 0, and the luminance j of the pixel P2 is 1. Therefore, 1 is added to (i, j) = (0,1). Furthermore, since round trip counting is performed, 1 is added to (i, j) = (1, 0).
As described above, the operation of counting up the elements of the matrix corresponding to the combination of the luminance of each pixel separated by the relative position function δ is performed for all the pixels. That is, this square matrix is a matrix of counters indicating the number of appearances of the combination of the luminance values of two pixels.
Note that it is up to the designer to perform a round-trip count.

  Due to the nature of the co-occurrence probability matrix, it is desirable to take a plurality of relative position functions and generate a plurality of matrices for each. This is because the pattern constituting the image is not always horizontal or vertical. It is desirable to take at least the vertical and horizontal relative position functions.

By these feature amount calculation methods, feature amounts composed of a matrix composed of a large number of numerical data are created for each divided region of the input image data.
A “distance” is calculated by comparing the obtained feature quantity with a predetermined sample value or other feature quantity held in advance. This calculation is called feature amount distance calculation, which will be described later.
Note that the designer determines which of the feature quantity calculation methods listed above is adopted based on a trade-off between the accuracy of the obtained feature quantity and the calculation quantity.

[5. Feature distance calculation method]
With the feature amount calculation method described above, a feature amount is created for each divided region of the input image data. These are all matrices consisting of a lot of numerical data.
How to compare matrix data Here, the value to be obtained is the similarity between the two matrix data. In the first embodiment described above, the matrix data constituting the feature amount of the sample background data and the matrix data constituting the feature amount of the divided area.
Various methods for obtaining the similarity of matrix data are conventionally known. Examples are listed below.
(1) A method of taking the difference for each element of the matrix and taking the sum of each element of the obtained matrix (SAD: Sum of Absolute Difference).
(2) A method of taking a difference for each element of the matrix, squaring each element of the obtained matrix, and then summing each element (SSD: Sum of Squared Difference).
(3) A method of taking each element of a matrix as a vector component and taking an inner product of the matrices. That is, a method of taking the product of each element of the matrix and taking the sum of each element of the obtained matrix (inner product: Normalization).

By the comparison operation of these feature amounts, a “distance” consisting of a single scalar value is finally obtained.
A predetermined determination is made by comparing the obtained distance with a predetermined threshold that is held in advance.
Note that the designer determines which of the feature quantity distance calculation methods listed as examples is based on a trade-off between the accuracy of the obtained distance and the calculation amount.

  The feature amount calculation method and the feature amount distance calculation method described above are technologies that can be selected and used in common in the first embodiment described above and in the second and third embodiments described later. That is, in any embodiment, any of color histogram, frequency analysis, and co-occurrence probability matrix may be adopted as the feature amount calculation method, and any of SAD, SSD, and inner product is adopted as the feature amount distance calculation method. May be.

[6. Second Embodiment]
FIG. 13 is a block diagram of a monitoring apparatus according to the second embodiment of the present invention. This is based on the content of FIG. In addition, about the part which is common in FIG. 3 explaining 1st Embodiment, description is omitted in order to avoid duplication.
The second embodiment is different from the first embodiment in that an average value storage unit 1302 is provided instead of the background feature amount storage unit 202.
The average value storage unit 1302 stores the average value of the feature amounts for each subdivision area obtained from the feature amount calculation unit 314.
The distance calculation unit 316 calculates the distance between the feature amount for each subdivision area obtained from the feature amount calculation unit 314 and the average value stored in the average value storage unit 1302. As a result of the distance calculation by the distance calculation unit 316, it is determined that there is a suspicious object in the divided area determined not to be the background, and an alarm is output.

FIGS. 14A, 14B, and 14C are image diagrams illustrating an outline of the operation of the monitoring apparatus according to the second embodiment, illustrating input image data.
In FIG. 14A, input image data includes image data 1201 including a sea 1402 and a ship 1404 that is a suspicious object.
In FIG. 14B, the average value storage unit 1302 stores the average value of the feature amounts S (1) to S (25) for each small divided region calculated by the feature amount calculation unit 314. . That is, it is a value obtained by dividing the sum of the feature amounts of all the divided areas by the number of divided areas.
The distance calculation unit 316 calculates the distance between the average value and the divided area. This distance calculation is performed one by one. That means
Calculate the distance between the average value B and the divided area S (1),
Calculate the distance between the average value B and the divided area S (2),
In order, and finally,
The distance between the average value B and the divided area S (25) is calculated, and the process ends.

15 and 16 are flowcharts showing the operation of the monitoring apparatus according to the second embodiment.
When processing is started (S1501), sample input still image data is taken into the RAM (S1502).
Next, the horizontal region division unit 313a performs horizontal region division, and then the feature amount for each small divided region divided by the small region division unit 313b is calculated by the feature amount calculation unit 314 (S1503).
Then, an average value is calculated from the feature values for all the obtained small divided regions (S1504).
From this point onward, the calculated distance between the feature value and the average value for each subdivision area is calculated in order.
First, the variable i is initialized to 1 (S1505). The variable i is a variable that specifies the feature amount for each subdivision area.
Then, the distance between the average value B and the small divided region feature amount S (i) is calculated (S1506).
Next, the result of the distance calculation is compared with a threshold value separately held in advance (S1507). If the distance is equal to or greater than the threshold, the flag variable f (i) is increased to 1 (S1508). The flag variable f (i) is a variable provided in the same number as that of the small divided areas. The flag is raised by 1 and the flag is lowered by 0.

Next, the variable i is verified (S1609). If it is not the maximum value as a result of the verification, i is incremented (S1610). If the variable i has reached the maximum value, it means that the distance calculation has been completed for all the small divided areas. Therefore, all flag variables f (i) are checked to verify whether there is a variable with a flag raised (S1611).
As a result of the verification, if the flag is not raised at all, it can be determined that all the divided areas are the background and there is no suspicious object.
On the other hand, as a result of verification, if even one of the flags is raised, it can be determined that the subdivided area is not the background and the suspicious object exists there.
If the presence of a suspicious object is recognized, an alarm is output (S1612) and the process ends (S1613).

Summarize what has been described above.
The monitoring device 1301 of this embodiment is mainly composed of software that runs on a microcomputer.
When the input still image data is input to the monitoring device 1301, it is stored in the image storage unit 212 (substantially the RAM 205). Next, the horizontal area dividing unit 313a divides the image into horizontal divided areas. And after dividing | segmenting into an equal area shape in the small area division part 313b, the feature-value calculation part 314 produces feature-value data for every small-division area | region. Thereafter, the average of the feature amount data of all the divided areas is calculated and stored in the average value holding unit 1302 (the entity is the RAM 205). Next, a distance between the obtained feature amount data and the average value in the average value holding unit 1302 is calculated, and the distance obtained as a result is compared with a predetermined threshold value. A region having a distance less than the threshold is determined to be the background. Then, the flag indicating the divided area determined to be the background is lowered.
The above processing is performed for each horizontal division region created by the horizontal region division unit 313a.
Through the above processing, it is possible to discriminate between the background area and the other area areas from the input still image data through the feature amount calculation and the distance calculation.

[7. Third Embodiment]
FIGS. 17A and 17B are block diagrams of a monitoring apparatus according to the third embodiment of the present invention. This is based on the content of FIG. In addition, about the part which is common in FIG. 13 explaining 2nd Embodiment, it abbreviate | omits description in order to avoid duplication.
In FIG. 17A, the third embodiment is different from the second embodiment in that a clustering processing unit 1703 is provided between the feature amount calculation unit 314 and the comparison unit 1704. The clustering processing unit 1703 and the threshold 1702 are compared.
The clustering processing unit 1703 calculates the distance between the feature amounts for each subdivision area obtained from the feature quantity calculation unit 314, and subdivision areas that can be determined to be close to each other, that is, subdivision areas that can be determined to be similar to each other. A process called “clustering” is performed in which images are considered to be similar and grouped. By this processing, each similar subdivided region is classified as a set belonging to the same image portion.
The comparison unit 1704 compares the number of elements of each subdivision region set obtained from the clustering processing unit 1703 with a threshold 1702. As a result of the comparison by the comparison unit 1704, a set of small divided areas determined not to be the background is determined to have a suspicious object, and an alarm is output.

FIG. 17B is a block diagram showing an internal configuration of the clustering processing unit 1703 in FIG.
The distance calculation target setting unit 1713 determines which feature amounts of the feature amount 1712 obtained from the feature amount calculation unit 314 are to be distance calculation targets.
The distance calculation unit 1714 calculates the distance between the two feature amounts selected by the distance calculation target setting unit 1713.
The comparison unit 1716 compares the distance obtained from the distance calculation unit 1714 with a threshold value 1715. This comparison result leads to the determination of whether or not the two subdivided areas are merged.
The grouping processing unit 1717 merges the two subdivided regions according to the comparison result of the comparison unit 1716, or leaves them as they are without being merged.
As described above, through a series of processes, the feature amounts for each small divided region are classified into groups of several similar feature amounts. After this processing, the data representing the number of groups of feature values is compared with a threshold value 1702 by the comparison unit 1704, leading to a determination as to whether or not it is the background.

FIGS. 18A, 18B, 18C, and 18D are image diagrams illustrating an outline of the operation of the monitoring apparatus according to the third embodiment, illustrating input image data. In FIG. 18, for convenience of explanation, a case where horizontal region division is not performed will be described.
As input image data, there is image data 1801 including the sea and a ship that is a suspicious object.
The clustering processing unit 1703 compares all the feature amounts S (1) to S (25) for each divided region calculated by the feature amount calculating unit 314, and performs a process of collecting similar divided regions.
First, the distance calculation unit 1714 calculates the distance between the upper left region S (1) of the region obtained by dividing the image data 1801 in FIG. 18A and the adjacent region S (2). The distance between the two feature amounts obtained as a result of the distance calculation is compared with a threshold 1702 prepared in advance, and it is determined whether or not the divided areas are similar to each other. In FIG. 18 (a), since the regions S (1) and S (2) are both sea portions, their feature amounts are similar, so the distance is close. Therefore, they are considered to be the same pattern and are combined into one. This is the process shown in FIG. That is, it is a concept of erasing the boundary line by combining the areas S (1) and S (2).
From FIG. 18B, the calculation of the distance between adjacent regions and the similarity determination by comparing the obtained distance with the threshold value 1702 are advanced. That is, as shown in FIG. 18C, the boundary lines of similar image segmented areas are erased one after another.
Finally, as shown in FIG. 18 (d), the divided regions S (12), S (13), and S (14) whose feature amounts are significantly different from those of other divided regions and that are recognized to be long are left. . This is an area where suspicious objects exist.

FIG. 19 is a flowchart illustrating the operation of the monitoring apparatus according to the third embodiment.
When processing is started (S1901), input still image data as a sample is taken into the RAM (S1902).
Next, the horizontal region division unit 313a performs horizontal region division, and then the feature amount for each small divided region divided by the small region division unit 313b is calculated by the feature amount calculation unit 314 (S1903).
Then, clustering processing is performed based on the obtained feature quantities for each of the small divided regions (S1904). As a result of the clustering process, the small divided areas are grouped into similar areas.
Next, the area area, that is, the number of subdivided areas constituting a similar area is compared with the threshold 1702, and it is verified whether there is a narrow area that cannot be recognized as a background (S1905).
As a result of the verification, if the number of small divided areas constituting all the areas is equal to or greater than the threshold value 1702, it can be determined that all the small divided areas are the background and there is no suspicious object.
On the other hand, as a result of verification, if the number of small divided areas constituting a specific area is less than the threshold value 1702, it can be determined that the area is not a background and a suspicious object exists there.
If the presence of a suspicious object is recognized, an alarm is output (S1906) and the process ends (S1907).

Summarize what has been described above.
The monitoring device 1501 of this embodiment is mainly composed of software that runs on a microcomputer.
When the input still image data is input to the monitoring apparatus 1501, it is stored in the image storage unit 212 (substantially the RAM 205). Next, the horizontal area dividing unit 313a divides the image into horizontal divided areas. And after dividing | segmenting into an equal area shape in the small area division part 313b, the feature-value calculation part 314 produces feature-value data for every small-division area | region.
Thereafter, the clustering processing unit 1703 calculates the distance between the feature amount data of all the small divided areas, determines that the obtained small divided areas are the same pattern areas, and integrates them. Perform clustering processing. In this way, a plurality of subdivided areas are classified into a collection of several areas.
Next, the number of subdivided areas constituting the aggregate of a plurality of classified subdivided areas is compared with a predetermined threshold 1702. A set consisting of a number of small divided areas equal to or greater than the threshold value 1702 is determined to be the background. Then, the flag indicating the divided area determined to be the background is lowered.
The above processing is performed for each horizontal division region created by the horizontal region division unit 313a.
Through the above processing, it is possible to discriminate between the background area and the other area areas from the input still image data through the feature amount calculation and the comparison calculation.

[8. Fourth Embodiment]
FIG. 20 is an overall block diagram of an image processing apparatus according to the fourth embodiment of the present invention.
In FIG. 20, each unit is a software program executed on a microcomputer except for an image storage unit 212 whose entity is a RAM 205. These software are stored in the ROM 204.
The image signal obtained from the imaging camera 202 is once held in the image storage unit 212 as a still image.
The horizontal area dividing unit 213 gives horizontal divided areas divided in the horizontal direction to the input still image data in the image storage unit 212.
The feature amount calculation unit 2013 obtains still image data in the image storage unit 212 and information on the horizontal division region obtained from the horizontal region division unit 213, and creates feature amount data for each horizontal division region. This is array data composed of relative addresses of image data created by a co-occurrence probability matrix described later.
The pixel determination unit 2014 creates bitmap data that identifies a suspicious object in still image data based on the array data created by the feature amount calculation unit 2013.
The pixel set determination unit 2015 calculates the total area of the image portion that seems to be a suspicious object in the still image data and the center coordinates from the created bitmap data.

Note that the image processing apparatus 2001 according to the present embodiment is made for the purpose of being installed mainly on the coast along with the imaging camera 202. The imaging camera 202 is reciprocally driven so as to overlook the coastal sea area by a driving mechanism such as a motor (not shown).
To detect suspicious objects on the seaside, you have to increase the magnification of the camera. Increasing the magnification of the camera means that the shooting angle becomes smaller, and for this reason, it is necessary to drive the camera back and forth.
Reciprocating the camera is difficult to compare the previous still image with the current still image and find a suspicious object or the like from the difference between the image data. The invention according to the present embodiment is based on such a background situation.

FIG. 21 schematically illustrates the inside of the RAM 205. Inside the RAM 205, there are an input image bitmap data storage area 2102 that can be called an image storage unit 212, an array data area 2103 for a co-occurrence probability matrix, and an image flag bitmap data area 2104.
FIG. 22 illustrates data stored in the RAM 205.
FIG. 22A shows input image bitmap data stored in the input image bitmap data storage area 2102 of the RAM 205. That is, it is still image data itself captured by the imaging camera 202. Each pixel is data representing luminance from 0 to 255, for example. In FIG. 22A, a sea 2202, a sky 2203, and a ship 2204 that is a suspicious object are shown.
FIG. 22B shows image flag bitmap data stored in the image flag bitmap data area 2104 of the RAM 205. In the processing described later, the suspicious object region is detected from the input image bitmap data. In this figure, a bit in an area corresponding to the ship 2204 is set.
FIG. 22C is an enlarged view of a part of the image flag bitmap data. Each pixel consists of 1 bit. If the bit is set (if the bit value is 1), it indicates a suspicious pixel, and if the bit is down (if the bit value is 0) ), Which shows the pixel that seems to be the background.

FIG. 23 schematically shows array data stored in the array data area 2103 of the RAM 205. Actually, the contents of the RAM 205 do not have such a shape, but it should be considered that it is virtually constructed in such a shape by a software program.
In FIG. 23A, array data 2301 is a collection of variable-length array data based on a co-occurrence probability matrix. The co-occurrence probability matrix is a square matrix composed of 256 elements each having a vertical k and a horizontal l of 0 to 255. The vertical k and horizontal l correspond to the luminance of the image data. A cube is stacked on each element on this matrix. The cube stores a relative address indicating one pixel of image data.

(K, l) [1] in FIG. 23A is the first array element of the array data including the matrix elements of the l-th column in the k-th row.
Similarly, (k, l) [2] is the second array element of array data composed of matrix elements of k rows and 1 columns.
The array data (k, l) has five array elements and exists up to (k, l) [5].
(K, l−1) [1] in FIG. 23A is the first array element of the array data including the matrix elements of the (l−1) th column in the kth row.
Similarly, (k, 1-5) [1] is the first array element of array data composed of matrix elements of k rows and 1-5 columns.
Similarly, (k−3, l) [1] is the first array element of array data composed of matrix elements of k−3 rows and l columns.
In the cubes stacked in this way, a relative address indicating one pixel of the image data stored in the input image bitmap data area 2102 is stored.

FIG. 23 (a) shows only a part of the array data that is piled up steadily due to space limitations. However, it can be estimated that the shape is close to a well-known Gaussian curved surface as shown in FIG.
A detailed method for creating array data based on the co-occurrence probability matrix will be described later.

FIG. 24 is a diagram schematically showing the relationship between the RAM 205 and each data stored in the RAM 205.
A background portion indicating a large area in the input image bitmap data area 2102 is concentrated in a portion having many elements in the array data area 2103 constituting the co-occurrence probability matrix. The relative address of the image data stored as the array element of the concentrated portion corresponds to the relative address of the background portion of the image flag bitmap data area 2104, and the flag at that location is lowered.
In FIG. 24, image data corresponding to a part surrounded by a dotted line, that is, an empty part of an image showing a part of the input image bitmap data area 2102, is placed in each of the stacked cubes in the array data area 2103. It corresponds. In other words, the array addresses forming these cubes store the relative addresses of the pixels surrounded by the dotted line portion in the image data as elements. These relative addresses lower the flag of the corresponding part of the image flag bitmap data area 2104 from 1 to 0.

First, array data is created from the input image bitmap data in FIG. 24 by the method of creating a co-occurrence probability matrix described in FIGS.
However, this embodiment takes an approach different from the conventionally known co-occurrence probability matrix handling.
In the prior art, the co-occurrence probability matrix is used as a method for comparing non-defective products with defective products during the manufacturing process, as disclosed in Patent Document 3, for example. A good product is photographed in advance and a co-occurrence probability matrix is stored, and a product flowing into the production line is photographed. A co-occurrence probability matrix is created from the photographed image data, and compared with a non-defective product matrix that holds this, abnormalities such as product scratches and dirt are discriminated from the differences. This is the use of a co-occurrence probability matrix that has been well known.
In the present embodiment, each matrix element is compared with this co-occurrence probability matrix using a predetermined threshold value. Then, it is determined that the matrix elements equal to or greater than the threshold value are regions that occupy a large area in the image data, that is, the background. Then, an area in the image data corresponding to the background is excluded.
For this purpose, it is necessary to use matrix elements as array data and store the relative addresses of the pixels in the image data in the array elements.
After all the processes are completed, it is determined that the portion where the flag is not lowered in the image flag data is a narrow portion showing luminance different from the background in the image data, that is, a region indicating a suspicious object.
As an image, it can be considered that the Gaussian curved surface in FIG. 23B is cut right by a plane parallel to the xy plane indicating the threshold value. The relative address of the image data corresponding to the matrix element cut by the threshold plane, that is, touching the threshold plane is the background.

25 and 26 are flowcharts showing the operation of the image processing apparatus 2001 of the present embodiment.
When the process is started (S2501), a relative position function δ, which is a rule for creating a co-occurrence probability matrix for the first time, is set (S2502). Next, the examination target address a of the input image bitmap data is set at the head of the image data (S2503). That is, the address a is a relative address in the input image bitmap data, and an initial value of the relative address is given in this process.
Next, it is verified whether or not a pixel exists at a point separated from the relative address a by the relative position function δ (S2504). For example, when the relative position function δ = (1,0 °), if the relative address a is the right end of the input image bitmap data, the relative address b = δ (a) does not exist.
If there is a relative address b = δ (a), the luminance of the relative address a is substituted for k, and the luminance of the relative address b is substituted for l. Then, the relative address a is added to the array of matrix elements (k, l) in the array data area 203 (S2505). That is, one cube shown in FIG. 23A is stacked and the relative address a is stored therein. If there is no b = δ (a) in step S2504, the process of step S2505 is skipped because the co-occurrence probability matrix cannot be added.
Next, the relative address a is advanced by one (S2506). Next, it is verified whether or not the relative address a exists (S2507). If there is, the process is repeated again from step S2504. If not, it is the end of the input image bitmap data, and since the processing of the co-occurrence probability matrix is finished, the process proceeds to the next process.

When the co-occurrence probability matrix processing is completed for all input image bitmap data, the threshold value is calculated using the matrix elements (S2508). There are various methods for calculating the threshold value. For example, it is conceivable to use a preset fixed value or to average all of one or more matrix elements.
When the threshold value is obtained, initial values of matrix elements to be evaluated in the array data area 203 are set (S2609).
Next, the number of matrix elements is compared with a threshold value (S2610). If the number of elements is equal to or greater than the threshold value, the flag of the corresponding address in the image flag bitmap data area 2104 corresponding to the relative address of the input image bitmap data in the array element belonging to the matrix element is lowered (S2611). . In other words, it is a judgment that there is a background. If the number of elements is less than the threshold, nothing is done. In other words, it is a judgment that there is a suspicious object.
Then, the matrix element to be evaluated is advanced by one (S2612).
Next, it is verified whether there is a matrix element to be evaluated (S2613). If there is, the evaluation is performed again (S2610). If not, it means the end of processing.
Finally, the area and central coordinates of the flagged area remaining in the image flag bitmap data area 2104, that is, the area indicating the suspicious object are calculated (S2614), and the process ends (S2615).

In FIG. 25, the processing composed of steps S2502, S2503, S2504, S2505, S2506, and S2507 corresponds to the processing of the feature amount calculation unit 2013 in FIG. That is, it is a calculation process of a co-occurrence probability matrix.
In FIG. 25, step S2508, and in FIG. 26, the process consisting of steps S2609, S2610, S2611, S2612, and S2613 corresponds to the process of the pixel determining unit 2014 in FIG. That is, it is a process for determining the address of the suspicious object.
In FIG. 26, the process consisting of step S2614 corresponds to the process of the pixel set determining unit 2015 in FIG. That is, this is a process of counting the number of suspicious object addresses and calculating the center of the area indicating the suspicious object address.

[9. Fifth Embodiment]
Next, a fifth embodiment of the present invention will be described.
FIG. 27 is a schematic diagram illustrating a relative position function δ used in the image processing apparatus according to the fifth embodiment.
In this embodiment, the internal configuration of the image processing apparatus is almost the same as the configuration of the block diagram of FIG. 20 described in the fourth embodiment, but a plurality of relative position functions δ are taken and a plurality of In this configuration, an occurrence probability matrix is created.
In the fourth embodiment described above, the relative position function δ = (1,0 °) has been described as an example. By the way, it is well known in the prior art that a co-occurrence probability matrix is obtained by a plurality of relative position functions δ. This is a measure for more accurately quantifying the characteristics of image data having a certain pattern.
In FIG. 27, it can be seen that there are eight pixels even in the vicinity of a certain pixel.
δ1 has a distance of 1 and an angle of 0 °.
δ2 has a distance of √2 and an angle of 45 °.
Similarly, δ3 = (1,90 °), δ4 = (√2,135 °), δ5 = (1,180 °), δ6 = (√2,225 °), δ7 = (1,270 °) Δ8 = (√2,315 °).
That is, eight relative position functions δ1 to δ8 can be created.
Using these eight relative position functions δ1 to δ8, eight co-occurrence probability matrices can be created.

FIG. 28 is a flowchart of the image processing apparatus according to the present embodiment.
When the process is started (S2801), the counter variable i is first initialized to 0 (S2802). Next, the existence of the relative position function δ (i), which is a rule for creating the co-occurrence probability matrix, is confirmed (S2803).
If there is δ (i), it is set (S2804), and based on this, a feature amount calculation process based on the co-occurrence probability matrix is performed (S2805). Next, pixel determination processing is performed using the obtained array data of the co-occurrence probability matrix (S2806).
Here, step S2805 is equivalent to steps S2503, S2504, S2505, S2506, and S2507 in FIG.
Step S2806 is equivalent to step S2508 in FIG. 25 and steps S2609, S2610, S2611, S2612, and S2613 in FIG.
Next, the examination target address a of the input image bitmap data is set at the head of the image data (S2503). That is, the address a is a relative address in the input image bitmap data, and an initial value of the relative address is given in this process.
That is, steps S2805 and S2806 are equivalent to the subroutine R2522 enclosed by the dotted line in FIG. This is a process of creating array data including a co-occurrence probability matrix and determining a pixel of a suspicious object in the image flag bitmap data area 2104 therefrom.

When the processing is finished, the counter variable i is advanced (S2807). Then, the presence / absence of the next relative position function δ (i) is confirmed (S2803), and if it is set (S2804), the processing is continued. If not, the processing is completed for all the relative position functions δ (i).
When i image flag bitmap data are obtained for all the relative position functions δ (i), these are integrated and calculated (S2808). The operation may be a logical product of all the bits, or all the bits may be added and compared with an appropriate threshold value. This calculation process increases the accuracy of determining the suspicious object region.
Finally, the area and center coordinates of the suspicious object region are calculated (S2809), and the process ends (S2810).

Summarize what has been described above.
The image processing apparatus 2001 of this embodiment is mainly composed of software that runs on a microcomputer.
When still image data is input to the image processing apparatus 2001, it is expanded in the input image bitmap data area 202 in the RAM 105. Thereafter, array data is created in the array data area 203 by the algorithm of the co-occurrence probability matrix. Next, a threshold value is calculated from the obtained array data, and an image pixel corresponding to a number of matrices satisfying the threshold value is determined to be the background. The relative address of the pixel of the image determined to be the background is read from the array data, and the flag of the corresponding address in the image flag bitmap data area 2104 is lowered.
Through the above processing, the background and other portions can be discriminated from the co-occurrence probability matrix generated from the still image data.

[10. Sixth Embodiment]
Next, a sixth embodiment of the present invention will be described.
FIG. 29 is an example of a still image captured by the imaging camera 202.
Compare with the image in Figure 22 (a). In FIG. 22A, the suspicious object 304, which is a ship, is shown, but in FIG. 29, a shore 2902 and a lighthouse 2903 are shown on it. It is clear that these are not suspicious objects, but there is a high possibility that they will be mistaken as suspicious objects as they are in the fourth embodiment. This is because these are not recognized as the background because they occupy a small area when viewed from the whole image data. Therefore, it is necessary to perform a process of previously removing a part that may be mistaken as a suspicious object from the object of the process for creating the co-occurrence probability matrix in advance.

FIG. 30A is an overall block diagram of an image processing apparatus according to the sixth embodiment. An image processing apparatus 3001 shown in FIG. 30A is almost the same as the image processing apparatus 2001 of FIG. 20 in the fourth embodiment. In FIG. 30 (a), parts having the same functions as those in FIG. 20 are denoted by the same reference numerals, and detailed description thereof is omitted.
The image processing apparatus 3001 according to the present embodiment performs a process of excluding a specific area in the image data from a target for determining a suspicious object in advance, based on the captured still image data, before the actual operation. For this reason, as compared with the fourth embodiment shown in FIG. 20, a processing area setting unit 3002, a display device 3003, and an input device 3004 are newly provided.
The display device 3003 is, for example, a well-known LCD display. The input device 3004 is a known mouse, for example.
Based on the result obtained from the pixel determination unit 3014, the processing region setting unit 3002 causes the display device 3003 to display a region that seems to be a suspicious object and overlaps with the image data held in the image storage unit 112. The user operates the input device 3004 to correct the still image displayed on the display device 3003 and the excluded area displayed in an overlapped state on the still image.
As described above, the exclusion region set in the feature amount calculation unit 3013 by the processing region setting unit 3002 is excluded from the calculation of the co-occurrence probability matrix performed in the feature amount calculation unit 3013.
If FIG. 29 is taken as an example, an exclusion region surrounding the shore 2902 and the lighthouse 2903 is created by the processing region setting unit 3002 and held in the feature amount calculation unit 3013. The feature amount calculation unit 3013 forcibly regards the pixels corresponding to the exclusion region as the background without calculating the co-occurrence probability matrix among the pixels of the image data.

[11. Seventh Embodiment]
Next, a seventh embodiment of the present invention will be described.
FIG. 30B is an overall block diagram of the image processing apparatus according to the seventh embodiment. An image processing apparatus 3011 shown in FIG. 30B is substantially the same as the image processing apparatus 3001 of FIG. 30A in the sixth embodiment. In FIG. 30 (b), parts having the same functions as those in FIG. 30 (a) are given the same reference numerals, and detailed descriptions thereof are omitted.
Similar to FIG. 30A, the image processing apparatus 3011 according to the present embodiment starts from a target for determining a suspicious object in advance in a specific area in image data based on the captured still image data before the actual operation. Perform the removal process. The difference from the sixth embodiment is that the exclusion area is not automatically corrected by the operator and is set automatically. For this reason, the display device 3003 and the input device 3004 are not provided as compared with the sixth embodiment shown in FIG. In addition, the processing region setting unit 3003 is connected to the pixel set determination unit 3015.
The processing region setting unit 3003 regards the result obtained from the pixel set determination unit 3015 as an excluded region and sets it in the feature amount calculation unit 3013.
The set exclusion region is excluded from the calculation of the co-occurrence probability matrix performed in the feature amount calculation unit 3013.

As described above, the image processing apparatuses described in the first, second, third, fourth, fifth, sixth, and seventh embodiments are the same as those described in FIGS. Is particularly suitable for monitoring a single background such as For this reason, the imaging camera is often rotated to monitor a wide range.
FIG. 31A shows an external view of the imaging camera in the present embodiment. FIG. 31B is a block diagram showing a combination of the imaging camera and the image processing apparatus in FIG.
In FIG. 31 (a), the imaging camera 202 is installed in the turning device 3102 and driven to rotate left and right in order to monitor a wide range.
In FIG. 31B, inside the turning device 3102 are provided a motor 3103 for rotationally driving the imaging camera 202 and an angle detector 3105 that is attached to the shaft 3104 of the motor and is rotationally driven together with the imaging camera 202. ing. Examples of the angle detector 3105 include a tachometer using optical detection, magnetic detection, and the like.
A detection signal from the angle detector 3105 is input to a bus 106 of a microcomputer that constitutes a main component of the image processing apparatus 201, and a still image obtained from the imaging camera 202 is taken into the RAM 105 at every predetermined angle. The above processing is performed.
In the sixth and seventh embodiments, the exclusion area is set in advance for each shooting angle of the imaging camera 202.

The following application examples can be considered in the present embodiment.
(1) Instead of mounting by a microcomputer, a programmable logic device (PLD) may be used.
(2) In the first, second, and third embodiments described above, it is possible to add an exclusion process that excludes a background portion that is likely to be erroneously recognized as a suspicious object from the target of the feature amount calculation process.
(3) A plurality of the first, second, third, fourth, fifth, sixth and seventh embodiments described above can be simultaneously operated in a single monitoring device. The set of flags for each region obtained as a result of the arithmetic processing can be all merged by logical product or compared with a predetermined threshold after the addition processing.

(4) In the sixth and seventh embodiments described above, the exclusion region set by the processing region setting units 3002 and 3003 is reflected in the feature amount calculation unit 3013. Instead, the same effect can be obtained by reflecting the excluded region in the pixel set determining unit 2015 or the pixel set determining unit 3015. That is, the part corresponding to the exclusion area is regarded as the background, and the flag of the corresponding part in the image flag bitmap data area 2104 is lowered.
(5) Although the input image bitmap data area 2102 and the image flag bitmap data area 2104 are described as different areas in the RAM 205 in FIG. 21, they can be integrated.
For example, a structure that stores luminance data and flag data is configured for each pixel that configures image data. In this way, the absolute address for each pixel can be directly stored in the array data 2301, so that the arithmetic processing for converting the relative address is not necessary.
(6) Various methods for expressing the co-occurrence probability matrix configured by the array data area 2103 can be considered. For example, it can be realized by using a relational database or an associative array implemented by perl which is a well-known interpreter language processing system.

(7) Basically, how to determine the horizontal division region is arbitrary according to the input image data. However, it would be ideal if the horizontal division region determination procedure could be automated. Therefore, when the clustering technique described in the third embodiment is applied to the entire input image data, and it is found that a plurality of similar regions are distributed in the horizontal direction, horizontal division regions are determined so as to cover the regions. The method of, can be considered.
In this embodiment, it is expected that the horizontal dividing line is matched with the coastal horizontal line by taking a small divided area for clustering. This leads to a reduction in the number of recognition targets as backgrounds in the horizontal division region, and as a result, an improvement in the background recognition rate can be expected.

In the present embodiment, an object different from the background can be discriminated from still image data that is uniform in the horizontal direction and changes in the vertical direction, represented by a distant natural landscape. This processing is different from the method based on comparison with the immediately preceding image data, which is well known in the prior art, and is determined using only the input image data. For this reason, it is particularly suitable in the monitoring mode in which the imaging camera is installed on the swivel device and rotated to the left and right.
In the present embodiment, various determination methods are disclosed for the horizontal division region. In these embodiments, even if there is a change caused by a natural phenomenon such as the sea or the sky, it can be determined that the background has a certain feature including this.
Therefore, by combining the image processing apparatus according to the present embodiment with an imaging camera, an excellent monitoring apparatus that can recognize a natural phenomenon such as a sea wave or a cloud in the sky as a background without being misrecognized as compared with the prior art. Can be provided.

  The embodiment of the present invention has been described above. However, the present invention is not limited to the above-described embodiment, and other modifications may be made without departing from the gist of the present invention described in the claims. It goes without saying that application examples are included.

It is an image figure which shows roughly operation | movement of the image processing apparatus by the technical idea common to embodiment of this invention. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an overall block diagram of an image processing apparatus according to a technical idea common to the embodiments of the present invention, and a flowchart showing its operation. 1 is an overall block diagram of an image processing apparatus according to a first embodiment. It is an image figure which shows schematically operation | movement of 1st Embodiment. It is an image figure which shows roughly the method to designate the background area | region in 1st Embodiment. It is a flowchart which shows pre-processing among the processing contents of the monitoring apparatus of 1st Embodiment. It is a flowchart which shows the process at the time of real operation among the processing content of the monitoring apparatus of 1st Embodiment. It is a flowchart which shows the process at the time of real operation among the processing content of the monitoring apparatus of 1st Embodiment. It is the schematic explaining the color histogram which is one of the feature-value calculation methods. It is the schematic explaining frequency analysis which is one of the feature-value calculation methods. It is the schematic explaining the co-occurrence probability matrix which is one of the feature-value calculation methods. It is the schematic explaining the relationship between image data and a co-occurrence probability matrix. It is a whole block diagram of the image processing apparatus by 2nd Embodiment. It is an image figure which shows schematically operation | movement of 2nd Embodiment. It is a flowchart which shows the processing content of the monitoring apparatus of 2nd Embodiment. It is a flowchart which shows the processing content of the monitoring apparatus of 2nd Embodiment. It is a whole block diagram of the image processing apparatus by 3rd Embodiment. It is an image figure which shows schematically operation | movement of 3rd Embodiment. It is a flowchart which shows the processing content of the monitoring apparatus of 3rd Embodiment. It is a whole block diagram of the image processing apparatus by 4th Embodiment. It is the schematic which shows the inside of RAM of 4th Embodiment. It is an image figure which shows roughly the input image bitmap data and image flag bitmap data comprised in RAM of 4th Embodiment. It is an image figure which shows roughly the co-occurrence probability matrix comprised in RAM of 4th Embodiment. It is an image figure which shows roughly the correspondence of each data stored in RAM of 4th Embodiment. It is a flowchart which shows the processing content of the image processing apparatus of 4th Embodiment. It is a flowchart which shows the processing content of the image processing apparatus of 4th Embodiment. It is an image figure which shows the correspondence of the pixel of the image data by 5th Embodiment, and relative position function (delta) (i). It is a flowchart which shows the processing content of the image processing apparatus of 5th Embodiment. It is an image figure which shows roughly the input image bitmap data comprised in RAM by 6th Embodiment. It is a whole block diagram of the image processing apparatus by the 6th and 7th embodiment of this invention. 1 is an overall block diagram of an image processing apparatus applied to each embodiment of the present invention combined with an example of an imaging camera. FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 201 ... Monitoring apparatus, 202 ... Imaging camera, 203 ... CPU, 204 ... ROM, 205 ... RAM, 206 ... Bus, 207 ... Output interface, 212 ... Image storage part, 213 ... Horizontal area division part, 214 ... Area determination process part

Claims (3)

An image storage unit for storing input image data;
A horizontal region dividing unit for horizontally dividing the input image data in the image storage unit;
A feature amount calculation unit that calculates a co-occurrence probability matrix of a horizontal division region obtained from the horizontal region division unit, and compares the co-occurrence probability matrix obtained by the feature amount calculation unit with a predetermined threshold value, An image including a pixel determination unit that determines whether or not, and a region determination processing unit that includes a pixel set determination unit that outputs the presence / absence of a suspicious object or the like based on a determination result obtained by the pixel determination unit Processing equipment.   The feature quantity calculation unit constitutes a co-occurrence probability matrix with an array having addresses indicating pixels constituting image data in a horizontal division area obtained from the horizontal area division unit. Item 6. The image processing apparatus according to Item 1. Dividing the input image data in the horizontal direction to obtain a plurality of horizontal division regions;
Evaluating each pixel in the horizontal division region with a predetermined relative position function;
Storing the address of the pixel in the corresponding matrix element of the co-occurrence probability matrix in the evaluation step;
Comparing the co-occurrence probability matrices that have been created with a predetermined threshold;
And a step of determining whether each of the horizontally divided areas is a suspicious object or a background as a result of the comparison.

Images