JP3195537B2 - Monitoring device and monitoring method using the device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a monitoring device and a monitoring method using the monitoring device, and particularly to an outdoor device and the like.
Even in environments where significant disturbances occur for image processing, such as changes in illuminance, or when subjects and backgrounds are complex,
The present invention relates to a monitoring device suitable for detecting an abnormality occurring in a point or a device to be monitored and a monitoring method using the device.

[0002]

2. Description of the Related Art Conventionally, an on-site abnormality monitoring device installed in a large plant such as a power plant or a chemical plant reproduces raw signals such as video and sound received by a television camera or a microphone directly on a television or a speaker. Is constantly monitored by an operator, an inspector, or a maintenance engineer (hereinafter, simply referred to as an operator), thereby discovering the occurrence of an abnormality in a driving operation device or the like. In such a monitoring method, the operator must always determine whether or not there is an abnormality, and the determination of the presence or absence of an abnormality requires considerable skill, which increases the burden on the operator. there were.

On the other hand, recently, the operation of a large plant has been
Remote control for monitoring an operation controller and the like from a central control room using an abnormality monitoring device equipped with various monitoring instruments has come to be performed. With the development of image processing technology and devices, such a monitoring system is intended to process images and the like from a TV camera mechanically with an image processing device or the like, and to use it for determination of abnormality or early detection. In addition to reducing the burden on the operator, saving labor, and aiming at the practical use of a so-called alternative system for human visual functions.
However, it is said that the ratio of abnormalities such as leaks detected by visual and auditory inspections is limited to about 90% of the total number of occurrences even during patrol inspections by operators on site patrols. It is an important issue to improve the reliability of the method of determining the abnormality by processing the data in a timely manner.

FIG. 7 shows an example of a conventional monitoring system for finding an abnormality by image processing. In the figure, a TV camera 102 and a microphone 1
03 is installed, a monitoring robot 101 is mounted,
At predetermined monitoring routes and monitoring points, video data and audio data including the part to be monitored and its vicinity are collected, and this data is used to determine abnormalities installed in the central control room via a communication device. Transferred to device 104.

[0005] The transferred data is processed, for example, according to the flow shown in FIG. FIG. 8 shows an algorithm of a method for finding an abnormality by the image processing in FIG. In FIG. 8, first, an image in a normal state is fetched from among images including a portion to be monitored at each monitoring point and the vicinity thereof, stored in the memory (1), and is used as a reference image. Next, a screen newly captured by monitoring by a television camera mounted on the monitoring robot is stored in the memory (2), and a difference from the reference image is set as a difference image, which is used as base data. Then, the difference between the normal case and the abnormal case in the difference image is distinguished using the luminance threshold. That is, a portion where the luminance difference is larger than the threshold is processed as white, and a portion where the difference is smaller is processed as black.
A so-called binarization process is performed to clearly distinguish the difference between the normal state and the abnormal state. Next, a histogram operation is performed to determine the presence or absence of an abnormality based on the number of white pixels, and an alarm is issued in the case of an abnormality.

FIG. 9 shows an example in which oil mixed into a waterway is detected by using a conventional monitoring device equipped with such a television camera. In the figure, A1 and A2 are images in a normal state. A1 is taken in the image and A2 is stored in the memory (1) as a reference image. B1 is an image when oil is mixed during monitoring and is stored in the memory (2).
Next, A derived from the memory (1) and the memory (2)
The difference image C between 1 and B1 is binarized and a histogram is calculated. In this case, D is determined by the luminance threshold level.
1, D2, D3, etc. Here, D1 has a white ratio of 35%, but it can be seen that this value is excessive and the threshold value is too low as compared with the actual image of B1. In D3, the ratio of white color is 2%, and the actual B1
It can be seen that this value is too small and the threshold value is too high as compared with the image of FIG. Here, the threshold value in the case of D2 is appropriate.

[0007] As described above, when the brightness of the surface such as a water channel changes every moment, an object to be monitored is detected, and the difference image between the normal image and the monitored image is binarized to detect an abnormality. Has a white threshold depending on the brightness threshold setting,
That is, since the amount of change from the reference image in a normal state greatly varies, for example, from 2% to 35%, it is very difficult to set a threshold value used for determination in the final stage through a process of trial and error. It is. Each time the threshold value of the luminance is set to an optimum value in the binarization processing through a process of trial and error, the setting has the following problems (i) to (iii).

[0008] (i) When the monitoring place is outdoors, a large change in illuminance due to a change in sunshine, or a change in the position, size, or shape of a shadow of equipment or the like in the camera field of view changes irregularly. There is a problem that a disturbance occurs, and a disturbance occurs even indoors due to lighting conditions such as a fluorescent lamp, so that the luminance value in the difference image changes, and the result of the binarization process significantly changes.

[0009] (ii) The position of the subject on the reference screen initially captured and stored may be misaligned, or the monitoring device may stop at the monitoring point to reproduce the position.
There is a problem that the luminance value in the difference image changes and the result of the binarization process changes. (Iii) The above (i), (ii)
In order to avoid the disturbance in the image processing in, it is conceivable to predict the disturbance in advance and set the conditions, but it is necessary to store each reference image for a large number of monitoring items for each monitoring point, There is a problem that a huge memory is required.

There is also known a method of processing a luminance data value itself in addition to the above-described luminance threshold value to cope with a large change in illuminance outdoors or the like. According to this method, when the object to be monitored is a simple one such as an oil pan or floor having a substantially flat shape, it can be dealt with by using, for example, texture analysis and principal component analysis. However, when the background or the subject contains a large number of curved surfaces, edges, folds, and the like, for example, variations in the feature amounts in the texture analysis occur everywhere in the image, and thus cannot be used as an index for abnormality determination. . Also, when monitoring outdoors, the brightness at day and night is large,
In order to cope with irregular changes, it is necessary to use an image monitoring and input device that uses an infrared camera in addition to a visible camera. There were many unclear points such as the setting of indicators for this, and it could not be put to practical use.

On the other hand, as a conventional technique different from the above-mentioned technique for binarizing a difference image between a normal image and a monitoring image, a normal case and an abnormal case are learned in advance by a neural network. There is known a method of causing a learning image to be stored as a learning image. However, this method has a drawback that if the newly monitored image is significantly different from the learning image, the determination output value from the neural network may be output as an ambiguous value. This is caused by the limitation of the effectiveness of the generalization ability in the neural network, and its prediction is considered difficult at present. It is difficult at present to apply the method.

[0012]

That is, the above-mentioned prior art is effective when the background and the subject in the area including the part to be monitored and the vicinity thereof are simple, but is difficult to apply when the background and the subject are complicated. It is. In addition, in a monitoring device used outdoors in a plant where continuous operation is performed day and night, it is very dark in the morning and evening, and at night, it is necessary to use a visible camera and a high-sensitivity camera together in order to maintain image processing accuracy. However, it was difficult to automate it. Therefore, even when the background or the subject in the area to be monitored is complicated, a monitoring device and a monitoring method using the device can be sufficiently applied to an environment where the brightness greatly changes, such as day and night. Was desired.

An object of the present invention is to respond to such a demand. Even when a background or a subject in a region including a portion to be monitored and its vicinity is complicated, or in a place where the brightness greatly changes. Another object of the present invention is to provide a monitoring device which can sufficiently cope with the above problem and a monitoring method using the monitoring device.

[0014]

The invention claimed in the present application to achieve the above object is as follows. (1) In a monitoring apparatus for monitoring the state of a subject by processing luminance data of a monitoring image captured from an image capturing unit, the monitoring image captured from the image capturing unit is provided.
Compare the image with the previously captured reference image
A shift correcting means for correcting a shift between the two images, and a corrected image
Texture analysis means for calculating a feature amount by texture analysis of the whole; and a monitoring image corrected by the shift correction means.
Area dividing means for dividing an image into a monitoring target area and a background area
And the average luminance of the background area based on the divided area
Luminance evaluation means to be obtained and luminance obtained by the luminance evaluation means
A reference data selecting means for selecting the reference data of the texture features from the relationship between the luminance data and texture feature which has been previously obtained data, features selected by the feature and the reference data selecting means which has been determined by the texture analyzer A monitoring device comprising: an abnormality determining unit that determines whether the subject is abnormal by comparing the reference data with the amount of the reference data.

[0015] (2) In the monitoring apparatus according to (1), the Tex Chi catcher analyzing means and the reference data selecting means, variance, as the texture feature either one or more of the contrast and entropy A monitoring device characterized by being used.

(3) In the monitoring device according to (1), an image capturing unit that captures the monitoring image is selected based on a relationship between an average luminance of a background region of the monitoring image and illuminance or infrared intensity. A monitoring device, comprising: a switching device that performs switching. (4) the image capturing means, the monitoring device according to (3), wherein the visible light camera, is any one or a combination of two or more of highly sensitive cameras and infrared cameras.

(5) In a monitoring method for monitoring the state of a subject by processing luminance data of a monitoring image captured from an image capturing unit, a reference image in which the monitoring image captured from the image capturing unit is stored in advance. A shift correction step of correcting a shift of the image by comparing the image with the image, a texture analysis step of performing a texture analysis on the entire corrected image to calculate a feature amount, and a monitoring image after the shift correction step is used as a monitoring target area and a background. An area dividing step of dividing into an area and
A luminance evaluation step of obtaining an average luminance of a background area based on the divided areas, and luminance data obtained in the luminance evaluation step;
A step of selecting reference data of the texture feature amount from the relationship between the luminance data and the texture feature amount obtained in advance, and a step of selecting the texture feature amount of the entire monitoring image calculated in the texture analysis step and the reference data selection step. An abnormality judging step of judging the presence / absence of an abnormality in the subject by comparing the reference data of the texture feature amount obtained.

(6) In the monitoring method described in the above (5) , the presence or absence of abnormality of the subject is determined twice or more, and when all the same determinations are obtained, the determination result is determined as a final determination, and If the judgment cannot be obtained, the texture feature amount with the smallest difference value among all the reference data of the texture feature amount based on the relationship between the luminance data of the background region and the texture feature amount obtained in advance is set as the reference data. A monitoring method, comprising: determining the reference data of the determined texture feature amount and data of the entire monitoring image processed by the texture analysis to determine the presence or absence of an abnormality.

(7) In the monitoring method described in (6) , when the same determination cannot be obtained, the determination of the presence or absence of the abnormality of the subject is performed by the texture characteristic of the entire monitoring image processed by the texture analysis. The amount is compared with all the reference data of the obtained texture feature amounts, and when there is at least one of which the difference between the two texture feature amounts is equal to or less than a preset value, it is determined that the texture is normal. A monitoring method characterized by determining an abnormality when the value is large.

(8) In the monitoring method according to any one of the above (5) to (7) , the reference data of the texture feature amount when it is determined that the subject has an abnormality due to disturbance such as illuminance, A monitoring method comprising a step of registering as new reference data of a texture feature amount for comparison with data of the entire monitoring image subjected to texture analysis.

In the present invention, one or more of variance, contrast and entropy are used as the texture feature. In the present invention,
It is preferable to provide a shift correction unit that compares the captured monitoring image with the previously captured reference image and corrects the shift between the two images, on the upstream side of the texture analysis unit. Further, on the upstream side of the reference data selecting means, there may be provided an area dividing means for dividing the monitoring image into a monitoring target area and a background area, and a luminance evaluating means for calculating an average luminance of the background area based on the divided areas. preferable.

In the present invention, it is preferable to provide a switching device for selecting image capturing means for capturing the monitoring image based on the relationship between the average luminance of the background region of the monitoring image and the illuminance or infrared intensity. Examples of the image capturing means include a visible camera, a high-sensitivity camera, and an infrared camera, and any one or a combination of two or more is used.

In the present invention, the presence or absence of an abnormality in a subject is determined twice or more, and when all the same determinations are obtained, the determination result is determined as a final determination. Among all the reference data of the texture features based on the relationship between the luminance data of the background region and the texture features, the texture feature with the smallest difference value is determined as the reference data, and the determined texture feature is It is preferable to determine whether there is an abnormality by comparing the reference data with the data of the entire monitoring image processed by the texture analysis.

In the present invention, the determination of the presence / absence of abnormality in the subject is performed twice or more. If the same determination cannot be obtained, the texture characteristic amount of the entire monitoring image processed by texture analysis and the obtained All the texture data are compared with the reference data. If there is at least one that has a difference between the two texture features less than a preset value, it is judged to be normal. You can also judge.

In the present invention, a new texture feature for comparing the reference data of the texture feature when the subject is determined to be abnormal due to disturbances such as illuminance with the data of the entire monitoring image subjected to texture analysis. It is preferable to have a step of registering as reference data of the amount, and it is preferable to perform this automatically.

[0026]

Next, the present invention will be described in more detail by way of examples. Embodiment 1 An embodiment of a monitoring method according to the present invention will be described with reference to FIG. 1 showing an algorithm of the monitoring method. In FIG.
An image of an area including a part to be monitored and its vicinity (hereinafter, referred to as a monitoring image) is a luminance evaluator 1 described later.
It is captured by the visible camera 1 or the high-sensitivity camera 1 ′ selected by the switch 11 based on the instruction from the camera 2, and is sent to the displacement correction device 2. The monitoring image sent to the shift correcting device 2 is compared with a reference image called from a reference image storage device 13 that stores a reference image for each monitoring point, and the difference image is minimized. Is corrected.

Next, the monitoring image whose deviation has been corrected is sent to the area dividing device 3, where it is divided into, for example, 10 blocks vertically and horizontally composed of 32 × 32 pixels and a total of 100 blocks. At the same time, the divided image block is divided into two areas, a bright part and a dark part, and then sent to the texture analyzer 4 where the light and dark areas are dispersed according to the following equation (1). The quantity VR is calculated.

[0028]

(Equation 1)

The calculated feature value VR is supplied to the abnormality determination device 7.
Sent to As described above, in the present embodiment, the variance is used as the feature value VR in the texture analysis.
On the other hand, the divided image divided into, for example, 100 blocks by the above-described region dividing device 3 is further divided into a monitoring target region and a background region by the region dividing device, and then sent to the luminance evaluator 12, Here, the average luminance of the background other than the monitoring target portion is determined. In the present embodiment, the average luminance of the background is used because what is called so-called brightness constancy is used because the human visual system is stable regardless of the intensity of light in the background. That's why.

The average luminance of the background determined by the luminance evaluator 12 is sent to the reference data selector 5. As shown in the flowchart of FIG. 6, the reference data selector 5 stores the average luminance determined by the luminance evaluator 12 and the database stored in the reference data selector 5 in advance as shown in FIG. The illuminance of the target area is estimated based on the relationship between the illuminance of the visible camera or the high-sensitivity camera and the average luminance of the background, and based on this illuminance, for example, from 80,000 lux of high illuminance to 3 lux of low illuminance A reference data storage device 6 in which texture features such as variance, entropy or contrast over a wide range are stored in advance.
Out of the feature amounts stored in, the corresponding line segment ○-● from the relationship between the illuminance (Lx) and the difference between the reference variance shown in FIG.
Is selected and transmitted to the abnormality determination device 7 as reference variance value data corresponding to the difference between the reference variance values.

FIGS. 3 and 4 show the relationship between the illuminance of the visible camera and the high-sensitivity camera and the average luminance of the background, respectively. As described above, the instruction is sent from the luminance evaluator 12 to the switch 11, and the camera to be used is automatically switched by the switch 12. Next, based on the flowchart shown in FIG. 2, a description will be given of processing from capture of an image to determination of the presence or absence of an abnormality in the abnormality determination device 7.

As described above, the reference variance data is searched by the reference data selector 5 of FIG. 1 for each of the captured image data. The search for the reference data is performed twice, for example. If the variance value data cannot be searched for both times, it is expected that a significant change in the illuminance or the change of the monitoring target image has occurred. Examines whether there is a new registration.

When the reference variance data is retrieved, it is sent to the abnormality determination device 7, where the reference variance data and the data shown in FIG.
Is calculated based on the image processing data sent from the texture analyzer 4 shown in FIG.
The difference from the reference variance value data is compared. This difference is also compared twice, for example, and if the difference is large in both cases, that is, if the difference is abnormal, it is determined to be abnormal and output to, for example, a monitor and an alarm. Conversely, if the difference between the two times is small, that is, if the difference is normal, it is determined to be normal and, for example, monitor output is performed. When the first and second determinations are different in the determination of the abnormality and the normality, it is assumed that disturbance by an operator or the like or an erroneous determination of the device is made, so that it is determined that the determination is impossible.

FIG. 10 shows that the abnormality judging device 7 judges that the reference variance value data selected in the search for the reference variance value data and the image processing data from the texture analyzer 4 are normal. As shown, this is a case where, for example, a difference value as shown in FIG. 12 is obtained when compared with the normal dispersion value data at the illuminance of 80,000 lux in the reference data storage device 6. On the other hand, what is determined to be abnormal is, for example, the image processing data sent from the texture analyzer 4 is, for example, the image data shown in FIG.
This is a case where, for example, a difference value shown in FIG. 13 is obtained when compared with the variance value data in the normal state when the illuminance is 80,000 lux shown in FIG.

In the present embodiment, the threshold value serving as an index for judging an abnormality has an arbitrary width, thereby making it possible to cope with the variation in the actually measured values. In the present embodiment, contrast or entropy can be used instead of variance as a feature amount in texture analysis. The contrast and entropy are expressed by the following equations (2) and (3), respectively.

[0036]

(Equation 2)

[0037]

(Equation 3)

Further, a visible camera and an infrared camera can be used as a combination of cameras mounted on the monitoring device. However, in this case, since infrared images are targeted,
Infrared intensity is used instead of illuminance.

According to the first aspect of the present invention,
Even if the body and the background are complicated, a small abnormality buried in the normal data can be reliably detected, so that the abnormality monitoring accuracy is improved.

According to the invention described in claim 2 of the present application ,
In addition to the effects of the present invention, the influence of disturbance can be further reduced.

[0041]

According to the third and fourth aspects of the present invention,
For example, in addition to the effects of the above invention, for example, switching between a visible camera and a high-sensitivity camera or an infrared camera can be automatically performed by illuminance based on luminance data. Regardless of location or location, continuous monitoring for 24 hours is possible.

According to the fifth aspect of the present invention , an object
Even when the body and the background are complicated, a small abnormality buried in the normal data can be reliably detected, so that the accuracy of the abnormality monitoring device is improved.

According to the invention described in claim 6 of the present application, in addition to the effects of the above invention, the reliability of the monitoring result is improved.

According to the invention of claim 7 of the present application, the reliability of the monitoring result is improved as in the above invention.

According to the invention as set forth in claim 8 of the present application, in addition to the effects of the above-mentioned invention, the width of the monitored object is expanded.

[Brief description of the drawings]

FIG. 1 is a diagram showing an algorithm of an embodiment of a monitoring method according to the present invention.

FIG. 2 is a diagram showing a flowchart from capture of an image to determination of the presence or absence of an abnormality.

FIG. 3 is a diagram showing a relationship between an average luminance and illuminance of a background and a use range of a visible camera.

FIG. 4 is a diagram showing a relationship between average luminance and illuminance of a background, and a use range of a high-sensitivity camera.

FIG. 5 is a diagram showing a relationship between a difference from a reference dispersion amount and illuminance.

FIG. 6 is a view showing a flowchart of variance data selection in a reference data selector.

FIG. 7 is an explanatory diagram showing a conventional abnormality monitoring system.

FIG. 8 is a diagram showing an algorithm of an image processing method of a conventional abnormality monitoring system.

FIG. 9 is a diagram showing an example of abnormality monitoring in a conventional abnormality monitoring device.

FIG. 10 is a diagram showing variance value data in a normal state when the illuminance is 80,000 Lx.

FIG. 11 is a diagram showing variance data at the time of abnormality when the illuminance is 80,000 Lx.

FIG. 12 is a diagram illustrating a difference amount of a dispersion amount in a normal state when the illuminance is 80,000 Lx.

FIG. 13 is a diagram showing a difference amount of the dispersion amount at the time of abnormality when the illuminance is 80,000 Lx.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Visible camera, 1 '... High-sensitivity camera, 2 ... Misalignment correction device, 3 ... Region division device, 4 ... Texture analyzer, 5 ... Reference data selector, 6 ... Reference data storage device, 7 ... Abnormality judgment device, 8 alarm device, 9 monitor, 10 video recorder, 11 switcher, 12 luminance evaluator, 13 reference image storage device.

──────────────────────────────────────────────────続 き Continuing on the front page (72) Atsushi Nakahara, 6-9 Takaracho, Kure-shi, Hiroshima Prefecture Inside the Kure Plant, Baccock Hitachi Ltd. (72) Nobuo Matsuo 6-9 Takaracho, Kure-shi, Hiroshima Prefecture, Babcock Day (56) References JP-A-7-85385 (JP, A) JP-A-6-60182 (JP, A) JP-A-6-165182 (JP, A) (58) Fields surveyed ( Int.Cl. ⁷ , DB name) H04N 7/18 G08B 13/196

Claims (8)

(57) [Claims] 1. A monitoring apparatus for monitoring the state of a subject by processing luminance data of a monitoring image fetched from an image fetching means.
The monitoring image and the reference image
A displacement correcting means for comparing and correcting the displacement between the two images;
Texture analysis means for calculating the feature amount by texture analysis of the entire image thus obtained, and correction by the shift correction means.
Area that divides the monitoring image into a monitoring target area and a background area
Dividing means for dividing the background area based on the divided area;
Brightness evaluation means for obtaining the average brightness;
A reference data selecting means for selecting the reference data of the texture features from the relationship between the luminance data and texture feature which has been previously determined to luminance data, selected by the feature and the reference data selecting means which has been determined by the texture analyzer A monitoring device for determining whether or not the subject is abnormal by comparing the reference data of the characteristic amount with the reference data. 2. A monitoring device as claimed in claim 1, wherein the vertex Chi catcher analyzing means and the reference data selecting means, the dispersion is used as the texture feature either one or more of the contrast and entropy A monitoring device characterized in that: 3. The monitoring apparatus according to claim 1 , wherein the image capturing means for capturing the monitoring image is selected based on a relationship between an average luminance of a background area of the monitoring image and illuminance or infrared intensity. A monitoring device comprising a device. 4. The image capturing means according to claim 1, wherein :
Claims : A combination of one or more of a high sensitivity camera and an infrared camera.
Item 4. The monitoring device according to Item 3 . 5. A monitoring method for monitoring the state of a subject by processing luminance data of a monitoring image fetched from an image fetching means, wherein a monitoring image fetched from said image fetching means is stored in advance. A shift correction step of correcting a shift of an image by comparing with a texture analysis step of performing a texture analysis on the entire corrected image to calculate a feature amount; and a monitoring target area and a background area after the shift correction step. And a luminance evaluation step of obtaining an average luminance of a background area based on the divided area; luminance data obtained in the luminance evaluation step; luminance data and a texture feature amount obtained in advance. Selecting the reference data of the texture feature amount from the relationship between the texture feature amount and the texture feature amount of the entire monitoring image calculated in the texture analysis step. Monitoring method characterized in that it comprises an abnormality determination step of determining the presence or absence of an abnormality of the subject by comparing the reference data of the texture feature values selected in the reference data selecting step. 6. The monitoring method according to claim 5 , wherein the determination of the presence or absence of the abnormality of the subject is performed twice or more, and when all the same determinations are obtained, the result of the determination is final.
If the same judgment cannot be obtained, the texture feature amount that minimizes the difference value among all the reference data of the texture feature amount based on the relationship between the luminance data of the background region and the texture feature amount that has been obtained in advance is used as a reference. A monitoring method, wherein the monitoring data is determined as data, and the determined reference data of the texture feature amount is compared with data of the entire monitoring image processed by the texture analysis to determine whether there is an abnormality. 7. In the monitoring method according to claim 6 , when the same determination cannot be obtained, the determination as to whether or not the subject is abnormal is made based on a texture feature amount of the entire monitoring image processed by the texture analysis. Comparing all the reference data of the obtained texture feature amounts with each other. If there is at least one whose difference between the texture feature amounts is equal to or smaller than a preset value, it is determined that the texture feature amounts are normal, and if all of them are large, A monitoring method characterized by determining an abnormality. 8. The monitoring method according to claim 5, wherein the reference data of the texture feature when the subject is determined to be abnormal due to disturbance such as illuminance is texture-analyzed. A method of registering as reference data of a new texture feature amount for comparison with data of the entire image for use.