JP2022120480A - Analysis system, analysis device, analysis method, and program - Google Patents

Abstract

To reduce the man-hours required by a maintenance person to classify images of a device.SOLUTION: An image analysis system 20 includes an imaging apparatus 130 for imaging a device, and an analysis device 100 which analyzes an image of the device. The imaging apparatus 130 transmits an image to the analysis device 100. The analysis device 100 receives an operation to select a plurality of normal images indicating a normal state of the device from among a plurality of accumulated images, extracts a first feature quantity and a second feature quantity of each of the normal images, defines an area that includes all of the normal images in a coordinate space formed of the first feature quantity and the second feature quantity, receives an operation to select a new normal image from among the accumulated images, and recalculates a range of the area on the basis of the fact that the new normal image is not included in the area.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The present disclosure relates to an image analysis system, and more particularly to an image analysis technique that indicates the state of equipment.

2. Description of the Related Art In facilities such as power plants and substations that have a large number of devices, patrol inspections are carried out in order to maintain the devices and to detect abnormalities due to aged deterioration at an early stage. In the patrol inspection, maintenance personnel visit the facility directly to check the gauges installed in each device or to visually check the state of the device.

From the viewpoint of reducing costs and improving the efficiency of maintenance work, an inspection method is being studied in which surveillance cameras are installed on the premises of facilities and the images captured by the surveillance cameras are checked from a remote location. The use of surveillance cameras is expected to reduce the number of maintenance personnel dispatched to the site or the frequency of visits.

However, since equipment such as substations has a very low frequency of failures, it takes a very long period of time to accumulate abnormal images. Therefore, it is difficult to prepare learning data for machine learning, and it is difficult to monitor equipment abnormalities using AI (Artificial Intelligence).

Regarding accumulation of abnormal images, for example, Japanese Patent Application Laid-Open No. 5-149728 (Patent Document 1) discloses an image processing apparatus capable of "artificially editing defective sample images from a single sample image". This image processing apparatus has a "feature extraction unit and a learning determination unit. The feature extraction unit receives an image signal from a television camera, extracts a feature amount, and generates a feature data string. is input, and recognition processing is performed using determination parameters generated in advance from images for teaching.This image processing apparatus has an image memory for storing sample images and images for teaching. A teaching image is edited from the sample image based on the image editing information input from the sample image, written in the image memory, and the determination parameter is generated using this image” (see [Summary]). .

JP-A-5-149728

According to the technique disclosed in Patent Literature 1, it is impossible to generate an abnormal image that occurs in the actual environment, and there is a possibility that the abnormality determination of the device cannot be performed normally. As a result, maintenance personnel need to classify a large number of images captured by surveillance cameras over a long period of time for purposes such as pattern matching or machine learning. Therefore, there is a need for a technique for reducing the man-hours required for image classification by maintenance personnel.

The present disclosure has been made in view of the background as described above, and an object in one aspect is to provide a technique for reducing man-hours for image classification by maintenance personnel.

According to one embodiment, an image analysis system is provided. The analysis system comprises an imaging device for imaging equipment and an analysis device for analyzing the image of the equipment. The imaging device transmits the image to the analysis device. The analysis device accepts an operation of selecting a plurality of normal images indicating that the state of the device is normal from among the plurality of accumulated images, and calculates the first feature amount and the second characteristic amount of each of the plurality of normal images. A feature amount is extracted, a region including all of the plurality of normal images is defined in a coordinate space composed of the first feature amount and the second feature amount, and a new normal image is selected from the accumulated plurality of images. Then, based on the fact that the new normal image is not included in the region, the range of the region is recalculated.

According to an embodiment, it is possible to reduce man-hours for image classification by maintenance personnel.

The above and other objects, features, aspects and advantages of this disclosure will become apparent from the following detailed description of the disclosure taken in conjunction with the accompanying drawings.

It is a figure showing an example of composition of analysis system 20 according to an embodiment. FIG. 4 is a diagram showing an example of thresholds for classifying images; FIG. 4 is a diagram showing a first example of feature amounts extracted from an image; FIG. 10 is a diagram showing a second example of feature amounts extracted from an image; FIG. 10 illustrates a first example of a threshold update function for classifying images; FIG. 10 illustrates a second example of a threshold updating function for classifying images; FIG. 10 is a diagram illustrating an example of a normal image filter generation function; FIG. 5 is a diagram showing an example of the influence of imaging time and sunshine conditions on an image; 3 is a diagram showing an example of a hardware configuration of an information processing device 900 used as an analysis device 100; FIG. 3 is a diagram showing an example of internal processing of the analysis device 100; FIG.

Hereinafter, embodiments of the technical concept according to the present disclosure will be described with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. In the following description, when referring to a plurality of identical configurations, they may be expressed as configurations 123A and 123B. Moreover, when collectively referring to them, they are expressed as a configuration 123 .
<A. System configuration>
FIG. 1 is a diagram showing an example of the configuration of analysis system 20 according to the present embodiment. Analysis system 20 may collect and analyze images of equipment 140 installed in any facility 10, such as a power plant, substation, or factory. Images collected by analysis system 20 may be used for anomaly determination of device 140 by pattern matching, machine learning of models for determining anomalies of device 140, or visual inspection. The monitored equipment 140 may include, for example, a switchgear installed at a substation, a transformer, or any equipment installed at any facility.

(a. System components)
Analysis system 20 includes analysis device 100 , terminal 110 , access point (AP) 120 , and surveillance camera 130 . Analysis device 100 , terminal 110 , access point 120 , and monitoring camera 130 are interconnected via local network 40 . Also, the analysis device 100 or the terminal 110 may be connected to a wide area network outside the facility 10 . In a certain aspect, analysis system 20 may include only analysis device 100 and cooperate with terminals, access points, and surveillance cameras installed in facility 10 in advance.

The monitoring camera 130 images the equipment 140 inside the facility 10 . Also, the monitoring camera 130 transmits the image to the analysis device 100 via the local network 40 within the facility 10 . The image can be used to detect anomalies such as rusting or oil leakage in the device 140 body. In one aspect, multiple surveillance cameras 130 may be installed within facility 10 . Also, in another aspect, monitoring camera 130 may capture a video instead of an image and transmit the video to analysis device 100 . Furthermore, in another aspect, surveillance camera 130 is a fixed camera, a mobile camera running on a rail, a camera mounted on a drone, a camera mounted on a self-propelled robot, or a combination thereof. etc.

The analysis device 100 associates an image (or video) acquired from the monitoring camera 130 with information of the device 140 to be imaged, such as the manufacturer, model, model, category of the device 140, or start time of use, etc. The image is saved in a storage device (not shown) such as a hard disk. The analysis apparatus 100 has a function of extracting a feature amount of an image, a function of calculating a threshold value for classifying images based on the feature amount, a function of filtering normal images, and the like. Details of these functions will be described later.

In the present embodiment, the “normal image” is an image indicating that the imaged device 140 is in a normal state. Conversely, an “abnormal image” is an image indicating that the imaged device 140 is in an abnormal state. For example, when rusting or oil leakage of the device 140 can be confirmed from the image A, the image A is classified as an abnormal image. Conversely, if no device 140 anomalies can be identified from image A, then image A is classified as a normal image. 'threshold for classifying image' is the threshold for classifying an image as either normal or abnormal.

The terminal 110 is used, for example, by maintenance personnel of the facility 10 to check the images accumulated in the analysis device 100 . The terminal 110 may be installed inside the facility 10 or may be installed at an external monitoring station 30 or the like. A maintenance worker uses the terminal 110 to check a plurality of images accumulated in the analysis device 100 and classify each image into a normal image or an abnormal image. The normal images and abnormal images classified by the maintenance personnel can be used in the analysis device 100 for the calculation of thresholds for classifying the images and the image filtering function.

In one aspect, analysis system 20 may be located in a facility, such as in a control room. In other aspects, portions of analysis system 20 may be located external to the facility. In that case, for example, analysis device 100 may be virtually constructed in a cloud environment or may be a server.

(b. Overview of system operation)
Analysis system 20 is primarily used for collecting images of device 140, classifying the images, generating thresholds for classifying images, generating filters for normal images, updating thresholds for classifying images, and abnormal images. It has six functions of detection of An outline of each function will be described below.

The function of collecting images of equipment 140 is a function of capturing images of one or more equipment 140 using one or more monitoring cameras 130 in facility 10 . The monitoring camera 130 can capture images of the equipment 140 at regular intervals and transmit the images to the analysis device 100 . In one aspect, surveillance camera 130 may capture images of device 140 at predetermined time intervals (eg, every two hours). In another aspect, surveillance camera 130 may capture images of device 140 only when predetermined lighting conditions (such as sunny days) are met. By doing so, the analysis device 100 can collect images captured under similar conditions.

Analysis apparatus 100 analyzes only images captured under similar conditions, so that pattern matching, machine learning models, visual inspection by maintenance personnel, or the like can detect abnormalities in equipment 140 with high accuracy. In one aspect, surveillance camera 130 may determine whether or not a predetermined sunshine condition is satisfied based on the luminance obtained by the imaging device (optical sensor) of surveillance camera 130 .

The image classification function is a function for the analysis device 100 to classify images based on input from maintenance personnel. A maintenance person can browse each of the plurality of images accumulated in the analysis device 100 via the terminal 110 . Also, maintenance personnel can perform an operation via the terminal 110 to classify each of these images as either a normal image or an abnormal image.

The analysis device 100 associates each image with information indicating that it is a normal image or information indicating that it is an abnormal image, and stores the information in the storage unit based on the input operation by the maintenance personnel. Maintenance personnel do not need to classify all the images accumulated in the analysis device 100, but only need to classify some of the images.

The function of generating a threshold value for classifying images extracts one or more feature values from a set of images classified as normal images by maintenance personnel (hereinafter referred to as a “set of normal images”), and extracts the feature values , a threshold for classifying the image is calculated.

In one aspect, analysis device 100 may generate a threshold for classifying images using only a set of normal images. In another aspect, analysis device 100 classifies images using both a set of normal images and a set of images classified as abnormal images by maintenance personnel (hereinafter referred to as "abnormal image set"). may generate a threshold for Details of the threshold generation function for classifying images are described with reference to FIGS.

The normal image filter generation function generates a filter to be used when the maintenance personnel refers to the images accumulated in the analysis device 100 . The plurality of images accumulated in the analysis device 100 must be classified as either normal images or abnormal images by maintenance personnel until sufficient abnormal images are collected. However, since the frequency of occurrence of abnormality in the device 140 is low, maintenance personnel need to check a huge amount of images.

Therefore, the analysis apparatus 100 filters out images that are clearly likely to be normal images from images to be viewed by maintenance personnel. A filter may be generated based on a threshold for classifying the image. In a certain aspect, analysis device 100 may delete images that are clearly likely to be normal images using the filter. In this case, the analysis apparatus 100 can reduce the image storage capacity. In another aspect, analysis device 100 may simply set a flag for excluding an image that is clearly likely to be a normal image from images to be viewed. Details of the normal image filter generation function will be described with reference to FIG.

The threshold update function for determining abnormal images calculates a threshold for classifying images based on a set of normal images newly classified by maintenance personnel, or a set of normal images and a set of abnormal images. It is a function to redo. Details of the threshold update function for classifying images are described with reference to FIGS. 5 and 6. FIG.

The abnormal image detection function is a function of determining whether an image captured by the monitoring camera 130 is an abnormal image based on a threshold for classifying the image. In one aspect, when the analysis device 100 detects a certain number or more of abnormal images, it generates a pattern matching model or a machine learning model also based on these abnormal images, and determines abnormal images based on these models. may be performed.
<B. Image analysis procedure>
Next, the function of generating thresholds for classifying images will be described with reference to FIGS. 2 to 4. FIG. In the following description, the analysis apparatus 100 extracts a first feature amount and a second feature amount from each image, but this is an example and the feature amount extraction method is not limited to this. In a certain aspect, analysis device 100 may extract any number of feature amounts equal to or greater than one from each image.

FIG. 2 is a diagram showing an example of thresholds for classifying images. There are two conceivable procedures for generating thresholds for classifying images. A first threshold generation procedure and a second threshold generation procedure will be described with reference to FIG.

In the first threshold generation procedure, analysis device 100 uses only a set of normal images. First, the analysis device 100 acquires a first feature amount and a second feature amount from each image included in a set of normal images classified by maintenance personnel.

Next, analysis device 100 plots points 220 of each normal image in a two-dimensional space composed of the first feature amount and the second feature amount. For example, the coordinates of image A are (the first feature amount of image A, the second feature amount of image A).

Analysis device 100 then defines a region 200 that includes all normal images. In a certain aspect, analysis device 100 may set region 200 as a circle containing all normal image points. In another aspect, analysis device 100 may define region 200 as an ellipse that includes all normal image points. In another aspect, analysis device 100 may set region 200 to an arbitrary shape including all normal image points. In two-dimensional space, the boundary of the region 200 becomes the threshold for classifying the image.

In the second threshold generation procedure, analysis device 100 uses both a set of normal images and a set of abnormal images. First, the analysis apparatus 100 acquires a first feature amount and a second feature amount from each image included in either a set of normal images classified by maintenance personnel or an abnormal image. Next, analysis device 100 plots points 220 of each normal image and points 230 of each abnormal image in a two-dimensional space composed of the first feature amount and the second feature amount.

Analysis device 100 then defines a region 200 that includes all normal images. In a certain aspect, analysis device 100 may set region 200 as a circle containing all normal image points. In another aspect, analysis device 100 may define region 200 as an ellipse that includes all normal image points. In another aspect, analysis device 100 may set region 200 to an arbitrary shape including all normal image points.

Unlike the procedure for generating the first threshold, the analysis device 100 determines whether the region 200 includes the point 230 of the abnormal image. Based on the inclusion of the abnormal image point 230 in the area 200 , the boundary position of the area 200 is adjusted so that the area 200 does not include the abnormal image point 230 .

In the example shown in this embodiment, analysis apparatus 100 extracts two feature amounts from an image, but the number of feature amounts is not limited to this. In one aspect, analysis device 100 may extract any number of feature quantities from an image. Analysis device 100 may then plot the points of the image in an arbitrary dimensional space.

In one aspect, analysis device 100 uses a predetermined algorithm (for example, obtains the center or barycenter position from all normal image points in the first threshold generation procedure or the second threshold generation procedure, The area 200 may be defined based on the radius of a circle set to the farthest point of the normal image from the center of gravity. In another aspect, analysis device 100 may define region 200 based on a model obtained by machine learning. In that case, analysis device 100 may define region 200 of arbitrary shape. Analysis device 100 can classify an image newly received from surveillance camera 130 as either a normal image or an abnormal image based on region 200 .

FIG. 3 is a diagram showing a first example of feature amounts extracted from an image. In the example shown in FIG. 3, the analysis device 100 can extract any two elements among the RGB (Red, Green, Blue) elements of the image 310 received from the surveillance camera 130 as feature amounts. As an example, the analysis device 100 may extract the number of R pixels included in the image 310 as a first feature amount and the number of G pixels included in the image 310 as a second feature amount. In one aspect, analysis device 100 may extract any number of elements equal to or greater than one as feature amounts. In another aspect, analysis device 100 extracts an arbitrary number of one or more elements among CMYK (Cyan, Magenta, Yellow, Key plate) elements of image 310 received from surveillance camera 130 as feature amounts. good. Analysis device 100 can plot points 320 of each image 310 in two-dimensional space based on the first feature amount and the second feature amount indicating the number of pixels of each color.

FIG. 4 is a diagram showing a second example of feature amounts extracted from an image. In the example shown in FIG. 4 , analysis device 100 can extract any two feature quantities based on local feature points 415 of image 410 received from surveillance camera 130 . In one aspect, analysis apparatus 100 uses SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradients), SURF (Speeded Up Robust Features), BRIEF (Binary Robust Independent Elementary Features) as algorithms for extracting features. , ORB (Oriented FAST and Rotated BRIEF), etc. can be used. In a certain aspect, analysis device 100 may use only one algorithm, or may use multiple algorithms. Analysis device 100 can plot points 420 of each image 410 in two-dimensional space based on the first feature amount and the second feature amount obtained from these algorithms.

In one aspect, analysis device 100 may combine the method described with reference to FIG. 3 and the method described with reference to FIG. 4 to extract the feature amount of the image. For example, the first feature amount may be the number of R pixels, and the second feature amount may be the feature amount of local feature points obtained by SIFT. In another aspect, analysis device 100 may use a machine learning model using a neural network for extracting image feature quantities. In that case, the machine learning model can use a plurality of images accumulated in the analysis device 100 as learning data. The analysis apparatus 100 may execute the learning process of the machine learning model, or another apparatus may execute the learning process of the machine learning model.

FIG. 5 is a diagram showing a first example of a threshold updating function for classifying images. A first example of a threshold updating function for classifying images will be described with reference to FIG. As described with reference to FIG. 2, in two-dimensional space, the boundary of the region 200 is the threshold for classifying the image.

As an example, assume that analysis device 100 receives an operation from a maintenance worker and adds a new normal image point 550A to the two-dimensional space. The new normal image is an image preliminarily classified as a normal image by the image classification function of the analysis device 100 . In a certain aspect, analysis device 100 may receive an operation by a maintenance person from an input device provided in analysis device 100 . In another aspect, analysis device 100 may receive maintenance personnel's operation from terminal 110 .

Suppose that the newly added normal image point 550A was outside the region 200 . In this case, analysis device 100 expands region 200 to the coordinate method of point 550A of the normal image. The point 550A of the normal image has a large second feature amount. Therefore, the expanded region 500A is expanded in the positive direction of the second feature amount axis with respect to the original region 200 .

As another example, assume that analysis device 100 receives an operation from maintenance personnel and adds a new normal image point 550B to the two-dimensional space. In this case, analysis device 100 extends region 200 to the coordinate method of point 550B of the normal image. A point 550B of the normal image has a small first feature amount. Therefore, the expanded region 500B is expanded with respect to the original region 200 in the negative direction of the first feature amount axis.

As described above, when the newly added normal image point is not included in the region 200, the analysis device 100 expands the region 200 in the coordinate direction of the newly added normal image point. , may update the thresholds (boundaries of region 200) for classifying the image. In one aspect, if the region 200 is a circle or an ellipse, the dilation may move its center point toward the newly added normal image point.

In the example shown in FIG. 5, the area 200 is expanded while maintaining the circular or elliptical shape, but the expansion method is not limited to this. In one aspect, analysis device 100 may change normal image region 200 into an arbitrary shape so as to include newly added normal image points.

In another aspect, analysis device 100 may update normal image region 200 by a machine learning model using a neural network. In that case, the machine learning model can use normal image points on the two-dimensional space as training data. The analysis apparatus 100 may execute the learning process of the machine learning model, or another apparatus may execute the learning process of the machine learning model.

FIG. 6 is a diagram showing a second example of a threshold updating function for classifying images. A second example of a threshold update function for classifying images will be described with reference to FIG. The second example shown in FIG. 6 differs from the first example shown in FIG. 5 in that the region 200 can be enlarged while maintaining its shape.

It is assumed that analysis device 100 has received an operation from maintenance personnel and added a new normal image point 650 to the two-dimensional space. The new normal image is an image preliminarily classified as a normal image by the image classification function of the analysis device 100 . In a certain aspect, analysis device 100 may receive a maintenance worker's operation from an input device (not shown) provided in analysis device 100 . In another aspect, analysis device 100 may receive maintenance personnel's operation from terminal 110 .

Suppose that the newly added normal image point 650 was outside the region 200 . In this case, analysis device 100 expands region 200 to include point 650 of the normal image. Region 600 after enlargement maintains the same shape as region 200 but includes newly added normal image points 650 . If the region 200 were a circle or an ellipse, the radial distance would simply increase.

In a certain aspect, even when analysis device 100 plots the points of each image in a one-dimensional space or a space of three or more dimensions, the image is plotted in the same manner as the procedure described with reference to FIG. 5 or FIG. Thresholds for classification may be updated. In another aspect, analysis device 100 may update thresholds for classifying images based on new normal images and new abnormal images. In that case, analysis device 100 adjusts region 200 so that region 200 contains the new normal image and region 200 does not contain the new abnormal image.

FIG. 7 is a diagram illustrating an example of a normal image filter generation function. A normal image filter generation function and its use procedure will be described with reference to FIG. The surveillance camera 130 continues to transmit captured images to the analysis device 100 as needed. Maintenance personnel must verify these images and classify them as either normal or abnormal images. Filters can be used to reduce the man-hours of this image classification work by maintenance personnel.

First, analysis device 100 can define region 200 (calculate the coordinates of the boundary of region 200) based on the first sample of the normal image selected by maintenance personnel. For example, assume that a maintenance worker inputs an operation to the analysis device 100 to select 1000 images captured by the monitoring camera 130 as normal images. The analysis device 100 extracts feature amounts of 1000 images selected as normal images, and defines a normal image region 200 .

Second, when the normal image region 200 is a circle, the analysis device 100 generates a filter for excluding normal images indicated by points within a certain radius from the center of the circle from the images to be viewed. In the example shown in FIG. 7, the filter excludes normal images indicated by points in an area (area 700 where the image is excluded from the image to be viewed) within a radius R2 from the center of the circle (area 200) from the image to be viewed. do. Conversely, the filter includes normal images indicated by points outside the radius R2 and within the radius R1 from the center of the circle (region 200) in the image to be viewed. The filter also includes the image indicated by the points in the circle (region 200) in the viewed image.

In one aspect, the radius R2 may be calculated to have a constant ratio with the radius R1, or may be calculated so that the difference between the radii R1 and R2 is a predetermined value. Further, when the region 200 has an arbitrary shape other than an ellipse or a circle, the analysis device 100 reduces the region 200 at a predetermined ratio to define a region 700 where the image is excluded from the image to be viewed. may

Third, assume that the analysis device 100 newly receives a plurality of new images (10,000 images, for example) from the monitoring camera 130 . If 7000 of the new images are within the radius R2 from the center of the circle (region 200), the analysis device 100 filters out these 7000 images from the images to be viewed. . Conversely, since the remaining 3000 images in the plurality of new images are in the area outside the radius R2 from the center of the circle (area 200), the analysis device 100 regards these 3000 images as images of the visual target. include in

If the coordinate values of a point in image A are close to the coordinate values of the center of region 200, there is a high possibility that image A is a normal image. Conversely, if the coordinate values of a point of a certain image B are near the boundary of the region 200, it is difficult to automatically determine whether the image B is a normal image or an abnormal image. Therefore, the filter excludes only images relatively close to the center of region 200 from the viewed image.

In one aspect, each time a new image is received from surveillance camera 130, analysis device 100 classifies the image with a filter, and automatically classifies the image excluded from the target image as a normal image. good. In another aspect, each time a new image is received from surveillance camera 130, analysis device 100 may classify the image with a filter and automatically remove images that have been excluded from the viewed images. By doing so, the analysis apparatus 100 can reduce the storage capacity of images.

Also, in another aspect, if the point of the new image is outside the region 200, the analysis device 100 may set a flag indicating that the new image is a potential abnormal image. In that case, the analysis device 100 may preferentially present the image with a flag indicating that it is a potential abnormal image to the maintenance personnel. In another aspect, analysis device 100 may transmit the image of the viewing target to terminal 110 based on having received a request for the image of the viewing target from terminal 110 .

As described above, maintenance personnel need to classify a large number of images captured by surveillance cameras over a long period of time for purposes such as pattern matching or machine learning. However, since the frequency of occurrence of abnormality in the device 140 is low, the analysis device 100 is expected to contain a huge number of normal images. Therefore, the analysis device 100 can reduce the number of man-hours required for visual inspection of images by maintenance personnel by providing the filtering function described above.

FIG. 8 is a diagram showing an example of the effects of imaging time and sunshine conditions on an image. Images 810 , 820 , 830 and 840 are images captured by the same surveillance camera 130 . Each image was taken at a different time of day or lighting condition. For example, the image 810 and the image 820 are captured at significantly different times, so the brightness of the entire image is significantly different. Also, when the image 810 and the image 830 are compared, it can be seen that the position where the shadow falls and the overall brightness and the like change depending on the sunshine conditions and the like.

Therefore, monitoring camera 130 can suppress variations in image feature amounts due to the environment by capturing images of the same device 140 under predetermined sunshine conditions and/or times. By suppressing the variation in the feature amount of the image due to the environment, the accuracy of the threshold (boundary of the region 200) and the filter for classifying the image can be improved.

In one aspect, analysis device 100 may transmit a setting of predetermined sunshine conditions and/or time conditions to monitoring camera 130 . In another aspect, surveillance camera 130 may analyze the luminance (light amount) of an image using an image sensor or a photosensor to determine whether the image was captured under predetermined sunlight conditions. In another aspect, analysis device 100 may analyze brightness of an image and determine whether or not the image was captured under a predetermined sunshine condition. When the image is not captured under the predetermined sunshine conditions, the monitoring camera 130 automatically or based on receiving a command from the analysis device 100 after a certain period of time has elapsed since the previous image capturing, An image of the device 140 may be newly captured.

FIG. 9 is a diagram showing an example of the hardware configuration of an information processing device 900 used as the analysis device 100. As shown in FIG. In one aspect, analysis apparatus 100 may execute the processes described with reference to FIGS. 1 to 8 as a program on the hardware shown in FIG. In another aspect, information processing device 900 may be used as terminal 110 . The information processing device 900 includes a CPU (Central Processing Unit) 1, a primary storage device 2, a secondary storage device 3, an external device interface 4, an input interface 5, an output interface 6, and a communication interface 7. include.

The CPU 1 can execute programs for realizing various functions of the information processing device 900 . The CPU 1 is composed of, for example, at least one integrated circuit. The integrated circuit may be composed of, for example, at least one CPU, at least one FPGA (Field Programmable Gate Array), or a combination thereof.

The primary storage device 2 stores programs executed by the CPU 1 and data referenced by the CPU 1 . In one aspect, the primary storage device 2 may be realized by a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), or the like.

The secondary storage device 3 is a non-volatile memory and may store programs executed by the CPU 1 and data referred to by the CPU 1 . In that case, the CPU 1 executes the program read from the secondary storage device 3 to the primary storage device 2 and refers to the data read from the secondary storage device 3 to the primary storage device 2 . In one aspect, the secondary storage device 3 is implemented by a HDD (Hard Disk Drive), SSD (Solid State Drive), EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory, or the like. may

The external device interface 4 can be connected to arbitrary external devices such as printers, scanners and external HDDs. In one aspect, the external device interface 4 may be realized by a USB (Universal Serial Bus) terminal or the like.

The input interface 5 can be connected to any input device such as a keyboard, mouse, touchpad or gamepad. In one aspect, the input interface 5 may be realized by a USB terminal, a PS/2 terminal, a Bluetooth (registered trademark) module, or the like.

The output interface 6 can be connected to any output device such as a CRT display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. In one aspect, the output interface 6 may be realized by a USB terminal, a D-sub terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, or the like.

The communication interface 7 is connected to wired or wireless network equipment. In one aspect, the communication interface 7 may be realized by a wired LAN (Local Area Network) port, a Wi-Fi (registered trademark) (Wireless Fidelity) module, or the like. In another aspect, the communication interface 7 may transmit and receive data using communication protocols such as TCP/IP (Transmission Control Protocol/Internet Protocol) and UDP (User Datagram Protocol).

FIG. 10 is a diagram showing an example of internal processing of the analysis device 100. As shown in FIG. In one aspect, CPU 1 may load a program for performing the process of FIG. 10 from secondary storage device 3 to primary storage device 2 and execute the program. In other aspects, part or all of the process may also be implemented as a combination of circuit elements configured to perform the process.

In step S1010, CPU 1 obtains a normal image. More specifically, CPU 1 acquires an image from surveillance camera 130 via communication interface 7 and stores the image in secondary storage device 3 . Next, the CPU 1 receives an operation of selecting a normal image from the maintenance personnel via the input interface 5 or the communication interface 7 . In one aspect, the CPU 1 can acquire a plurality of normal images (set of normal images).

In step S1020, CPU 1 extracts the feature amount of the normal image. Further, the CPU 1 plots the points of each normal image in the multidimensional space of the feature amount based on the feature amount of each normal image. In one aspect, the multi-dimensional space may be a one-dimensional space, a two-dimensional space, or a space with three or more dimensions.

In step S1030, CPU 1 defines the boundaries of the normal image. The processing of this step corresponds to the processing of defining the region 200 including all the normal image points described with reference to FIG.

In step S1040, CPU 1 defines boundaries of images that do not require confirmation. An image that does not require confirmation is an image that is excluded from images to be viewed by maintenance personnel. The processing of this step corresponds to the processing of defining a region 700 for excluding an image from the image to be viewed and the processing of generating a filter based on the region 700 described with reference to FIG.

In step S1050, CPU 1 determines whether a new normal image has been added to the normal images obtained in step S1020. When CPU 1 determines that a new normal image has been added to the normal image acquired in step S1020 (YES in step S1050), CPU 1 shifts the control to step S1060. Otherwise (NO in step S1050), CPU 1 shifts the control to step S1080.

In step S1060, CPU 1 redefines the boundary (area 200) of the normal image. The processing of this step corresponds to the processing of updating the area 200 described with reference to FIGS. In step S1070, the CPU 1 defines boundaries of images that do not require confirmation (boundaries of the area 700). Since the boundary of the normal image has changed in step S1060, the CPU 1 redefines the boundary of the confirmation unnecessary image.

In step S1080, CPU 1 determines whether or not an operation to end learning has been accepted. When CPU 1 determines that an operation for ending learning has been received (YES in step S1080), CPU 1 ends the process. Otherwise (NO in step S1080), CPU 1 shifts the control to step S1060.

As described above, the analysis system 20 has a function of calculating a threshold value for distinguishing normal images and abnormal images based on samples of normal images. With this feature, analysis system 20 may provide the ability to classify images as either normal or abnormal, even in situations where the sample of abnormal images is small.

Furthermore, the analysis system 20 has a function of generating a filter for excluding images that are likely to be normal images from the images to be viewed. With these functions, the analysis system 20 can reduce the number of man-hours required for visual inspection of images by maintenance personnel.

It should be considered that the embodiments disclosed this time are illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the scope of the claims rather than the above description, and is intended to include all modifications within the meaning and scope of equivalents of the scope of the claims. In addition, it is intended that the disclosure content described in the embodiment and each modified example can be implemented singly or in combination as much as possible.

1 CPU, 2 primary storage device, 3 secondary storage device, 4 external device interface, 5 input interface, 6 output interface, 7 communication interface, 10 facility, 20 analysis system, 30 monitoring station, 40 local network, 100 analysis device , 110 terminal, 120 access point, 130 surveillance camera, 200,500,600,700 area, 310,410,810,820,830,840 image, 415 local feature point, 900 information processing apparatus.

Claims (25)

an imaging device for imaging the equipment;
and an analysis device that analyzes the image of the device,
The imaging device transmits the image to the analysis device,
The analysis device is
Receiving an operation of selecting a plurality of normal images indicating that the state of the device is normal from among the plurality of accumulated images;
Extracting a first feature amount and a second feature amount of each of the plurality of normal images,
defining a region including all of the plurality of normal images in a coordinate space consisting of the first feature amount and the second feature amount;
Receiving an operation to select a new normal image from among the plurality of accumulated images,
An analysis system that recalculates the extent of the region based on the lack of the new normal image in the region. Recalculating the extent of the region based on the new normal image not being included in the region includes increasing the radius of the region to include the coordinates of the new normal image. , an analysis system according to claim 1. wherein recalculating the range of the region based on the fact that the new normal image is not included in the region includes expanding the region in the direction of the coordinates of the new normal image. Item 1. The analysis system according to item 1. the region is a circle or an ellipse;
4. The analysis system according to any one of claims 1 to 3, wherein said analysis device excludes said normal image existing within a certain radius from the center of said region from images to be viewed. 5. The analysis system according to any one of claims 1 to 4, wherein said imaging device captures said image of said equipment in a predetermined time zone or in a predetermined sunshine condition. Capturing the image of the device under the predetermined sunshine condition includes determining whether the predetermined sunshine condition is satisfied based on the amount of light in the captured image. 6. The analysis system of claim 5, comprising: The analysis system according to any one of claims 1 to 6, wherein the analysis device generates a model for calculating the first feature amount and the second feature amount by machine learning using the image as learning data. . 8. The analysis system according to any one of claims 1 to 7, wherein said analysis device uses the coordinates of said image as learning data to generate a model defining said region by machine learning. a communication unit for receiving an image from an image capturing device that captures an image of the device;
an input unit that receives an operation;
A control unit that analyzes the image,
the input unit receives an operation of selecting a plurality of normal images indicating that the state of the device is normal from among the plurality of accumulated images;
The control unit
extracting a first feature amount and a second feature amount of each of the plurality of selected normal images;
defining a region including all of the plurality of normal images in a coordinate space consisting of the first feature amount and the second feature amount;
The input unit further receives an operation of selecting a new normal image from among the plurality of accumulated images,
The analysis device, wherein the control unit recalculates the range of the region based on the fact that the new normal image is not included in the region. Recalculating the extent of the region based on the new normal image not being included in the region includes increasing the radius of the region to include the coordinates of the new normal image. 10. The analysis device according to claim 9. wherein recalculating the range of the region based on the fact that the new normal image is not included in the region includes expanding the region in the direction of the coordinates of the new normal image. Item 9. The analysis device according to Item 9. the region is a circle or an ellipse;
12. The analyzing apparatus according to claim 9, wherein said control unit excludes said normal image existing within a certain radius from the center of said region from images to be viewed. 13. The controller according to any one of claims 9 to 12, wherein the control unit transmits an instruction to the imaging device to capture the image of the device in a predetermined time period or in a predetermined sunshine condition. Analysis device according to. Capturing the image of the device under the predetermined sunshine condition includes determining whether the predetermined sunshine condition is satisfied based on the amount of light in the captured image. 14. The analysis device of claim 13, comprising: The analysis device according to any one of claims 9 to 14, wherein the control unit generates a model for calculating the first feature amount and the second feature amount by machine learning using the image as learning data. . The analysis device according to any one of claims 9 to 15, wherein the control unit generates a model defining the region by machine learning using the coordinates of the image as learning data. imaging the device to obtain an image;
a step of receiving an operation of selecting a plurality of normal images indicating that the state of the device is normal from among the plurality of accumulated images;
extracting a first feature amount and a second feature amount of each of the plurality of normal images;
defining a region including all of the plurality of normal images in a coordinate space composed of the first feature amount and the second feature amount;
receiving an operation to select a new normal image from among the plurality of accumulated images;
and recalculating the extent of the region based on the fact that the new normal image is not included in the region. Recalculating the extent of the region based on the new normal image not being included in the region includes increasing the radius of the region to include the coordinates of the new normal image. 18. The analysis method according to claim 17. The step of recalculating the range of the region based on the fact that the new normal image is not included in the region includes expanding the region in the direction of the coordinates of the new normal image. Item 18. The analysis method according to Item 17. the region is a circle or an ellipse;
20. The analysis method according to any one of claims 17 to 19, further comprising the step of excluding said normal images existing within a certain radius from the center of said region from images to be viewed. 21. The analysis method according to any one of claims 17 to 20, further comprising the step of capturing said image of said equipment in a predetermined time period or in a predetermined sunshine condition. The step of capturing the image of the device under the predetermined sunshine condition includes determining whether the predetermined sunshine condition is satisfied based on the amount of light in the captured image. 22. The analysis method of claim 21, comprising: The analysis method according to any one of claims 17 to 22, further comprising the step of generating a model for calculating the first feature amount and the second feature amount by machine learning using the image as learning data. 24. The analysis method according to any one of claims 17 to 23, further comprising the step of generating a model defining said region by machine learning using coordinates of said image as learning data. A program for causing a computer to execute the method according to any one of claims 17 to 24.

Images