JP6833452B2 - Patrol inspection system, information processing device, patrol inspection control program - Google Patents

Description

The present invention relates to a patrol inspection system, an information processing device, and a patrol inspection control program using a flying object or the like.

Conventionally, for periodic inspections of power transmission equipment such as overhead power transmission lines (hereinafter referred to as power transmission lines) and patrols in the event of an accident, a patrolman rides on a helicopter and flies along the power transmission line to see the power transmission line with binoculars. However, he was checking for abnormalities and checking the power transmission equipment while moving on foot.

In addition, an image of a transmission line is taken from a helicopter, and a patrolman looks at the taken image to inspect the transmission line for a broken wire or an arc mark (lightning mark).

Japanese Unexamined Patent Publication No. 2005-57956

However, inspection / patrol by the patrolman with binoculars is a burden on the patrolman, and there is a risk that an abnormal part may be overlooked.

In addition to the cost of using a helicopter, inspection / patrol of power transmission equipment using a helicopter requires a plurality of patrols in addition to the helicopter pilot, so the cost for inspection / patrol is high. In addition, if an inspection / patrol is suddenly required, it cannot be handled simply and quickly.

In addition, the inspection work of the transmission line by taking an image also has a problem that a quick and accurate inspection cannot be performed because an abnormal part is detected while the patrolman is looking at all the taken images. Furthermore, a technique for detecting wire breaks and arc marks on an image using image processing technology has been considered, but the quality of the captured image is not constant due to fluctuations in the shooting environment and is detected. There are various states of abnormal parts such as broken wires and arc marks, which make it difficult to stably detect abnormal parts.

An object to be solved by the present invention is to provide a patrol inspection system, an information processing device, and a patrol inspection control program capable of stably detecting an abnormal portion generated in a power transmission facility by image processing.

According to the embodiment, the patrol inspection system patrols and inspects the object to be inspected based on the image data taken by the camera mounted on the aircraft or the like. The patrol inspection system includes a management device and an information processing device. The management device has a learning model generating means for generating a learning model for detecting an abnormality occurring in the object to be inspected from the image. The information processing device includes a storage means, an input means, an abnormal portion detecting means, and an abnormal portion displaying means. The storage means stores the learning model generated by the management device. The input means inputs the image data. The abnormality location detecting means detects the image data of the abnormality portion where the abnormality has occurred in the inspected object from the image data based on the learning model. The abnormal portion display means displays an image of the abnormal portion based on the image data of the abnormal portion. The learning model generating means is based on a similar image obtained by combining an abnormal image showing an abnormal portion generated in the object to be inspected and at least one normal image that is not an abnormal portion around the abnormal portion. Generate a learning model.

The block diagram which shows the structure of the patrol inspection system in this embodiment. The block diagram which shows the structure of the patrol inspection system in this embodiment. The block diagram which shows the structure of the information processing apparatus in this embodiment. The block diagram which shows the functional structure in the information processing apparatus in this embodiment. The figure which shows an example of the appearance of the drone in this embodiment. The block diagram which shows the main structure of the drone in this embodiment. The block diagram which shows the functional structure in the management apparatus in this embodiment. The flowchart for demonstrating the learning model generation process executed by the management apparatus in this embodiment. The figure which shows an example of the region of an abnormal image and the region of a normal image in this embodiment. The figure which shows an example of the region of an abnormal image and the region of a normal image in this embodiment. The figure which shows an example of the region of an abnormal image and the region of a normal image in this embodiment. The figure which conceptually shows the combination of the abnormal image and the normal image by the normal image / abnormal image combination control part in this embodiment. The figure which shows the relationship between the generated image condition and the recall rate stored by the index / condition-related storage part in this embodiment. The figure which shows the relationship between the generated image condition stored by the index / condition-related storage part in this embodiment, and the precision rate. The figure which shows an example of the patrol / inspection route selection screen in this embodiment. The figure for demonstrating the flight plan of the drone in this embodiment. The figure for demonstrating the flight plan of the drone in this embodiment. The figure which shows an example of the object to be inspected provided in the power transmission facility in this embodiment. The figure which shows an example of the object to be inspected provided in the power transmission facility in this embodiment. The figure which shows an example of the object to be inspected provided in the power transmission facility in this embodiment. The figure which shows an example of the flight plan (flight control data) generated by the flight plan processing part in this embodiment. The figure which shows an example of the acquired data generated by the acquired data generation part in this embodiment. The figure which shows an example of the acquired data generated by the acquired data generation part in this embodiment. The figure which shows an example of the inspection target item selection screen in this embodiment. The flowchart which shows the acquisition data management processing in the information processing apparatus in this embodiment. The flowchart explaining the abnormality place detection process in this embodiment. The figure which shows an example of the abnormality candidate image display screen in this embodiment. The figure which shows an example of the abnormality part confirmation screen in this embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
1 and 2 are block diagrams showing the configuration of the patrol inspection system 1 according to the present embodiment. In the patrol inspection system 1 of the present embodiment, for example, regarding the power transmission equipment (transmission line, steel tower) 17 for transmitting power from a power plant or a substation, an aircraft, for example, an unmanned aerial vehicle (hereinafter referred to as a drone 14) is used during normal times or when an accident occurs. It is a system for patrol inspection using (referred to as). The patrol inspection system 1 is based on the image data taken by the drone 14 and the camera mounted on the drone 14, and a plurality of different objects to be inspected (for example, overhead ground wire, transmission line, etc.) included in the power transmission facility 17. It is composed of a patrol inspection control system 2 that detects abnormal parts that occur in insulators, flash indicators, arc horns, steel towers, steel tower foundation concrete, etc., and records the detection results as a patrol inspection result report. In the present embodiment, the patrol inspection control system 2 is composed of, for example, a management device 12 connected to the information processing device 10 via a network 15. However, it is also possible to realize the patrol inspection control system 2 as one device. Further, the management device 12 is realized by cloud computing, and even if it is realized by one server connected via the network 15 (Internet) or a plurality of servers operating in cooperation with each other. good.

As shown in FIGS. 1 and 2, the patrol inspection system 1 includes a patrol inspection control system 2 (information processing device 10, management device 12), and a drone 14.
The information processing device 10 manages / controls the navigation of the drone 14, organizes / manages the acquired data (including image data (moving image, still image), position data, etc.) acquired from the drone 14, and acquires from the drone 14. Performs processing such as patrol / inspection management that summarizes the processing results for the data. The information processing device 10 is used by being transported to the vicinity of the power transmission facility 17 to be inspected by the drone 14 together with the drone 14, and detects an abnormal portion generated in the inspected object based on the image data taken by the drone 14. , Confirmation process of the image of the abnormal part is executed.

The management device 12 generates a learning model used for image processing for detecting an abnormal portion generated in an inspected object based on the image data in the information processing device 10, and generates a processing result created in the information processing device 10. Perform recording, etc.

The drone 14 has position data and the like generated by using GNSS (Global Navigation Satellite System) based on a flight plan (flight control data) for patrolling the power transmission facility 17 set by the information processing device 10. For example, it makes an autonomous flight using position data or the like generated from a GPS signal received from a GPS (Global Positioning System) satellite 16, takes a picture of a power transmission facility 17 to be patrol-inspected, and stores image data. The drone 14 can also receive flight control data from the information processing device 10 in real time during flight and fly by manual control according to the operation of the information processing device 10 by the operator.

FIG. 3 is a block diagram showing the configuration of the information processing device 10 according to the present embodiment. The information processing device 10 is realized by, for example, a personal computer. The information processing device 10 includes a processor 10a, a memory 10b, a storage device 10c, an input / output interface (I / F) 10d, a display device 10e, an input device 10f, a wireless communication device 10g, and a communication device 10h.

The processor 10a executes a basic program (OS) or an application program stored in the memory 10b to realize various functions. For example, the processor 10a manages the flight of the drone 14 by executing the patrol inspection control program, and detects an abnormal part generated in the inspected object of the power transmission facility based on the image taken by the drone 14. , The process of summarizing the patrol inspection results based on the detection results is executed (see FIG. 4).

The memory 10b stores a program executed by the processor 10a, temporary data, and the like.

The storage device 10c stores various programs and various data. The data stored in the storage device 10c includes flight control data for controlling the navigation of the drone 14, acquired data (image data, position data) received from the drone 14, and abnormalities caused in the object to be inspected from the image data. Includes learning model data used for image processing to detect locations, image data of abnormal locations, patrol inspection data summarizing patrol inspection results, and the like.

The input / output I / F10d is an interface for transmitting / receiving data to / from an external device. The input / output I / F10d can input / output data via, for example, a portable memory medium.

The display device 10e is an LCD (Liquid Crystal Display) or the like, and displays a screen corresponding to the processing of the processor 10a. The input device 10f is a keyboard, a pointing device, or the like, and is operated by an operator or the like.

The wireless communication device 10g controls wireless communication, and controls wireless communication with the drone 14 or with a base station housed in the wireless public network.

The communication device 10h controls communication with an external device, for example, a management device 12, a drone 14, and another information processing device 10 through the network 15. The network 15 includes a wireless or wired WAN (Wide Area Network), a LAN (Local Area Network), and the like.

FIG. 4 is a block diagram showing a functional configuration realized by executing a patrol inspection control program by the processor 10a in the information processing apparatus 10 according to the present embodiment.

The information processing device 10 realizes the functions of, for example, the navigation management unit 20, the acquired data organization management unit 21, and the patrol inspection management unit 22 based on the patrol inspection control program.

The navigation management unit 20 manages / controls the navigation of the drone 14, and includes a patrol inspection instruction input unit 20a and a flight plan processing unit 20b.

The patrol inspection instruction input unit 20a inputs an instruction regarding the content of the patrol inspection, for example, by an input operation to the input device 10f by an operator, for example, according to a selection for a selection menu displayed on the display device 10e. The contents of the patrol inspection include, for example, designation of the power transmission equipment 17 to be inspected, designation of the object to be inspected included in the power transmission equipment 17 to be inspected, and abnormal types (arc marks, etc.) to be inspected.

The flight plan processing unit 20b causes the drone 14 to navigate a flight plan according to the contents of the patrol inspection input in the patrol inspection instruction input unit 20a, that is, a route capable of photographing the object to be inspected for the patrol inspection. Generate flight control data. The patrol inspection instruction input unit 20a can generate a flight plan by using information acquired from the outside through the network 15, such as weather information and map information.

The acquired data organization management unit 21 organizes / manages the acquired data (including image data (moving image, still image), position data, etc.) acquired from the drone 14, and is a learning model storage unit 21a, a detection target. Instruction input unit 21b, abnormality location detection unit 21c, acquisition data storage unit 21d, image confirmation unit 21e, abnormality location display unit 21f, abnormality location storage unit 21g, confirmation instruction input unit 21h, confirmation data storage unit 21k, and abnormality location image. Includes a data storage unit 21m.

The learning model storage unit 21a stores the learning model generated by the management device 12. The learning model is for detecting an abnormality occurring in the inspected object from the image data acquired from the drone 14, and represents a feature of the abnormal state occurring in the inspected object. In the management device 12, an abnormality has occurred in the object to be inspected so that the image data is not constant due to changes in the shooting environment taken by the drone 14, or the abnormal portion to be detected is in various states. A learning model that can reliably detect the location is generated (details of the generation of the learning model will be described later). As an abnormality that has occurred in the object to be inspected, for example, in the case of a transmission line, there is an arc mark. Since the arc mark is an abnormality that occurs when there is a lightning strike on the transmission line, it is not an event that occurs frequently, and there are few samples of image data that capture the actual state. Therefore, it is difficult to reliably detect the abnormal portion from the image data taken by the drone 14 only by the learning model based on the image data of the abnormal portion (arc mark) actually generated. In the present embodiment, in the management device 12, not only the image data of the abnormal portion actually generated but also the pseudo image (similar image) showing the abnormal / deteriorated state occurring in the inspected object is deeply utilized by the neural network. It is generated using learning (deep learning) technology. In the present embodiment, the combination of the image of the abnormal part (abnormal image) and the normal image around the abnormal part and the ratio of combining the normal image and the abnormal image are changed by using the technique of deep learning (deep learning). By generating similar images while learning (generating) a learning model based on a large number of similar images. Therefore, by storing the learning model generated by the management device 12 in the learning model 21a and searching for an abnormal part from the image data using the learning model stored in the learning model 21a, it surely occurs in the object to be inspected. It is possible to detect an abnormal part.

The detection target instruction input unit 21b inputs an instruction regarding a detection target such as an object to be inspected included in the power transmission equipment 17 to be patrol-inspected and an abnormality type to be detected according to an input operation of the patrol staff.

The abnormality location detection unit 21c is a detection target by image processing based on the learning model (characteristics of the abnormality portion of the object to be inspected) stored in the learning model 21a from the acquired data stored in the acquisition data storage unit 21d. The abnormal portion corresponding to the instruction of the detection target input by the instruction input unit 21b is detected.

The acquired data storage unit 21d stores the acquired data (see FIG. 23) acquired from the drone 14. The acquired data includes, for example, moving image data (still image data may be used) obtained by photographing the inspected object, position data (latitude, longitude, altitude) indicating the position of the drone 14 at the time of image taking, and inspected object (patrol / patrol / Includes patrol / inspection target data for determining the shooting period (or start timing) for each inspection target).

The image confirmation unit 21e displays the acquired data stored in the acquired data storage unit 21d or an image according to the processing result by the abnormality location detection unit 21c. The image confirmation unit 21e includes an abnormal portion display unit 21f. The abnormal part display unit 21f is an abnormality candidate image display screen (see FIG. 27) for displaying an image (abnormal candidate image) of a portion as a candidate for an abnormal part detected from image data by the abnormal part detecting unit 21c, and an abnormal part. An abnormality location confirmation screen (see FIG. 28) for the worker or the like to finally confirm the candidate abnormality candidate image is displayed.

The abnormal location storage unit 21g stores the abnormal location image data as a candidate for the abnormal location detected from the image data by the abnormal location detection unit 21c. The abnormal location image data stored in the abnormal location storage unit 21g is read out by the acquired data organizing management unit 21 and displayed on the abnormal location candidate image display screen or the abnormal location confirmation screen.

The confirmation instruction input unit 21h inputs a confirmation instruction by a worker or the like to the abnormality candidate image display screen or the abnormality location confirmation screen. On the abnormality location confirmation screen, as a result of the worker confirming the abnormality candidate image, it is possible to instruct whether the abnormality location (defect) or erroneous recognition (no abnormality), and for the abnormality location, It is possible to indicate the type of abnormality (for example, arc mark, broken wire, charcoal adhesion, gloss from which charcoal is peeled off) and the like.

On the abnormality location confirmation screen, the confirmation data storage unit 21k includes image data (abnormal location image data) in which the abnormality candidate image is confirmed to be an abnormality location, an object to be inspected to be detected, an abnormality type to be detected, and the abnormality type. Each data indicating the detection position, detection date and time, etc. is stored as confirmation data. The confirmation data is used as patrol inspection data in, for example, the patrol inspection management unit 22.

The abnormality location image data storage unit 21m stores the abnormality location image data, the data to be inspected to be detected, and the data indicating the abnormality type to be detected in order to be used in the management device 12 for generating the learning model.

The patrol inspection management unit 22 performs patrol / inspection management that summarizes the processing results of the data acquired from the drone 14, and includes the patrol inspection data storage unit 22a and the patrol inspection result report creation unit 22b.

The patrol inspection data storage unit 22a stores patrol inspection data as a processing result by the acquired data organization management unit 21. The patrol inspection data includes, for example, the object to be inspected in which the abnormal part is detected, the image data of the abnormal part confirmed to be the abnormal part, the type of abnormality in the abnormal part, and the position where the abnormal part is detected (position in the inspected object, subject). Includes the position (latitude, longitude) of the abnormal part in the inspection object).

The patrol inspection result report creation unit 22b creates patrol inspection result report data summarizing the patrol inspection results based on the patrol inspection data stored in the patrol inspection data storage unit 22a.

FIG. 5 is a diagram showing an example of the appearance of the drone 14 in the present embodiment.
The drone 14 shown in FIG. 5 has a drone body 14a for accommodating a control unit and the like, four rotors 14b symmetrically provided upward from the drone body 14a, and a variable shooting direction below the drone body 14a. It has a camera 14c provided in the above. The rotor 14b is a rotary blade that generates an upward lift by rotation, and the rotation is controlled by a control unit, so that the drone 14 can be moved in the vertical direction and the horizontal direction. It is also possible to hover to continue flying at almost the same position. The camera 14c is a digital camera, a video camera, or the like that captures a still image or a moving image.

FIG. 6 is a block diagram showing a main configuration of the drone 14 in the present embodiment. The drone body 14a of the drone 14 includes a control unit 30a, an acquisition data generation unit 30b, a shooting control unit 30c, a flight control unit 30d, a sensor group 30e, a memory 30f, a GPS data reception unit 30g, a wireless communication unit 30h, and a camera unit 30k. , A drive unit 30 m is provided.

The control unit 30a controls the entire drone 14, and may be configured as a dedicated controller or may be a general-purpose unit including a processor and a memory. The control unit 30a has each functional module of the acquisition data generation unit 30b, the imaging control unit 30c, and the flight control unit 30d, which are realized by executing each control program.

The acquisition data generation unit 30b generates the acquisition data to be provided to the information processing apparatus 10. The acquisition data generation unit 30b is, for example, image data of an image (moving image or still image) taken by the camera 14c, and position data (latitude) indicating the position of the drone 14 at the time of image taking generated by the GPS data receiving unit 30g. , Longitudinal, altitude), and patrol / inspection showing the period (or start timing) of shooting for each object to be inspected (patrol / inspection target) based on the flight control data used for flight control by the flight control unit 30d. Acquired data is generated by associating with the target data (see FIG. 23).

The photographing control unit 30c controls the power transmission equipment 17 to be inspected by the camera 14c while the drone 14 is flying under the control of the flight control unit 30d. The shooting control unit 30c has, for example, a function of controlling the lens, shutter, aperture, focus mechanism, zoom mechanism, etc. of the camera 14c, and adjusting the shooting direction (lens direction). As a function of adjusting the shooting direction, the camera unit 30k may be driven to rotate the lens direction of the camera 14c in the horizontal or vertical direction with respect to the drone body 14a to change the shooting direction, or flight control. The drive of the drive unit 30 m may be controlled through the unit 30d, and the drone 14 may be tilted to change the shooting direction of the camera 14c. The camera 14c enables the drone 14 to capture not only a downward image but also a lateral or upward image.

The flight control unit 30d includes flight control data received from the information processing device 10, position data indicating the current position of the drone 14 generated by the GPS data reception unit 30g, obstacles detected by the sensor group 30e, and the inspected. Based on the position of an object or the like, the drive unit 30 m (motor) is driven to control the drone 14 to fly autonomously. When the flight control unit 30d acquires the flight control data, it acquires the position data from the GPS data receiving unit 30g and controls the drone 14 to fly within a predetermined range from the inspected object of the power transmission equipment 17.

The sensor group 30e acquires various information such as position, altitude, velocity and direction. For example, a 3-axis accelerometer that measures the acceleration of an object, a gyroscope that detects an angle and an angular velocity of an object, and a 3-axis magnetic sensor. Etc. are included. Further, the sensor group 30e includes an obstacle camera for detecting an obstacle or an object to be inspected. By providing a plurality of obstacle cameras (for example, four), the flight control unit 30d processes the image of the obstacle or the inspected object in three dimensions, and the position of the obstacle or the inspected object is determined in the three-dimensional space. It can be made recognizable. This enables flight control such as avoiding contact / collision with an obstacle or an object to be inspected and flying at a predetermined distance to the object to be inspected.

The memory 30f stores various data, for example, flight control data received from the information processing device 10, acquired data generated by the acquired data generation unit 30b, and the like. The memory 30f can be, for example, a removable and portable memory medium, and when the acquired data is provided to the information processing device 10, it is removed from the drone 14 and the information processing device 10 is made to read the data. You can also.

The GPS data receiving unit 30g receives a GPS signal from the GPS satellite 16 and generates position data (latitude, longitude, altitude) and time data at the current position of the drone 14 based on the GPS signal. The position data generated by the GPS data receiving unit 30g is used for autonomous flight control by the flight control unit 30d and generation of acquired data.

The wireless communication unit 30h controls wireless communication with the information processing device 10 or with a base station housed in the wireless public network. The wireless communication unit 30h may transmit the acquired data (image data) captured during the flight of the drone 14 to the information processing device 10 in real time under the control of the control unit 30a. Further, in order to fly the drone 14 by manual control, the wireless communication unit 30h can also receive flight control data according to the operation of the information processing device 10 by the operator during the flight.

The camera unit 30k supports the camera 14c at the lower part of the drone main body 14a, and is driven by the control of the photographing control unit 30c to adjust the photographing direction by the camera 14c.

The drive unit 30m is driven according to the control of the flight control unit 30d, and rotates the plurality of rotors 14b individually. By controlling the rotation of the plurality of rotors 14b individually, it is possible to adjust the posture such as moving the drone 14 in the vertical direction or the horizontal direction, hovering, or tilting the drone body 14a.

Next, the management device 12 in this embodiment will be described.

Assuming that the management device 12 has the same configuration as the information processing device 10 shown in FIG. 3, detailed description thereof will be omitted. The management device 12 realizes various functions for generating a learning model used in image processing in the information processing device 10 by executing a learning model generation program by a processor.

FIG. 7 is a block diagram showing a functional configuration realized by executing a learning model generation program by a processor in the management device 12 in the present embodiment.
Based on the learning model generation program, the management device 12 includes a data storage unit 40a (abnormal portion image data 40b, learning image data 40c, teaching metadata 40d, verification image data 40e), a learning image reading unit 40f, and teaching metadata. Reading unit 40g, edge cutting unit 40h, adjacent normal image search unit 40k, normal image / abnormal image combination control unit 40m, similar image generation unit 40n, trained model management unit 40p, verification image reading unit 40q, inference execution unit 40r, The functions of the index / condition related storage unit 40s, the related correspondence unit 40t, the inquiry unit 40u, and the learning model storage unit 40v are realized.

Next, a learning model generation process for generating a learning model by the management device 12 will be described. FIG. 8 is a flowchart for explaining the learning model generation process executed by the management device 12 in the present embodiment. In the following description, an abnormal portion to be detected is an abnormal portion of an overhead ground wire or an arc mark generated in a transmission line included in the transmission equipment 17, and a learning model for detecting this abnormal portion is generated as an example. .. The same applies to the learning model for detecting the abnormal part occurring in the other inspected object, and detailed description will be omitted assuming that the learning model corresponding to the abnormal part of each inspected object is generated.

In the present embodiment, since there are few samples of the image data of the arc mark generated by the lightning strike on the overhead ground line or the transmission line, not only the abnormal part image data of the abnormal part (arc mark) actually generated but also the abnormal part image data , Deep learning using a neural network by generating a pseudo image (similar image) based on the image data (learning image data) that captures the abnormal / deteriorated state that is artificially added to the object to be inspected. A large number of learning models are trained (generated) using the technology of deep learning).

The data storage unit 40a stores in advance the data necessary for generating the learning model in the management device 12. The data stored by the data storage unit 40a includes the abnormal portion image data 40b, the learning image data 40c, the teaching metadata 40d, and the verification image data 40e.

The abnormal portion image data 40b is actual image data of the abnormal portion acquired by actually performing a patrol inspection of the inspected object using the information processing device 10 and the drone 14. As the abnormality location image data 40b, image data in which the abnormality location and the abnormality type are confirmed by an operator or the like from the image data taken by the drone 14 is used.

The learning image data 40c is learning pseudo image data obtained by capturing a pseudo-added abnormal / deteriorated state. The learning image data 40c is provided with, for example, a state in which the wire is cut, a state in which the transmission line is charred, a state in which the charcoal is removed and the transmission line is shining (glossy), and the like are added to the transmission line with a tool or the like. , Generated by photographing this power line.

The teaching metadata 40d is data necessary for generating a similar image (learning model) instructed (taught) by an operation of a worker or the like with respect to the abnormal portion image data 40b and the learning image data 40c. The teaching metadata 40d includes, for example, a region including an object to be inspected in an image, a region including a portion corresponding to an abnormal portion (abnormal image), a type such as abnormality / deterioration (abnormal type), and a region distinguishing from a detection target (abnormal type). Data indicating the area to be excluded from processing) is included.

The generation of the learning image data 40c and the teaching metadata 40d is executed by the management device 12 or the information processing device 10. For example, when it is executed in the information processing apparatus 10, an image is displayed based on the abnormal portion image data 40b or the learning image data 40c, and instructions (teaching) about the area for this image are given according to the operation of a worker or the like. And enter.

For example, when the object to be inspected is a power transmission line, a worker or the like operates a pointing device or the like to set a rectangular area (for example, diagonal of a rectangle) to include the entire power transmission line in the image as the area to be inspected. (Positions indicating the two vertices) are taught, and the portion of the abnormal part (abnormal image) such as the arc mark on the transmission line is taught in the rectangular area. Further, the worker or the like instructs to add a label of the arc mark as information indicating the type of abnormality such as abnormality / deterioration. In addition, if there is an area that you want to clearly distinguish from the arc mark of the abnormal part such as the snow accretion ring attached to the power transmission line, the worker etc. can teach the area corresponding to the snow accretion ring in the same way. it can.

The information processing device 10 generates teaching metadata 40d in response to an instruction (teaching) input by an operation of a worker or the like, and creates a file associated with each of the abnormal portion image data 40b or the learning image data 40c. Remember.

The verification image data 40e is image data for verification for evaluating the recognition accuracy of the abnormal portion using a learning model generated based on the abnormal portion image data 40b, the learning image data 40c, and the teaching metadata 40d. is there. The verification image data 40e includes, for example, a plurality of image data.

Hereinafter, a case where a learning model is generated using the learning image data 40c will be described as an example. It should be noted that the case where only the learning image data 40c is used, the case where the learning image data 40c and the abnormal portion image data 40b are used together, and the case where only the abnormal portion image data 40b is used can be arbitrarily selected. You can do it.

First, the learning image reading unit 40f reads the learning image data 40c stored in advance in the data storage unit 40a (step A1). Further, the teaching metadata reading unit 40g reads the teaching metadata 40d corresponding to the learning image data 40c read by the learning image reading unit 40f.

The edge cutting unit 40h uses the teaching metadata 40d read by the teaching metadata reading unit 40g with respect to the learning image data 40c read by the learning image reading unit 40f, for example, an image processing method for edge detection. The outline of the transmission line is cut out using (step A2). That is, the edge cutting portion 40h cuts out a region corresponding to the transmission line from the region including the entire transmission line in the learning image shown by the teaching metadata 40d.

The edge cutting portion 40h can not only detect the edge of the entire contour of the region corresponding to the transmission line, but can also cut out the region corresponding to the transmission line by a simpler process as described below. The extent to which the entire transmission line is included is taught by the rectangular area. The image of the transmission line included in this rectangular area exists by descending to the right with the upper left as one of the vertices or rising to the right with the lower left as one of the vertices. The outline of the image of the transmission line can be approximated by a straight line when viewed locally. Therefore, for example, by detecting edges at at least two locations separated in the left-right direction, it is determined whether the image of the transmission line included in the rectangular region is downward-sloping or upward-sloping based on the position of the edge. can do. Then, a straight line passing through the position of the edge detected in the left-right direction can be regarded as the outline of the image of the approximated transmission line. As a result, it is not necessary to detect the entire contour of the region of the transmission line at the edge, so that the processing load can be reduced.

Next, the adjacent normal image search unit 40k sets a region corresponding to the abnormal portion indicated by the teaching metadata 40d, that is, a portion of the abnormal portion (abnormal image region) such as an arc mark (step A3), and sets the abnormal image region. Based on this, a plurality of normal image areas around the abnormal image in the image of the object to be inspected (inside the contour of the transmission line) are searched (step A4).

For example, the adjacent normal image search unit 40k searches for a normal image area with a rectangular area having the same size as the rectangular area indicating the abnormal image area indicated by the teaching metadata 40d. Here, the adjacent normal image search unit 40k targets, for example, a region excluding a region distinguished from the detection target indicated by the teaching metadata 40d (a region excluded from processing such as a snow accretion ring), and has the following priority. A plurality of normal image regions are searched based on the degrees (1) to (4).

(1) Normal image area closest to either the left or right (usually adjacent) of the rectangular area of the abnormal image, (2) Normal image area closest to either the top or bottom of the rectangular area of the abnormal image (usually adjacent) , (3) Normal image area located in any of the upper left, upper right, lower left, and lower right of the rectangular area of the abnormal image, (4) Normal image closest to (usually adjacent to) the previously searched normal image area region.

9, 10 and 11 show an example of an abnormal image region 56 indicated by the teaching metadata 40d and a plurality of normal image regions 58 searched based on the positions of the abnormal image region 56. There is. As shown in FIGS. 9 to 11, the area 52 including the power transmission line 50 in the image, the area 56 including the part corresponding to the abnormal part (abnormal image), and the area 54 distinguishing from the detection target (the area of the snow accretion ring). ) Is indicated by the teaching metadata 40d. The area 56 may be designated by one rectangular area or a plurality of rectangular areas may be adjacent to one abnormal place. Further, FIGS. 9, 10 and 11 show an example in which the area 56 is designated (taught) by a square. In the example shown in FIG. 9, one abnormal portion is indicated by two regions 56, in the example shown in FIG. 10, four regions 56 are also indicated, and in the example shown in FIG. 11, one region 56 is also indicated.

In FIG. 9, according to the priority described above, the area 58 of the normal image adjacent to the left and right of the area 56 is searched, the searched area 58 and the area 58 of the normal image adjacent to the left and right are searched, and so on. Areas 58 adjacent to the left and right are being searched. In FIG. 9, the area 54 (snow accretion ring) to be distinguished from the detection target is indicated. The area 58 is being searched with the area 54 out of the search target.

FIG. 10 shows an example in which the normal image region 58 located at the lower left or upper right of the abnormal image region 56 is searched according to the above-mentioned priority. In FIG. 11, according to the priority described above, the normal image region 58 is searched to the left and below the abnormal image region 56, and the normal image region 58 adjacent to the left of the searched region 58 is searched. ..
In the examples shown in FIGS. 9 to 11, the areas 56 and 58 are described as squares, but they may be rectangular. Further, the adjacent normal image search unit 40k may extract all the normal images that can be extracted from the image of the object to be inspected (transmission line 50), and is necessary for the processing of the normal image / abnormal image combination control unit 40m. It may be a predetermined number of sheets.

Next, the normal image / abnormal image combination control unit 40m determines the ratio of combining the abnormal image of the region 56 indicated by the teaching metadata 40d and the abnormal image of the plurality of regions 58 searched by the region adjacent normal image search unit 40k. Set (step A5). For example, the normal image / abnormal image combination control unit 40m shall change the ratio of combining the normal image and the abnormal image stepwise, for example, from 1: 1 to 10: 1.

The similar image generation unit 40n uses a deep learning technique utilizing a neural network, and depending on the ratio of combining the normal image and the abnormal image set by the normal image / abnormal image combination control unit 40m, A plurality of normal images searched by the adjacent normal image search unit 40k and an abnormal image indicated by the teaching metadata 40d are combined (step A6), and a pseudo image (similar image) showing a state of abnormality / deterioration occurring in the object to be inspected. Is generated (step A7). A pseudo image that combines a normal image and an abnormal image can be generated by using, for example, an existing image processing method.

For example, when the ratio of combining the normal image and the abnormal image is 1: 1, one normal image is selected from a plurality of normal images and combined with respect to one abnormal image. A plurality of pseudo images can be generated by changing the target to be selected from a plurality of normal images and combining each with an abnormal image.

Further, not only the abnormal image and the normal image included in the learning pseudo image shown by one learning image data 40c are combined, but also the abnormal image and the normal image included in the plurality of other learning pseudo images are combined. It can also be combined. Further, as long as the ratio of the normal image and the abnormal image is matched, it is possible to combine a plurality of normal images and a plurality of abnormal images. For example, when the ratio of combining the normal image and the abnormal image is 1: 1, it is possible to combine the two normal images and the two abnormal images. In this case, the combination of the two normal images can be changed from three or more normal images, and similarly, the combination of the two abnormal images can be changed from three or more abnormal images. .. Therefore, the number of combinations of the plurality of normal images and the plurality of abnormal images can be increased by changing the combination of the normal image and the abnormal image.

FIG. 12 conceptually shows the combination of the abnormal image and the normal image by the normal image / abnormal image combination control unit 40m in the present embodiment. As described above, by combining the plurality of abnormal images (A) and the plurality of normal images (B), a large number of similar images can be generated.

As described above, the similar image generation unit 40n sets the normal image and the abnormal image by deep learning for each of the ratios of combining the normal image and the abnormal image which are changed stepwise by the normal image / abnormal image combination control unit 40m. To generate similar images (steps A5 to A7).

The trained model management unit 40p learns similar images according to the ratio of the normal image and the abnormal image generated by the similar image generation unit 40n (step A9). That is, the trained model management unit 40p associates the similar image generated by deep learning with the condition for generating the similar image, for example, the condition (generated image condition) such as the ratio of the abnormal image and the normal image combined. , Stored / managed in the learning model storage unit 40v.

In the process of generating a series of similar images described above, similar images (learning models) corresponding to each of the abnormal types occurring in the inspected object are generated. For example, it is assumed that similar images are individually generated and learned for each of the subdivided categories such as wire breakage, charcoal adhesion, and charcoal peeling gloss for arc marks generated in overhead ground wires and power transmission lines.

When a similar image (learning model) is learned by the trained model management unit 40p, the inference execution unit 40r causes the verification image reading unit 40q to read the verification image data 40e from the data storage unit 40a (step A10), and the verification image. An index representing the accuracy (recognition accuracy) of authentication using the learning model for the data 40e (verification image) is inferred (step A11). That is, the trained model management unit 40p is generated for a plurality of verification images in which an image including an abnormal portion and an image not including an abnormal portion are mixed, for example, for an abnormal type of an abnormal portion included in the verification image. The abnormal part detection (authentication) process is executed using the learning model. Here, the inference execution unit 40r executes the detection (authentication) process of the abnormal portion by using each of the learning models having different generated image conditions (ratio of the combination of the abnormal image and the normal image) that generated similar images. By doing so, the index of authentication accuracy for each generated image condition is inferred.

The inference execution unit 40r obtains, for example, "reproducibility" and "adaptation rate" as indexes representing recognition accuracy. In addition, other indicators may be obtained.

The "reproduction rate" indicates a ratio of not overlooking an abnormal part generated in an object to be inspected (transmission line, etc.), and has a meaning of a ratio of reproducing an abnormal part. For example, when an abnormal part is detected (recognized) from 20 verification images as a result of executing an abnormal part detection (authentication) process using a learning model for a verification image including 25 abnormal images. Is a recall rate (a rate not to be overlooked) of 80%.

The "compliance rate" indicates the percentage of abnormal locations (not missed) as a result of detecting abnormal locations on the object to be inspected (transmission line, etc.), and the detected (recognized) abnormal locations. It has the meaning of conformity ratio. For example, when an abnormal part is detected from 20 verification images as a result of executing an abnormal part detection (authentication) process using a learning model, when there are actually 15 verification images including the abnormal part, , The precision rate (rate of not missing) is 75% (in this case, 5 sheets (25%) are missed).

The index / condition-related storage unit 40s generates an index (“recall rate” and “fitness rate”) obtained by the inference execution unit 40r and a learning model (similar image) used for obtaining this index. The relationship with (the ratio of the abnormal image and the normal image combined) is memorized (step A12).

FIG. 13 is a diagram showing the relationship between the generated image condition stored by the index / condition-related storage unit 40s and the recall rate, and FIG. 14 is a diagram matching the generated image condition stored by the index / condition-related storage unit 40s. It is a figure which shows the relevance of a rate. In FIGS. 13 and 14, index values are shown for each category of wire breakage (Cut), charcoal adhesion (Chacoal), and charcoal peeling gloss (Glare) as abnormal types of arc marks.

The inquiry unit 40u desires, for example, the type of abnormality (for example, arc mark of a transmission line, broken wire, charcoal adhesion, gloss, etc.) of an abnormal part to be detected, and the worker or the like according to the operation of the worker or the like. Enter at least one of the recall rate or the precision rate to be specified. The inquiry unit 40u is a learning unit that can obtain, for example, the best detection result of an abnormal part with respect to the related corresponding unit 40t according to the specified abnormality type of the input abnormal part and at least one of the recall rate and the precision rate. Inquire about the model (generated image condition).

In response to the inquiry (query) from the inquiry unit 40u, the related correspondence unit 40t can obtain the best detection result of the abnormal portion based on the index and the generated image condition stored by the index / condition related storage unit 40s. The learning model (generated image condition) is responsive and presented to the worker or the like.

For example, it is assumed that "general arc marks" and "both recall rate / precision rate" are specified, and an inquiry (query) "What is the best generated image condition?" Is input. In this case, when the index and the generated image condition shown in FIGS. 13 and 14 are stored, the related corresponding unit 40t has a recall rate of each of wire breakage (Cut), charcoal adhesion (Chacoal), and gloss (Glare). And the precision rate are used to obtain the generated image condition (ratio of normal image and abnormal image) that gives the best numerical value in total by a predetermined calculation method. As a result, it is required that the best result can be obtained when the ratio of combining the normal image and the abnormal image is 8: 1. That is, in the abnormal part detection process (image processing for detecting the abnormal part) of the information processing apparatus 10, when the abnormal part is detected for "the entire arc mark" from the image of the object to be inspected, the ratio of the normal image to the abnormal image is When a learning model corresponding to 8: 1 is used, the abnormal portion can be detected most satisfactorily.

In addition, it is assumed that "broken wire" and "both recall rate / conformity rate" are specified, and an inquiry (query) "What is the best generated image condition?" Is input. In this case, the related corresponding unit 40t uses the recall rate and the matching rate of the broken wire to determine the generated image condition (ratio of normal image and abnormal image) in which the best numerical value is obtained by a predetermined calculation method. Ask. As a result, it is required that the best result can be obtained when the ratio of the normal image to the abnormal image is 9: 1. That is, in the abnormal part detection process of the information processing apparatus 10, when the abnormal part is detected for "wire breakage" from the image of the object to be inspected, a learning model corresponding to a ratio of the normal image to the abnormal image of 9: 1 is provided. When used, the abnormal part can be detected most satisfactorily.

In this way, based on the result of the inquiry from the inquiry unit 40u, the learning model used for the abnormal portion detection process in the information processing device 10 can be determined and stored in the information processing device 10 (learning model 21a). For example, in the information processing apparatus 10, when "general arc mark", "broken wire", "charcoal adhesion", and "gloss" are individually selected as the abnormality type so that the abnormality portion can be detected, each of them corresponds to each. The learning model is determined. Further, in the information processing apparatus 10, when either the "recall rate" or the "compliance rate" is selected so that the abnormal portion can be detected, either the "recall rate" or the "compliance rate" is selected. The learning model is determined based on this.

In the above description, the inquiry unit 40u responds to the generated image condition in response to the inquiry in response to the instruction input by the operation of the worker or the like. By setting the conditions of the learning model used in the information processing device 10 in advance, the learning model of the generated image conditions satisfying this condition may be transmitted to the information processing device 10. The conditions of the learning model can be the same as the above-mentioned inquiry (designation of at least one of the abnormality type, recall rate, and precision rate).

Further, in the above description, one "best generated image condition" is determined and used as a learning model used in the abnormal portion detection process of the information processing apparatus 10, but for example, a plurality of generated image conditions (for example, a normal image) are used. / Abnormal images may be used in the information processing device 10 by combining the learning models of 9: 1 and 8: 1).

In this way, in the management device 12 in the present embodiment, the technique of deep learning (deep learning) is utilized by using the abnormal image to be detected such as abnormality / deterioration and the normal image in the vicinity of the image. Similar images can be automatically generated efficiently and accurately, and a learning model used for abnormality detection processing in the information processing apparatus 10 can be generated.

The process of generating the learning model in the management device 12 described above is described by taking an abnormal portion (such as an arc mark of a transmission line) generated in the power transmission facility 17 as an example, but is an industrial device / structure or the like. It can be widely applied to systems / devices that discriminate abnormalities / deteriorations by image recognition.

Next, the operation of the patrol inspection by the information processing device 10 and the drone 14 in the present embodiment will be described. Here, for example, it is assumed that an abnormal portion occurs due to a lightning strike on the power transmission facility 17.
For example, when a worker or the like performs a patrol inspection using a drone 14 on a power transmission facility 17, the information processing device 10 and the drone 14 can be mounted on an automobile or the like, and the drone 14 can fly to the power transmission facility 17. Move within a range.

First, the information processing device 10 sets a flight plan according to the power transmission equipment 17 to be inspected. The information processing device 10 displays a patrol / inspection route selection screen 60 for setting a flight plan, as shown in FIG. 15, and allows an operator or the like to select the screen 60. The patrol / inspection route selection screen 60 is provided with, for example, a regular patrol button 61, an accident patrol button 62, and a tower patrol button 63, and any of them can be arbitrarily selected. The accident patrol is executed to identify the cause of the accident by patrol when an accident such as a lightning strike on the power transmission facility 17 occurs. Regular patrols and tower patrols are patrols for regular patrols or inspections.

When the accident patrol is selected by the accident patrol button 62, the flight plan processing unit 20b of the information processing device 10 inputs the selection contents by the patrol inspection instruction input unit 20a, and is provided on, for example, a steel tower as shown in FIG. Generates a flight plan (flight control data) that takes images of the inspected objects (flash indicator, arc horn, insulator, etc.) and the overhead ground wire and power transmission line. Further, when the regular patrol is selected by the regular patrol button 61, the flight plan processing unit 20b similarly captures an image of, for example, an overhead ground wire and a power transmission line as shown in FIG. 17 (flight). Control data) is generated. Although not shown, when tower patrol is selected by the tower patrol button 63, the flight plan processing unit 20b can capture an image of the target tower from the bottom to the top (flight control data). To generate.

Regarding the power transmission equipment 17 to be inspected by patrol, basic information such as the position where the steel tower is installed and the shape of the steel tower is stored in the information processing device 10 in advance, and can be specified by a worker or the like. The flight plan processing unit 20b generates flight control data based on basic information such as the position of the power transmission facility 17 to be patrol-inspected and the flight plan of the patrol-inspection selected on the patrol / inspection route selection screen 60.

In the patrol / inspection route selection screen 60, normal buttons 61a, 62a, 63a and winter buttons 61b, 62b, 63b are provided for each of the regular patrol button 61, the accident patrol button 62, and the tower patrol button 63, respectively. ing.

Generally, a lightning strike is discharged downward from above the power transmission facility 17, so that, for example, when there is a lightning strike on the overhead ground wire, an abnormal portion such as an arc mark is generated on the upper surface side of the overhead ground wire. Therefore, by selecting the normal buttons 61a, 62a, 63a (default), a flight plan is created in which the drone 14 can take an image above the overhead ground wire.

On the other hand, during the winter in the Hokuriku region, winter lightning that is discharged upward from the ground surface may occur. Therefore, if the power transmission equipment 17 is struck by lightning due to winter lightning, an abnormal portion such as an arc mark may occur on the lower surface side of the power transmission line. Therefore, by selecting the winter buttons 61b, 62b, 63b, it is possible to create a flight plan capable of taking an image of the lower surface side of the overhead ground wire or the transmission line.

Note that FIGS. 16 and 17 show an example in which the drone 14 is made to fly above the power transmission facility 17 to take an image including, for example, an overhead ground wire and a plurality of power transmission lines. In order to take images of a plurality of power transmission lines individually, they may be flown back and forth between the towers a plurality of times. In addition, in order to take an image of an abnormal part other than a lightning strike, for example, an abnormal part caused by contact with a crane, a flying object, a tree, etc. from the side surface side of the transmission line, an image is taken from the side surface side of the transmission line. You may fly it.

18, FIG. 19 and FIG. 20 show an example of an object to be inspected provided in the power transmission facility 17. FIG. 18 shows one tower 17a. Normally, overhead ground wires 17b and three transmission lines 17c, 17d, 17e are erected on the tower 17a. The transmission lines 17c, 17d, and 17e are erected on the steel tower 17a via insulators 17h, respectively. FIG. 19 shows an example of the insulator 17h. The insulator 17h is provided with an arc horn 17g in order to prevent the insulator 17h from being destroyed in the event of a lightning strike on the steel tower 17a. Further, the steel tower 17a is equipped with a flash indicator 17f for notifying that there is a lightning strike when there is a lightning strike on the steel tower 17a or the overhead ground wire 17b. FIG. 20 shows an example of the flash indicator 17f. FIG. 20 (A) shows the state of the flash indicator 17f in a normal state, and FIG. 20 (B) shows the flash indicator 17f when there is a lightning strike on the tower 17a. As shown in FIG. 20B, when there is a lightning strike on the steel tower 17a or the overhead ground wire 17b, the flash display 17f opens the lid 17f2 from the display main body 17f1 due to the current flowing through the steel tower 17a and is housed inside. The display cloth 17f3 that has been used is exposed.

In the management device 12, the learning model for the arc horn 17g, the insulator 17h, and the flash indicator 17f is created in the same manner as the learning model for the abnormal state occurring in the transmission line described above, so that these inspected objects can also be inspected. It can be detected from the image taken by the drone 14. For example, for the arc horn 17g, a learning model capable of detecting an arc mark caused by a lightning strike is generated. For insulator 17h, a learning model that can detect damage (cracks) caused by lightning strikes is generated.

FIG. 21 shows an example of a flight plan (flight control data) generated by the flight plan processing unit 20b in the present embodiment. The flight control data is for flying a series of flight routes for photographing the power transmission equipment 17 to be inspected by the drone by the drone 14, and for example, the patrol / inspection target data classified for each patrol / inspection target corresponds. It is attached.

Next, the operation of the drone 14 in the present embodiment will be described with reference to the flowchart shown in FIG.
The control unit 30a of the drone 14 stores the flight control data received from the information processing device 10, and when the flight start is instructed, the flight control unit 30d drives the drive unit 30m to respond to the flight control data. Flight control for flying the route is started (step B1).

Further, the control unit 30a generates the acquired data by the acquired data generation unit 30b and starts the storage. That is, the acquired data generation unit 30b stores the patrol / inspection target data (step B2) indicating the period (or start timing) of shooting for each inspected object (patrol / inspection target), and is photographed by the camera 14c. Recording of image data (here, for example, moving image data) and storage of position data generated by the GPS data receiving unit 30g are started (steps B2, B3, B4).

FIG. 23 shows an example of the acquired data generated by the acquired data generation unit 30b in the present embodiment. As shown in FIG. 23, the acquisition data generation unit 30b includes moving image data (FIG. 23 (A)) taken by the camera 14c, position data (FIG. 23 (C)) generated by the GPS data reception unit 30g, and so on. And the patrol / inspection target data (FIG. 23 (B)) are associated and stored.

Since the first patrol / inspection target is "steel tower B-1", "steel tower B-1" is stored as patrol / inspection target data. The moving image data and the position data are continuously stored. The position data may be stored at predetermined time intervals. Further, instead of the moving image data, the still image data may be continuously stored.

As shown in FIG. 23, by storing the patrol / inspection data, after the shooting by the drone 14 is completed, the recording range for each inspected object included in the moving image data is based on the patrol / inspection data. Can be easily searched. For example, it is possible to search the range M2 of the moving image data in which the "fictitious ground wire" is captured, based on the patrol / inspection target data indicating the "fictitious ground wire". Further, since the moving image data and the position data are stored in association with each other, for example, when an abnormal part is photographed in the range M2 of the moving image data, the position of the drone 14 when the abnormal part is photographed is shown. Based on the position data PE, the position (latitude, longitude) of the abnormal part occurring on the "fictitious ground wire" can be roughly grasped.

In the above description, the acquired data is stored in the drone 14 (for example, the memory 30f), but it is also possible to transmit the acquired data to the information processing device 10 by wireless communication in real time (step B5).

When the flight of the range corresponding to the patrol / inspection target is completed (step B6, Yes), the acquired data generation unit 30b has not completed the flight of the range corresponding to all the patrol / inspection targets (step B7, No.). ), When the patrol / inspection target moves to the next inspection target, new patrol / inspection target data is stored (step B2). Hereinafter, in the same manner, the moving image data and the position data taken for the next patrol / inspection target are stored (steps B3 and B4).

In the same manner thereafter, the recording of the acquired data is continued, and when the flight in the range corresponding to all the patrol / inspection targets is completed (step B7, Yes), the control unit 30a returns the drone 14 to the original flight start position. Flight control is started and the storage of acquired data is completed.

Next, the acquired data management process in the information processing apparatus 10 according to the present embodiment will be described with reference to the flowcharts shown in FIGS. 25 and 26.
The acquired data stored by the drone 14 is transmitted to the information processing device 10 and stored in the acquired data storage unit 21d. The acquired data storage unit 21d can store the acquired data stored by the drone 14 in a plurality of flights. The information processing device 10 starts organizing the acquired data for detecting and confirming the abnormal portion generated in the inspected object based on the image data by the operation of the operator or the like.

First, the information processing device 10 inputs the designation of the patrol / inspection route to be processed through the patrol / inspection route selection screen 60, and selects the processing target from the acquired data stored by the flight of the designated patrol / inspection route. (Step C1). Further, the information processing apparatus 10 displays an inspection target item selection screen 70 for selecting an inspection target object to be inspected, as shown in FIG. 24, and a worker or the like displays one of the inspection target items. Let me choose. In addition to the buttons 71, the inspection target item selection screen 70 is provided with an arc mark button 72, a wire break button 73, a charcoal adhesion button 74, and a gloss button 75 for overhead ground wires and power transmission lines, for example. There is. The inspection target item selection screen 70 is provided with selection target items for designating other objects to be inspected (insulator, arc horn, flash indicator) (not shown).

For example, when all the buttons 71 are selected, the abnormality location detection unit 21c targets all the objects to be inspected taken by the drone 14 on the designated patrol / inspection route, and inspects all the objects to be inspected. A learning model for detection is set (step C3).

Similarly, for example, when the arc mark button 72 is selected, the abnormal portion detection unit 21c sets the arc mark (including each category of wire breakage, charcoal adhesion, and gloss) as a detection target, and the arc mark (wire breakage). , Charcoal adhesion, gloss), and a learning model for detecting (step C3). Similarly, when any of the wire break button 73, the charcoal adhesion button 74, and the gloss button 75 is selected, a learning model for detecting an abnormal portion corresponding to each is set. As described above, the learning model for detecting the abnormal part is combined by the ratio (ratio) of the combination of the normal image and the abnormal image that can detect the abnormal part best for each abnormality type in the management device 12. It is generated based on similar images.

By making it possible to select the inspection target item on the inspection target item selection screen 70, it is possible to execute the abnormality detection processing using the learning model corresponding to the specific abnormality portion, so that the recognition accuracy is improved and the processing is performed. The time can be shortened.

After the setting of the inspection target item (learning model) is completed, the abnormality location detection unit 21c reads the acquired data to be processed stored in the acquired data storage unit 21d, and reads the patrol / inspection target data included in the acquired data. Determine (step C4). Here, the abnormal location detection unit 21c determines whether the patrol / inspection target data corresponds to the detection target of the abnormal location specified by the inspection target item.

For example, on the inspection target item selection screen 70, when the arc mark button 72 for the overhead ground wire and the transmission line is selected as the inspection target item, the patrol / inspection target data may be “steel tower B-1”. For example, it is determined that the abnormal part is not the detection target. When the abnormality location detection unit 21c determines that the abnormality location is not the detection target (steps C5, No), the moving image data is skipped to the patrol / inspection target data (for example, “fictitious ground wire”) to be detected for the abnormality detection. Then, the moving image data in which the detection target of the abnormal portion is captured is set as the processing target. That is, the processing time can be shortened by omitting the processing of the moving image data obtained by photographing the inspected object that is not the detection target of the abnormal portion according to the inspection target item selected on the inspection target item selection screen 70. Try.

The abnormal location detection unit 21c executes an abnormal location detection process for detecting the abnormal location to be detected by using a learning model set according to the inspection target item for the moving image data in which the detection target of the abnormal location is captured. (Step C7).

FIG. 26 is a flowchart showing an abnormal portion detection process in the present embodiment.
The abnormality location detection unit 21c detects a region corresponding to the object to be inspected by existing image processing from the moving image data of the acquired data stored in the acquisition data storage unit 21d, and detects an abnormality image candidate region from that region. That is, the part of the object to be inspected that may be in an abnormal state (shape, color, etc.) different from the surroundings is detected. The abnormality location detection unit 21c determines whether the image data of the abnormality image candidate region is an image corresponding to the abnormality type to be detected by using a learning model showing the characteristics of the abnormality type occurring in the object to be inspected. Step D2).

Here, when it is determined that the image represents the abnormal type (step D3, Yes), the abnormal portion detecting unit 21c is the image data (abnormal portion image data) determined to be the image corresponding to the abnormal type (abnormal image). ) Is cut out, and the position data corresponding to the abnormal location image data is acquired. The abnormality location detection unit 21c stores abnormality type data indicating which abnormality type is detected as an abnormality based on the learning model corresponding to which abnormality type, that is, which abnormality type is more than the location. For example, when the arc mark button 72 is selected on the inspection target item selection screen 70, the determination of the abnormal portion is executed using the learning model corresponding to each category of wire breakage, charcoal adhesion, and gloss. When it is determined as an abnormal part by the learning model corresponding to the broken wire, the abnormal type data indicating "broken wire" is stored.

When the abnormal portion is detected by the abnormal portion detecting process, the abnormal portion detecting unit 21c stores the data of the abnormal portion in the abnormal portion storage unit 21g (step D5). The data of the abnormal part includes, for example, the abnormal part image data, the position data (and time) corresponding to the abnormal part image data, the abnormal type data, and the patrol / inspection target data corresponding to the range of the moving image data including the abnormal part image data. , Includes patrol / inspection routes corresponding to moving image data.

The abnormality location detection unit 21c executes the above-described processing until the acquired data to be processed is completed (steps C4 to C13). If, for example, the abnormal part to be detected is changed through the inspection target item selection screen 70 by an instruction from a worker or the like while the process of detecting the abnormal part is being executed (step C10, Yes). ), The abnormal part detection unit 21c resets the learning model according to the changed abnormal part (inspection target item) to be detected, and restarts the process of detecting the abnormal part in the same manner as described above.

On the other hand, the abnormality location display unit 21f displays an abnormality candidate image display screen for allowing a worker or the like to confirm the detection result based on the abnormality location data stored in the abnormality location storage unit 21g (step C9). ..

FIG. 27 is a diagram showing an example of the abnormality candidate image display screen 80 according to the present embodiment.
On the abnormality candidate image display screen 80, for example, the abnormality type 82 indicated by the abnormality type data is displayed together with the abnormality portion image 81 (abnormality candidate image) based on the abnormality portion image data. Further, on the abnormality candidate image display screen 80, reference information 83 related to the abnormality portion image 81 is displayed. Reference information 83 includes, for example, "Point B-1" indicating a patrol / inspection route, a shooting date / time, a detection position (latitude, longitude, altitude) indicated by position data, and an inspected object "upper feed" indicated by patrol / inspection target data. Includes "electric wire" and "upper" (indicating that the image was taken from the upper part of the transmission line). Further, the reference information 83 is provided with a map button 84. When the map button 84 is selected, the abnormality location display unit 21f displays a map including the detection position (adding the position of the power transmission equipment 17 and the like), and maps the position where the abnormality location of the power transmission equipment 17 is detected. Make it visible above.

When the abnormal part (candidate) is detected from the moving image data by the abnormal part storage unit 21g, the abnormal part display unit 21f displays each information regarding the newly detected abnormal part (candidate) on the abnormality candidate image display screen 80. , As described above, additional displays are sequentially displayed. Therefore, the worker or the like can confirm the detected content at the time when the abnormal portion is detected while executing the process of detecting the abnormal portion in the acquired data.

Further, when the abnormality location display unit 21f selects, for example, the abnormality location image 81 on the abnormality candidate image display screen 80, the worker or the like finally confirms the abnormality portion corresponding to the selected abnormality portion image 81. Display the abnormal part confirmation screen for.

FIG. 28 is a diagram showing an example of the abnormality location confirmation screen 90 in the present embodiment.
On the abnormality part confirmation screen 90, the abnormality part image 91 which enlarged the abnormality part image 81 selected on the abnormality candidate image display screen 80 and the abnormality part image which is not displayed on the abnormality candidate image display screen 80 are reduced. Images 92a, 92b, 92c are displayed.

Further, the reference information 93 related to the abnormal portion image 91 is displayed on the abnormal portion confirmation screen 90. The reference information 93 includes, for example, "Point B-1" indicating a patrol / inspection route, a shooting date and time, and a detection position (latitude, longitude, altitude). Further, on the abnormality location confirmation screen 90, an abnormality location confirmation button 94 for inputting an abnormality type confirmation instruction by a worker or the like, and an erroneous recognition for inputting an instruction confirmed not to be an abnormality location by a worker or the like. A button 95 is provided. The abnormality location confirmation button 94 is provided with buttons corresponding to each of a plurality of abnormality types to which confirmation instructions can be input, such as an arc mark button 94a, a wire break button 94b, a charcoal adhesion button 94c, and a gloss button 94d. There is.

On the abnormal location confirmation screen 90, when any of the direction buttons provided on the left and right of the abnormal location image 91 or the abnormal location images 92a, 92b, 92c is selected, the abnormal location image of another abnormal location is displayed. 91 can be displayed to change the abnormal portion to be confirmed.

As a result of referring to the abnormal portion image 91 displayed on the abnormal portion confirmation screen 90, the worker or the like selects, for example, the wire break button 94b when it is confirmed that the wire is broken. When an abnormality confirmation instruction is input by selecting the wire break button 94b (steps C11, Yes), the confirmation instruction input unit 21h confirms each data indicating the object to be inspected, the abnormality type, the detection position, the detection date and time, and the like. Is stored in the confirmation data storage unit 21k (step C12). The confirmation data stored in the confirmation data storage unit 21k is used, for example, in the patrol inspection management unit 22 as patrol inspection data for creating patrol inspection result report data summarizing the results of the patrol inspection.

Further, the confirmation instruction input unit 21h displays the same abnormality location image data, the inspection target object to be detected, and the abnormality type data to be detected in the management device 12 as the confirmation data stored in the confirmation data storage unit 21k. It is stored in the abnormal location image data storage unit 21m for use in generating a learning model. The data stored in the abnormal location image data storage unit 21m is transmitted to the management device 12 at an arbitrary timing.

If it is confirmed that the abnormal portion is not the abnormal portion as a result of referring to the abnormal portion image 91, the erroneous recognition button 95 is selected by the operator or the like. In this case, the confirmation instruction input unit 21h deletes the corresponding abnormal location image data from the abnormal location storage unit 21g.

In this way, in the information processing apparatus 10, a learning model capable of satisfactorily detecting an abnormal portion to be detected in the inspected object is set according to the inspection target item selected on the inspection target item selection screen 70. .. Since the learning model is generated in the management device 12 by using the deep learning technique including a pseudo image (similar image), the abnormal part can be detected even in a situation where the image sample of the actual abnormal part is small. It can be detected stably by image processing.

Further, the abnormal part image detected as a candidate for the abnormal part by the abnormal part detection process is referred to on the abnormality candidate image display screen 80 or the abnormal part confirmation screen 90, and the worker checks whether or not the abnormal part actually occurs. It can be confirmed by judgment such as. The abnormal part image data determined to be an abnormal part by using the abnormal part confirmation button 94 on the abnormal part confirmation screen 90 is used to generate a learning model and is reflected in the learning model corresponding to the abnormal type to be detected. Can be made to.

Further, the data of the abnormal part confirmed by using the abnormal part confirmation button 94 on the abnormal part confirmation screen 90 is used as the patrol inspection data in the patrol inspection management unit 22, and the patrol inspection result report summarizing the results of the patrol inspection. Data can be created. As a result, it is possible to reduce the workload of workers who record the results of patrol inspections.

Further, since the confirmation of the abnormal portion on the abnormality candidate image display screen 80 or the abnormality candidate image display screen 80 can be executed in parallel with the abnormality portion detection process, the confirmation work can be shortened.

The method described in each of the above embodiments is a program that can be executed by a computer, such as a magnetic disk (hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a magneto-optical disk (MO), a semiconductor memory, or the like. It can also be stored and distributed on the storage medium of. Further, the storage medium may be in any form as long as it is a storage medium capable of storing a program and readable by a computer.

Further, in order for the OS (operating system) running on the computer, the database management software, the MW (middleware) such as the network software, etc. to realize the above embodiment based on the instructions of the program installed on the computer from the storage medium. You may execute a part of each process of.

Further, the storage medium in each embodiment is not limited to a medium independent of the computer, but also includes a storage medium in which a program transmitted by a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

Further, the storage medium is not limited to one, and the case where the processing in each of the above embodiments is executed from a plurality of media is also included in the storage medium in the present invention, and the medium configuration may be any configuration.

The computer in each embodiment executes each process in each of the above embodiments based on a program stored in a storage medium, and is composed of one device such as a personal computer and a plurality of devices in a network. Any configuration such as a connected system may be used.

Further, the computer in each embodiment is a general term for a device and a device capable of realizing the function of the present invention by a program, including an arithmetic processing device, a microcomputer and the like included in the information processing device.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 ... Patrol inspection system, 2 ... Patrol inspection control system, 10 ... Information processing device, 10a ... Processor, 10b ... Memory, 10c ... Storage device, 10d ... Input / output I / F, 10e ... Display device, 10f ... Input device, 10g ... wireless communication device, 10h ... communication device, 12 ... management device, 14 ... drone, 14a ... drone body, 14b ... rotor, 14c ... camera, 20 ... navigation management unit, 20a ... patrol inspection instruction input unit, 20b ... flight Plan processing unit, 21 ... Acquisition data organization management unit, 21a ... Learning model storage unit, 21b ... Detection target instruction input unit, 21c ... Abnormal part detection unit, 21d ... Acquisition data storage unit, 21e ... Image confirmation unit, 21f ... Abnormality Location display unit, 21g ... Abnormal location storage unit, 21h ... Confirmation instruction input unit, 21k ... Confirmation data storage unit, 21m ... Abnormal location image data storage unit, 22 ... Patrol inspection management unit, 22a ... Patrol inspection data storage unit, 22b ... Patrol inspection result report creation unit, 30a ... Control unit, 30b ... Acquisition data generation unit, 30c ... Shooting control unit, 30d ... Flight control unit, 30e ... Sensor group, 30f ... Memory, 30g ... GPS data reception unit, 30h ... Wireless communication unit, 30k ... camera unit, 30m ... drive unit, 40a ... data storage unit, 40b ... abnormal location image data, 40c ... learning image data, 40d ... teaching metadata, 40e ... verification image data, 40f ... learning image Reading unit, 40g ... Teaching metadata reading unit, 40h ... Edge cutting unit, 40k ... Adjacent normal image search unit, 40m ... Normal image / abnormal image combination control unit, 40n ... Similar image generation unit, 40p ... Learned model management unit , 40q ... Verification image reading unit, 40r ... Inference execution unit, 40s ... Index / condition related storage unit, 40t ... Related correspondence unit, 40u ... Inquiry unit, 40v ... Learning model storage unit.

Claims (10)

It is a patrol inspection system for patrol inspection of the object to be inspected based on the image data taken by the camera mounted on the aircraft.
Including management device and information processing device
The management device is
It has a learning model generation means for generating a learning model for detecting an abnormality occurring in the object to be inspected from the image.
The information processing device
A storage means for storing the learning model generated by the management device, and
An input means for inputting the image data and
An abnormality location detecting means for detecting image data of an abnormality portion in which an abnormality has occurred in the object to be inspected from the image data based on the learning model.
The image data of the abnormal part possess an abnormal portion display means for displaying an image of the original to the anomaly,
The learning model generating means is based on a similar image obtained by combining an abnormal image showing an abnormal portion generated in the object to be inspected and at least one normal image that is not an abnormal portion around the abnormal portion. A patrol inspection system that generates a learning model. The information processing device
Further having a setting means for setting flight control data for flying the aircraft to capture an image including the object to be inspected.
The patrol inspection system according to claim 1, wherein the abnormal portion detecting means detects image data of the abnormal portion based on the learning model corresponding to the object to be inspected taken by flight based on the flight control data. .. The information processing device
Further, it has an instruction input means for inputting an instruction indicating an abnormality type to be detected according to the object to be inspected.
The patrol inspection system according to claim 1, wherein the abnormal portion detecting means detects image data of an abnormal portion corresponding to the abnormal type based on the learning model corresponding to the abnormal type. The learning model generation means is generated based on a similar image obtained by combining the abnormal image and the normal image at a plurality of different ratios and combining them according to any ratio for each type of abnormality occurring in the object to be inspected. The patrol inspection system according to claim 1 , wherein a learning model is selected. The abnormal location display means displays a confirmation screen for displaying an image of an abnormal location candidate and an abnormal type based on the image data of the abnormal location detected by the abnormal location detecting means, and gives a confirmation instruction to the confirmation screen. The patrol inspection system according to claim 1, wherein the data is input. The abnormal part display means
Based on the image data of the abnormal portion detected by the abnormal location detecting means, a confirmation screen for displaying the image of the abnormal location candidate and the abnormality type is displayed, and a confirmation instruction for the confirmation screen is input.
The patrol inspection system according to claim 1 , wherein the image of the abnormality location candidate confirmed by the confirmation instruction is stored as an abnormality image used for generating a similar image by the learning model generation means. The information processing device inputs the image data and position data indicating the position of the aircraft at the time of shooting from the aircraft, and corresponds to the image data of the abnormal portion, the abnormality type, and the image data of the abnormal portion. The patrol inspection system according to claim 3, further comprising a patrol inspection result report creating means for generating patrol inspection result data including position data. It is an information processing device for patrol inspection of the object to be inspected based on the image data taken by the camera mounted on the aircraft.
A storage means for storing a learning model for detecting an abnormality occurring in the object to be inspected from the image,
An input means for inputting the image data and
An abnormality location detecting means for detecting image data of an abnormality portion in which an abnormality has occurred in the object to be inspected from the image data based on the learning model.
And abnormal location display means for displaying the image data of the anomaly,
A learning model that generates the learning model based on a similar image that combines an abnormal image showing an abnormal part generated in the object to be inspected and at least one normal image that is not an abnormal part around the abnormal part. An information processing device having a generation means and. Computer
Storage means for storing a learning model for detecting an abnormality occurring in the point test object from an image,
Input means for inputting image data and
An abnormality location detecting means for detecting image data of an abnormality portion in which an abnormality has occurred in the object to be inspected from the image data based on the learning model.
And abnormal location display means for displaying the image data of the anomaly,
A learning model that generates the learning model based on a similar image that combines an abnormal image showing an abnormal part generated in the object to be inspected and at least one normal image that is not an abnormal part around the abnormal part. patrol inspection control program for functioning as a generator. It is an information processing device for inspecting the object to be inspected based on the image data taken by the camera.
A storage means for storing a learning model for detecting an abnormality occurring in the object to be inspected from the image,
An input means for inputting the image data and
An abnormality location detecting means for detecting image data of an abnormality portion in which an abnormality has occurred in the object to be inspected from the image data based on the learning model.
Anomalous part display means for displaying image data of the abnormal part and
A learning model that generates the learning model based on a similar image that combines an abnormal image showing an abnormal part generated in the object to be inspected and at least one normal image that is not an abnormal part around the abnormal part. With the means of generation
Information processing device with.