JP2021140435A - Damage specification device, damage specification method, and damage specification program - Google Patents

Abstract

To accurately specify damage that has occurred in an outdoor structure.SOLUTION: A damage specification device comprises: an image acquisition unit which acquires a first image of a first outdoor structure; a boundary decision unit which decides a boundary of the first outdoor structure and a boundary of the portions in the acquired first image; a portion specification unit which specifies a type and portions of the first outdoor structure on the basis of the decided boundary; a material specification unit which specifies materials of the specified portions of the first outdoor structure; a damage determination unit which determines damage that has occurred in the specified portions of the first outdoor structure on the basis of the specified type, portions of the first outdoor structure and materials of the portions; a model generation unit which makes an artificial intelligence learn the type and portions and determined damage of the first outdoor structure by applying an identifier with which each determined damage can be identified to the artificial intelligence to generate a learned damage specification model; a reception unit which receives the input of the second image of the second outdoor structure; and a damage specification unit which specifies the damage that has occurred in the portions of the second outdoor structure by using the learned damage specification model generated by the model generation unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to a damage identification device, a damage identification method and a damage identification program, and more particularly to a computer-embedded image analysis program and an apparatus for identifying damage occurring in an outdoor structure from identification results by artificial intelligence.

The aging of infrastructure structures is accelerating, and it is said that the need for maintenance and renewal will increase further in the future. As the working population continues to decline, improving the efficiency of labor related to maintenance and renewal has become an urgent issue. In order to improve the efficiency of labor related to maintenance and renewal, there are an increasing number of research cases in which AI-based image recognition technology, whose performance has improved significantly in recent years, is applied. By using an advanced neural network called deep learning, it is possible to realize image recognition that is close to human intuition, which was difficult in the past, and it can be expected to assist or replace the visual inspection of damaged parts. Many studies have already reported that damage to structures is detected in a rectangular region (bounding box) (Non-Patent Document 1), and the degree of damage and its shape are detected in a region (segmentation) (Non-Patent Document 2). It is expected to replace and assist visual inspections using these technologies.

In inspection / diagnosis work, it is necessary to take measures for maintenance / renewal, such as evaluating the risk of damage, estimating the cause of damage, and examining countermeasures, based on the damage confirmed visually. In order to make such a secondary judgment, not only the type, position, and shape of the damage, but also the background information on which structural part the damage occurs is important. If the part of the structure can be detected by AI (Artificial Intelligence), AI can be used to estimate secondary judgments such as the cause, danger and countermeasures of damage by combining with the damage detection result.

In addition, among infrastructure structures, for example, unlike tunnels and pipes, bridges are outdoor structures, and due to restrictions on shooting points, there are many situations in which structures are photographed in a relatively panoramic view. It often contains multiple types of parts. According to the Ministry of Land, Infrastructure, Transport and Tourism (“Bridge Periodic Inspection Procedure”), 29 types of superstructures including main girders, 8 types of substructures including beams, and 6 types including bearings are used as bridge parts and members. Eight types of road parts / members including bearings, balustrades, etc., and six types of drainage facilities, etc., for a total of 57 types of parts / members are specified, and many parts / members are arranged throughout the structure. ing. The procedure also stipulates methods for determining damage and its degree, determining damage countermeasure categories, and assessing bridge soundness. The damage countermeasure category also changes depending on the location of the damage. For example, if the floor slab has free lime or exposed reinforcing bars, it is judged that "urgent measures are required".

JSCE Proceedings F6 (Safety Issues) Vol. 73 (2017) No. 2 "Crack detection system for concrete surface based on deep convolutional neural network" Proceedings of the 33rd Annual Meeting of the Japanese Society for Artificial Intelligence 2019 "Transfer learning of rebar exposure segmentation by visual inspection damage images"

However, at present, the method of identifying damage occurring in infrastructure structures often relies on the visual inspection of workers, and it is possible for veteran workers and relatively inexperienced workers to identify damage. There is a big gap in the time and accuracy required for. Therefore, it was not possible to accurately identify the damage occurring in the infrastructure structure. An object of the present invention is to accurately identify damages occurring in an infrastructure structure by machine learning by artificial intelligence.

In order to achieve the above object, the damage identification device according to the present invention is
An image acquisition unit that acquires the first image of the first outdoor structure,
A boundary determining unit that determines the boundary of the first outdoor structure and the boundary of the portion of the first outdoor structure in the acquired first image, and
Based on the determined boundary, the type of the first outdoor structure and the part specifying part for specifying the part of the first outdoor structure, and the part specifying part.
A material specifying part that specifies the material of the specified part of the first outdoor structure, and
The first outdoor structure identified based on the identified type of the first outdoor structure, the identified part of the first outdoor structure and the material of the identified part of the first outdoor structure. Damage determination part that determines the damage that occurred in the site of
An artificial intelligence is trained together with the type of the first outdoor structure, the site of the first outdoor structure, and the determined damage by assigning an identifier that can identify each of the determined damages. A model generator that generates a damaged specific model,
The reception section that accepts the input of the second image of the second outdoor structure,
Using the trained damage identification model generated in the model generation step, a damage identification part for identifying damage occurring in a portion of the second outdoor structure, and a damage identification portion.
Equipped with.

In order to achieve the above object, the damage identification method according to the present invention is:
An image acquisition step for acquiring the first image of the first outdoor structure,
A boundary determination step for determining the boundary of the first outdoor structure and the boundary of the portion of the first outdoor structure in the acquired first image, and
A site identification step for identifying the type of the first outdoor structure and the site of the first outdoor structure based on the determined boundary, and
A material identification step for specifying the material of the identified part of the first outdoor structure, and
The first outdoor structure identified based on the identified type of the first outdoor structure, the identified part of the first outdoor structure and the material of the identified part of the first outdoor structure. Damage determination steps to determine the damage that occurred in the area
An artificial intelligence is trained together with the type of the first outdoor structure, the site of the first outdoor structure, and the determined damage by assigning an identifier that can identify each of the determined damages. A model generation step to generate a damaged specific model, and
The reception step that accepts the input of the second image of the second outdoor structure,
Using the learned damage identification model generated in the model generation step, the damage identification step for identifying the damage occurring at the site of the second outdoor structure and the damage identification step.
including.

In order to achieve the above object, the damage identification program according to the present invention
An image acquisition step for acquiring the first image of the first outdoor structure,
A boundary determination step for determining the boundary of the first outdoor structure and the boundary of the portion of the first outdoor structure in the acquired first image, and
A site identification step for identifying the type of the first outdoor structure and the site of the first outdoor structure based on the determined boundary, and
A material identification step for specifying the material of the identified part of the first outdoor structure, and
The first outdoor structure identified based on the identified type of the first outdoor structure, the identified part of the first outdoor structure and the material of the identified part of the first outdoor structure. Damage determination steps to determine the damage that occurred in the area
An artificial intelligence is trained together with the type of the first outdoor structure, the site of the first outdoor structure, and the determined damage by assigning an identifier that can identify each of the determined damages. A model generation step to generate a damaged specific model, and
The reception step that accepts the input of the second image of the second outdoor structure,
Using the learned damage identification model generated by the model generation unit, a damage identification step for identifying damage occurring at a site of the second outdoor structure, and a damage identification step.
Let the computer run.

According to the damage identification device of the present invention, damage occurring in an outdoor structure can be identified with high accuracy by machine learning by artificial intelligence.

It is a figure for demonstrating the damage identification system including the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a block diagram for demonstrating the structure of the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating an example of the material identification table stored in the storage part of the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating an example of the damage determination table stored in the storage part of the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the damage (corrosion) determined by the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the damage (deterioration of an anticorrosion function) determined by the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the damage (peeling, the rebar exposure) determined by the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating damage (leakage, free lime) determined by the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a flowchart for demonstrating the procedure of the trained damage identification model generation by the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a flowchart for demonstrating the procedure of damage identification by the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the evaluation of the trained damage identification model generated by the damage identification apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating an example of the outdoor structure table which the damage identification apparatus which concerns on 1st Embodiment of this invention has.

Hereinafter, embodiments for carrying out the present invention will be described in detail exemplarily with reference to the drawings. However, the configuration, numerical values, processing flow, functional elements, etc. described in the following embodiments are merely examples, and modifications and changes thereof are free, and the technical scope of the present invention is described below. It is not intended to be limited.

[First Embodiment]
The damage identification device 104 as the first embodiment of the present invention will be described with reference to FIGS. 1 to 10. The damage identification device 104 is used as an outdoor structure to identify damage that has occurred on, for example, a bridge. FIG. 1 is a diagram for explaining a damage identification system 100 including a damage identification device 104 according to the present embodiment.

The damage identification device 104 of the present embodiment is implemented by using, for example, the damage identification system 100 having the system configuration shown in FIG. The damage identification system 100 is installed, for example, in a drone 102 having a function capable of imaging an outdoor structure 120 connected to each other via a wired or wireless communication network 101, and, for example, in a management office. A known machine capable of implementing a personal computer 103 and a neural network using AI (Artificial Intelligence), preferably a convolutional neural network (CNN) or PSPNet (Pyramid Scene Parsing Network) as an algorithm. Includes a damage identification device 104 with a built-in learning tool (software).

The damage identification device 104 identifies the type, site, and material of the outdoor structure 120, preferably using the image of the outdoor structure 120 captured by the drone 102, and identifies the damage. The personal computer 103 controls the imaging of the outdoor structure 120 by the drone 102 to image a part or the whole of the outdoor structure. The captured image is transmitted from the drone 102 to the personal computer 103. The personal computer 103 transmits the received image of the outdoor structure 120 to the damage identification device 104, and displays the identification result of the portion of the outdoor structure 120 transmitted from the damage identification device 104 on a display unit 105 such as a display. Let me.

FIG. 2 is a block diagram for explaining the configuration of the damage identification device 104 according to the present embodiment. The damage identification device 104 includes an image acquisition unit 201, a boundary determination unit 202, a site identification unit 203, a material identification unit 204, a damage determination unit 205, a model generation unit 206, and a storage unit 209. The material identification table 291 and the damage determination table 292 are stored in the storage unit 209. The damage identification device 104 further includes a reception unit 207 and a damage identification unit 208. In the following description, a bridge will be described as an example of the outdoor structure 120.

Here, the damage identification device 104 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a network interface, and a storage. Here, the CPU is a processor for arithmetic control, and realizes each functional configuration of the damage identification device 104 shown in FIG. 2 by executing a program. The CPU may have a plurality of processors and execute different programs, modules, tasks, threads, etc. in parallel. The ROM stores fixed data such as initial data and programs and other programs. In addition, the network interface communicates with other devices and the like via the network. The CPU is not limited to one, and may be a plurality of CPUs, or may include a GPU (Graphics Processing Unit) for image processing. Further, it is desirable that the network interface has another CPU independent of the CPU and writes or reads transmission / reception data in the RAM area. Further, it is desirable to provide a DMAC (Direct Memory Access Controller) for transferring data between the RAM and the storage. Further, the CPU recognizes that the data has been received or transferred to the RAM and processes the data. Further, the CPU prepares the processing result in the RAM, and leaves the subsequent transmission or transfer to the network interface or DMAC.

The RAM is a memory used by the CPU as a temporary storage work area. An area for storing data necessary for realizing the present embodiment is secured in the RAM. The storage stores a database, various parameters, modules, or data or programs necessary for realizing the present embodiment. For example, the storage stores a control program for controlling the entire damage identification device 104.

Further, the damage identification device 104 may further include an input / output interface. A display unit, an operation unit, and a storage medium are connected to the input / output interface. Further, a speaker which is an audio output unit, a microphone which is an audio input unit, or a GPS (Global Positioning System) position determination unit may be connected to the input / output interface. The RAM or storage may store programs and data related to general-purpose functions and other feasible functions of the damage identification device.

The image acquisition unit 201 acquires the image 210 of the bridge 211. The image 210 acquired by the image acquisition unit 201 is captured by the drone 102, but the imaging method of the image 210 is not limited to this. For example, it may be captured by a digital camera, a film camera, a camera attached to a mobile terminal such as a smartphone, a camera mounted on an artificial satellite, or the like.

The image acquisition unit 201 acquires, for example, 884 images 210, but the number of images 210 acquired by the image acquisition unit 201 is not limited to this. The larger the number of images 210 acquired by the image acquisition unit 201, the better the learning accuracy by artificial intelligence, which is preferable. Further, the resolution of the image 210 may be either high resolution or low resolution. A high-resolution image is preferable because the learning accuracy by artificial intelligence is improved, but a resolution is selected according to the processing capacity of the damage identification device 104, the communication capacity of the communication network 101, and the like. Then, the image acquisition unit 201 transmits the acquired image 210 to the boundary determination unit 202.

The boundary determination unit 202 determines the boundary of the portion of the bridge 211 in the acquired image 210. The boundary determination unit 202 determines the boundary of the portion constituting the bridge 211 shown in the image 210 transmitted from the image acquisition unit 201.

Boundary determination is made using techniques such as instance segmentation and semantic segmentation. Here, instance segmentation is a method of dividing an area for each object. In other words, it is a method of dividing the area where the object is located in pixel units while distinguishing each object. In addition, semantic segmentation is a method of dividing an area for each type. Since many parts of the bridge 211 cannot be individually judged as unique objects, semantic segmentation was used in this embodiment.

Further, for determining the boundary, for example, the difference in brightness of the image 210, the shape and shading of the shadow, the difference in contrast, and the like may be used. Further, before determining the boundary, the boundary determining unit 202 may adjust the contrast, lightness, darkness, color balance, etc. of the image 210 before determining the boundary. The boundary determination unit 202 may draw the determined boundary on the image 210 as a boundary line. This is because drawing the boundary line in this way improves the visibility for the user. Then, the boundary determination unit 202 transmits the image 210 whose boundary is determined to the site identification unit 203.

The site identification unit 203 identifies the site of the bridge 211 based on the determined boundary. The site identification by the site identification unit 203 is performed based on, for example, the shape and size of the site of the bridge 211 represented by the determined boundary. The site specifying unit 203 records the specified site in the image 210 by associating the position in the image 210, for example, the site name data and the coordinate data (x, y) in the image 210. Instead of recording the part name data and the coordinate data in the image 210, the site name data and the coordinate data may be stored in a predetermined storage, and these data may be appropriately read out as needed.

Here, the part specifying part 203 specifies a main girder, a horizontal girder, a vertical girder, a floor slab, a column part / wall part, a beam part, a chest wall, a hard wall, a bearing body, a balustrade, and the like as a part of the bridge 211. These parts are large parts or important parts among the parts constituting the bridge 211. Of these parts, the main girder, horizontal girder, vertical girder and deck are called superstructures. In addition, columns / walls and beams form piers, and battlements and hard walls form piers, which are called substructures. Furthermore, the main body of the bearing is called the bearing, and the balustrade is called the street. These parts are defined in the "Regular Inspection Guidelines for Bridges" established by the Ministry of Land, Infrastructure, Transport and Tourism.

Further, when the above-mentioned parts are overlapped with each other in the image 210, the part identification part 203 preferentially identifies the part on the front side. This is because when the above-mentioned parts overlap each other, the part shown on the front side shows the entire part, so that the accuracy of machine learning is higher.

The material specifying unit 204 specifies the material of the portion of the bridge 211. That is, it is specified what kind of material each part constituting the bridge 211 is made of. The material specified by the material identification unit 204 includes, for example, at least one of concrete and steel. When the material of the part does not correspond to either concrete or steel, the material specifying unit 204 may specify the material as another material or not applicable, for example.

As a method of specifying the material by the material specifying unit 204, for example, there is a method of specifying by referring to the material specifying table 291 shown in FIG. FIG. 3 is a diagram for explaining an example of the material identification table 291 stored in the storage unit 209 of the damage identification device 104 according to the present embodiment. As shown in FIG. 3, the material identification table 291 stores the part 302 of the structure and the material 303 of the part in association with the type 301 of the structure. The structure type 301 is, for example, a structure existing outdoors such as a bridge, a radio tower (for example, a Japanese radio tower), or a building. The part of the structure is a small unit or a medium unit part necessary for constructing each structure. For example, in the case of the bridge 211, the superstructure and the substructure, and further, as a subordinate concept thereof, mainly. Includes digits, horizontal digits, etc. The material 303 of the part is concrete and steel. The material identification method is not limited to the method of referring to the material identification table 291 shown here.

The damage determination unit 205 determines the damage that has occurred in the part of the bridge 211 based on the materials of the bridge 211, the part of the bridge 211, and the part of the bridge 211 specified as the type of the outdoor structure 120. As a method of determining damage by the damage determination unit 205, for example, there is a method of determining with reference to the damage determination table 292 shown in FIG.

FIG. 4 is a diagram for explaining an example of the damage determination table 292 stored in the storage unit 209 of the damage identification device 104 according to the present embodiment. The damage determination table 292 stores the damage 402 in association with the material 401 of the site. Part material 401 includes concrete and steel. Damage 402 includes peeling, internal structure exposure, water leakage / free lime, rebar exposure and rust juice when the site material 401 is concrete, and corrosion and corrosion protection degradation when the site material 401 is steel. Then, the damage determination unit 205 determines the damage that has occurred at the site of the bridge 211 with reference to the damage determination table 292.

As shown in FIG. 4, by referring to the damage determination table 292, damage that cannot occur depending on the material of the site can be eliminated in advance. For example, if a part of the part made of steel is marked with chalk, the chalk may be judged as free lime, but in steel, free lime is not generated. If the material is steel, the damage of free lime is excluded in advance. Further, for example, since corrosion does not occur in a portion where the material is concrete, damage called corrosion is excluded in advance when the material is concrete. Here, the exposed reinforcing bar is, for example, a state in which the peeling of the concrete has progressed to a considerable extent and the reinforcing bar inside the concrete can be seen, or a state in which the exposed reinforcing bar is corroded or the like. Rust juice is the one in which the reinforcing bars inside the concrete are corroded and seep out to the concrete surface. As described above, even if the material 401 of the part is concrete, the damage caused by the reinforcing bars and the like contained in the concrete is not excluded even if the material 401 is concrete, and the damage of the concrete is not excluded. Treat as. By associating the portion of the bridge 211 with the material of the portion in this way, it is possible to prevent erroneous determination of damage that cannot occur depending on the material. The method of determining damage is not limited to the method of referring to the damage determination table 292 shown here.

Here, the damage determined by the damage determination unit 205 will be described with reference to FIGS. 5 to 8. FIG. 5 is a diagram for explaining damage (corrosion) determined by the damage identification device according to the present embodiment.

Corrosion is caused by the state where rust is intensively generated in ordinary steel materials that have been subjected to anticorrosion measures such as painting and plating, or the rust progresses extremely and the plate thickness is reduced or the cross section is defective (hereinafter referred to as "plate thickness reduction, etc."" ) Is occurring. In the case of weathering steel, it means that protective rust is not formed and abnormal rust is generated, or that the plate thickness is significantly reduced due to the progress of extreme rust.

In the present embodiment, the occurrence of rust accompanied by a decrease in plate thickness or the like is referred to as "corrosion", and the occurrence of slight rust that can be regarded as not accompanied by a decrease in plate thickness or the like is referred to as "deterioration of anticorrosion function". If it is difficult to determine whether or not the plate thickness has decreased, it is considered as "corrosion". In the period until protective rust is generated in weathering steel, it may be difficult to judge whether the rust is abnormal or not, but if it can be considered that there is no decrease in plate thickness, etc. Deterioration of anticorrosion function ".

Similarly, in the case of bolts, the occurrence of rust accompanied by thinning is treated as corrosion, and the occurrence of slight rust that can be regarded as not accompanied by thinning is regarded as "deterioration of anticorrosion function".

The degree of damage is determined by, for example, in the Ministry of Land, Infrastructure, Transport and Tourism (“Bridge Periodic Inspection Procedure”), the degree (a to e) is determined by a combination of the depth of damage and the area of damage. For example, if there is no damage, it is determined as a, if both the depth of damage and the area of damage are small, it is determined as b, and if the depth of damage is small and the area of damage is large, it is determined as c. Further, when the damage depth is large and the damage area is small, it is determined as d. If both the depth of damage and the area of damage are large, it is determined to be d (see the reference photograph in the right figure of FIG. 5).

Regarding the depth, when the depth is large, it means that the surface of the steel material is significantly expanded or a clear decrease in plate thickness is visible, and when the depth is small, the rust is superficial. Significant reduction in plate thickness is not visible. Furthermore, regarding the size, a large area means that the entire part is rusted or there are a plurality of rusted parts that are spread over the part, and a small area means that the area of the damaged part is small. Be local.

Next, the deterioration of the anticorrosion function will be described. FIG. 6 is a diagram for explaining damage (deterioration of anticorrosion function) determined by the damage identification device according to the present embodiment. First, for steel members, the general properties of deterioration of the anticorrosion function and the characteristics of damage will be described. Category 1 "painting" is characterized by deterioration of the anticorrosion coating film. Category 2 "plating, metal spraying" is characterized by discoloration, cracking, blistering, and peeling due to deterioration of the anticorrosion film. The category 3 "weathering steel" is characterized in that no protective rust is formed.

The relationship between this damage and other damages will be described. In painting, hot-dip galvanizing, and metal spraying, the occurrence of rust that accompanies a decrease in plate thickness is referred to as "corrosion." The occurrence of slight rust that can be regarded as not accompanied by a decrease in plate thickness is regarded as "deterioration of anticorrosion function". In weathering steel materials, when abnormal rust accompanied by a decrease in plate thickness occurs, it is referred to as "corrosion", and when coarse rust or scaly rust occurs, it is referred to as "deterioration of anticorrosion function".

The degree of damage is determined, for example, in the Ministry of Land, Infrastructure, Transport and Tourism (“Bridge Periodic Inspection Procedure”), the degree (a to e) is determined according to general properties. For example, category 1 "painting" is judged to be a if there is no damage, and as a general property, when the outermost anticorrosion coating film is discolored or local swelling occurs. , C is determined. Further, when the anticorrosion coating film is partially peeled off and the undercoat is exposed, it is determined to be d. Further, when the deterioration range of the anticorrosion coating film is wide and rust spots occur, it is determined as e.

Next, with respect to Category 2 "plating, metal spraying", if there is no damage, it is judged as a, and if the anticorrosion film is locally deteriorated and rust spots occur, it is judged as c. Further, when the deterioration range of the anticorrosion film is wide and rust spots occur, it is determined as e. Next, the category 3 "weathering steel" is determined to be a if there is no damage, and is determined to be b if there is no damage but no correction rust is generated. If the size of the rust is about 1 to 5 mm and is coarse, it is determined to be c, and if the size of the rust is scaly with a size of about 5 to 25 mm, it is determined to be d. If there is layered peeling of rust, it is determined to be e (see the reference photograph in the right figure of FIG. 6).

FIG. 7 is a diagram for explaining the damage (peeling / reinforcing bar exposure) determined by the damage identifying device according to the present embodiment. First, the general properties of peeling / exposed reinforcing bars and the characteristics of damage will be described. The state in which the surface of the concrete member is peeled off is defined as peeling, and the state in which the reinforcing bar is exposed at the peeled portion is defined as the exposed reinforcing bar.

The relationship between this damage and other damages will be described. "Peeling / exposed reinforcing bars" includes corrosion and breaking of exposed reinforcing bars, and does not mean damage such as "corrosion" and "breaking". Detachment and rebar exposure that occur on the deck shall also be included.

The degree of damage is determined, for example, in the Ministry of Land, Infrastructure, Transport and Tourism (“Bridge Periodic Inspection Procedure”), the degree (a to e) is determined according to general properties. If there is no damage, it is determined to be a. If only peeling occurs, it is determined to be c. If the reinforcing bar is exposed but the corrosion of the reinforcing bar is slight, it is determined to be d. If the reinforcing bar is exposed and the reinforcing bar is significantly corroded or broken, it is judged as e (see the reference photograph in the right figure of FIG. 7).

FIG. 8 is a diagram for explaining damage (leakage / free lime) determined by the damage identification device according to the present embodiment. First, regarding water leakage and free lime, the general properties and characteristics of damage will be explained. Leakage / free lime refers to a state in which water or lime is exuded or leaked from concrete joints or cracks.

The relationship between this damage and other damages will be described. Precipitates generated by water flowing on the surface of concrete members due to poor drainage are distinguished from free lime. In addition, water that is supplied from the outside and flows directly on the surface of the concrete member is considered to be leaking or stagnant. Concrete damage that also applies to other damages such as cracks, cracks, and peeling shall also apply to each item.

The degree of damage is determined by, for example, the Ministry of Land, Infrastructure, Transport and Tourism (“Bridge Periodic Inspection Procedure”), and the degree (a to e) is determined according to general properties. If there is no damage, it is determined to be a. If water leaks from the cracks but almost no rust juice or free lime is found, it is judged as c. If free lime is generated from the cracks but almost no rust juice is seen, it is judged as d. If significant water leakage or free lime (for example, icicle-like) is generated from the crack, or if significant mud or rust juice is mixed in the water leakage, it is judged as e (see the reference photograph in the right figure of FIG. 8). ). The damage determining unit 205 may determine the degree of damage in addition to determining the damage. The ranks a to e described above may be used to determine the degree of damage, but the ranking method is not limited thereto.

The model generation unit 206 assigns an identifier (ID: Identifier) that can identify each of the determined damages to the damages determined by the damage determination unit 205. The model generation unit 206 assigns, for example, the range (magnitude) of the damage, the name of the damage, the attributes of the damage (old, new, color), and the like as identifiers to the image 210. This identifier may be directly assigned to the image 210, or may be stored in a predetermined database or the like and read out as appropriate.

Then, the model generation unit 206 includes the assigned identifier, the type of the outdoor structure 120 (bridge 211), the part of the bridge 211, the image 210 (correct image) having the determined damage, and the image 210 (original image). Is input to artificial intelligence to make machine learning. When machine learning by artificial intelligence is completed, the model generation unit 206 generates a trained damage specific model. The model generation unit 206 may store the generated learned damage specific model in the storage unit 209. In this case, a new learning image (image 210) may be acquired, machine learning may be performed, and the saved learned damage identification model may be updated each time a trained damage identification model is generated.

Further, the model generation unit 206 may generate a learned damage specific model separately for the case where the material of the portion of the bridge 211 is concrete and the case where the material is steel. That is, the model generation unit 206 may generate a learned concrete damage identification model for identifying the damage occurring at the portion of the bridge 221 when the material is concrete. Similarly, the model generator 206 may generate a trained steel damage identification model for identifying damage occurring at the site of the bridge 221 when the material is steel.

In this way, by generating a trained damage identification model according to the material, in the damage identification performed later, when the material is steel, the damage that occurs in the concrete is eliminated, and then the damage that occurs in the steel. Can be identified. Similarly, when the material is concrete, damage to steel can be eliminated and damage to concrete can be identified. In this way, by generating the damage identification model according to the material of the site, it is possible to identify the damage with high accuracy in a short time.

Machine learning by artificial intelligence is performed using known algorithms. In this embodiment, for example, PSPNet is used. PSPNet is an algorithm that can detect features distributed over a wide area with higher performance than a conventional semantic segmentation model. In the image 210 of the bridge 211, the parts are often widely distributed in the local image of the bridge 211, and it is considered that the shape of the parts distributed over a wide area is suitable as a detection model. In machine learning, the loss function has no weight.

Further, the model generation unit 206 inflates and increases the number of images 210 to be trained by the artificial intelligence in order to improve the accuracy of machine learning by the artificial intelligence and generate a damage specific model with higher accuracy. The model generator 206 obtains inflated data using, for example, any one of enlargement, rotation, and inversion.

Further, the model generation unit 206 may use transfer learning in order to improve the accuracy of machine learning by artificial intelligence. Here, transfer learning is a method that aims to improve the performance of a model by diverting a trained model using a different data set to another problem and performing partial learning. In particular, it is a method that can be expected to improve inference performance and reduce learning time when teacher data is insufficient. In this embodiment, transfer learning was performed using the image data set Pascal VOC2012 Semantic Segmentation Dataset labeled with 22 types of segmentation.

The reception unit 207 receives the leading image 220 of the bridge 221. In the present embodiment, the received image 220 is an image captured by the drone 102, but the imaging method of the bridge 221 is not limited to the drone 102.

The damage identification unit 208 identifies the damage occurring at the site of the bridge 221 in the image 220 received by the reception unit 207 by using the trained damage identification model generated by the model generation unit 206. In the damage identification unit 208, in the received image 220, using the trained damage identification model, it occurs at the site of the bridge 221 due to the correlation between the damage shown in the image 220 and the damage in the trained damage identification model. The damage is identified. The damage identification unit 208 identifies as damage occurring at the site of the bridge 221 that has a high degree of correlation with the damage in the trained damage identification model. After that, the damage identification device 104 outputs the result image 230.

FIG. 9 is a flowchart for explaining a procedure for generating a learned damage identification model by the damage identification device 104 according to the present embodiment. FIG. 10 is a flowchart for explaining a procedure for identifying damage by the damage identifying device 104 according to the present embodiment. The flowcharts shown in FIGS. 9 and 10 are executed by a CPU (Central Processing Unit) (not shown) of the damage identification device 104 using a ROM (Read Only Memory) and a RAM (Random Access Memory), and are shown in FIG. Each functional configuration of the damage identification device 104 is realized.

First, with reference to FIG. 9, a process of generating a trained damage identification model by machine learning by artificial intelligence will be described. In step S901, the image acquisition unit 201 acquires the image 210 of the bridge 211. In step S903, the boundary determination unit 202 determines the boundary of the portion of the bridge 211 in the acquired image 210. In step S905, the site specifying portion 203 identifies the site of the bridge 211. In step S907, the material identification unit 204 identifies the material of the part.

In step S909, the damage determination unit 205 determines whether the material of the portion of the bridge 211 is steel or concrete. If the material is determined to be steel, the damage determination unit 205 proceeds to step S911. In step S911, since the material of the portion is steel, the damage determining unit 205 previously eliminates damage that occurs in concrete, that is, damage that does not occur in steel, and identifies damage that occurs in steel. do.

If the material is determined to be concrete in step S909, the damage determination unit 205 proceeds to step S913. In step S9103, since the material of the portion is concrete, the damage determining unit 205 previously eliminates damage that occurs in steel, that is, damage that does not occur in concrete, and identifies damage that occurs in concrete. ..

In step S915, the model generation unit 206 inputs an image (correct image) in which a site or damage is specified and an original image (image 210) into artificial intelligence for machine learning. In step S917, the model generation unit 206 determines whether or not machine learning has been completed for all of the acquired images 210. If it is determined that the machine learning has not been completed (NO in step S917), the damage identification device 104 returns to step S903. When it is determined that the machine learning is completed (YES in step S917), the damage identification device 104 proceeds to step S919. In step S919, the model generation unit 206 generates a trained damage identification model. The model generation unit 206 may generate a trained damage specific model according to the type of material of the site.

Next, with reference to FIG. 10, a process for identifying the damage occurring at the site of the bridge 221 in the newly acquired image 220 using the generated learned damage identification model will be described. In step S1001, the reception unit 207 receives the input of the image 220 of the bridge 221. In step S1003, the damage identification unit 208 identifies the damage occurring at the site of the bridge 221 using the trained damage identification model generated by the model generation unit 206. In step S1005, the damage identification device 104 outputs the identified result. In step S1007, it is determined whether or not the output of the result is completed for all the images 220 received by the reception unit 207. When it is determined that the output of the results has not been completed for all the images 220 (NO in step S1007), the damage identification device 104 returns to step S1003. When it is determined that the output of the results has been completed for all the images 220 (YES in step S1007), the damage identification device 104 ends the process.

Next, with reference to FIG. 11, the evaluation of the trained damage identification model generated by the damage identification device 104 will be described. FIG. 11 is a diagram for explaining the evaluation of the trained damage identification model generated by the damage identification device according to the present embodiment. The trained damage specific model was evaluated using the recall rate.

Here, the recall rate is represented by the ratio of the pixels correctly determined to be i among the pixels belonging to the class i. The recall rate is expressed as recall rate = TP / (TP + FN). Here, TP (True Positive) is a number that can correctly predict that the correct answer is correct, and FN (False Negative) is a number that mistakenly predicts that the correct answer is negative. .. The recall rate is an index used when it is desired to evaluate whether or not what is desired to be detected is detected, and is an index that can evaluate how strong it is against oversight.

As shown in FIG. 11, the recall rate of each damage is 59.1% for corrosion or deterioration of anticorrosion function, 49.6% for water leakage / free lime, and 26.4% for peeling / reinforcing bar exposure. It became. Here, the recall rate is used as an evaluation index, but the evaluation index is not limited to the recall rate. The evaluation index may be, for example, IoU (Intersection over Union) or MeanIoU.

In the above description, the bridge 211 has been described as an example of the outdoor structure 120, but the outdoor structure 120 is not limited to this, and for example, the radio tower (Japan Radio Tower (Tokyo Tower), Tokyo Sky Tree (Tokyo Tower), etc. It can also be applied to registered trademarks)) and buildings. In this case, the damage identifying device 104 identifies the type of outdoor structure and the like with reference to, for example, the outdoor structure table 1200 shown in FIG. The outdoor structure table 1200 stores the site 1202 in association with the outdoor structure type 1201. The damage identification device 104 identifies the type and portion of each outdoor structure with reference to the outdoor structure table 1200. The damage identifying device 104 identifies the type 1201 of the outdoor structure from the shape (silhouette), size, and the like of the outdoor structure in the image. Further, metadata such as position information (for example, GPS data) may be added to the image of the outdoor structure, and the type 1201 of the outdoor structure may be specified in consideration of the position information and the like. Further, the bridge includes, for example, a girder bridge, a truss bridge, an arch bridge, a ramen bridge, an oblique bridge, a suspension bridge, and the like, but the damage identification device 104 of the present embodiment can be applied to any bridge. ..

According to the present embodiment, since the damage generated in the portion of the outdoor structure can be easily identified, the deterioration of the outdoor structure can be easily diagnosed. Further, since the part of the outdoor structure and the material of the part are linked, damage and deterioration that cannot occur depending on the material can be eliminated in advance, so that damage identification and deterioration diagnosis can be performed with high accuracy.

[Other Embodiments]
Although the invention of the present application has been described above with reference to the embodiment, the invention of the present application is not limited to the above embodiment. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the structure and details of the present invention. Also included in the scope of the present invention are systems or devices in which the different features contained in each embodiment are combined in any way.

Further, the present invention may be applied to a system composed of a plurality of devices, or may be applied to a single device. Furthermore, the present invention is also applicable when the information processing program that realizes the functions of the embodiment is supplied directly or remotely to the system or device. Therefore, in order to realize the functions of the present invention on a computer, a program installed on the computer, a medium containing the program, and a WWW (World Wide Web) server for downloading the program are also included in the scope of the present invention. .. In particular, at least a non-transitory computer readable medium containing a program that causes a computer to execute the processing steps included in the above-described embodiment is included in the scope of the present invention.

Claims (11)

An image acquisition unit that acquires the first image of the first outdoor structure,
A boundary determining unit that determines the boundary of the first outdoor structure and the boundary of the portion of the first outdoor structure in the acquired first image, and
Based on the determined boundary, the type of the first outdoor structure and the part specifying part for specifying the part of the first outdoor structure, and the part specifying part.
A material specifying part that specifies the material of the specified part of the first outdoor structure, and
The first outdoor structure identified based on the identified type of the first outdoor structure, the identified part of the first outdoor structure and the material of the identified part of the first outdoor structure. Damage determination part that determines the damage that occurred in the site of
An artificial intelligence is trained together with the type of the first outdoor structure, the site of the first outdoor structure, and the determined damage by assigning an identifier that can identify each of the determined damages. A model generator that generates a damaged specific model,
The reception section that accepts the input of the second image of the second outdoor structure,
Using the learned damage identification model generated by the model generation unit, a damage identification unit that identifies damage occurring at a site of the second outdoor structure and a damage identification unit.
Damage identification device equipped with. The damage identification device according to claim 1, wherein the model generation unit generates a learned damage identification model for each of the specified materials. The damage identification device according to claim 1 or 2, wherein the material identification unit specifies at least one of concrete and steel as the material. When the material identification part identifies concrete as the material, the damage identification part identifies at least one of peeling, exposure of the internal structure, water leakage / free lime, exposed rebar, and rust juice as the damage.
The damage identification device according to any one of claims 1 to 3, wherein the model generation unit generates a learned damage identification model for concrete materials. When the material identification part identifies steel as the material, the damage identification part identifies at least one of corrosion and deterioration of anticorrosion function as the damage.
The damage identification device according to any one of claims 1 to 3, wherein the model generation unit generates a learned damage identification model for steel materials. The damage identifying device according to any one of claims 1 to 5, wherein the boundary determining unit uses semantic segmentation to identify the boundary of the first outdoor structure and the boundary of a portion of the first outdoor structure. .. The damage identification device according to any one of claims 1 to 6, wherein the model generation unit generates the trained damage identification model using PSPNet. The damage identification device according to any one of claims 1 to 7, wherein the model generation unit generates the trained damage identification model by using any one of expansion, rotation, and inversion as inflated data. The damage identification device according to any one of claims 1 to 8, wherein the model generation unit generates the trained damage identification model by using transfer learning. An image acquisition step for acquiring the first image of the first outdoor structure,
A boundary determination step for determining the boundary of the first outdoor structure and the boundary of the portion of the first outdoor structure in the acquired first image, and
A site identification step for identifying the type of the first outdoor structure and the site of the first outdoor structure based on the determined boundary, and
A material identification step for specifying the material of the identified part of the first outdoor structure, and
The first outdoor structure identified based on the identified type of the first outdoor structure, the identified part of the first outdoor structure and the material of the identified part of the first outdoor structure. Damage determination steps to determine the damage that occurred in the area
An artificial intelligence is trained together with the type of the first outdoor structure, the site of the first outdoor structure, and the determined damage by assigning an identifier that can identify each of the determined damages. A model generation step to generate a damaged specific model, and
The reception step that accepts the input of the second image of the second outdoor structure,
Using the learned damage identification model generated in the model generation step, the damage identification step for identifying the damage occurring at the site of the second outdoor structure and the damage identification step.
Damage identification methods including. An image acquisition step for acquiring the first image of the first outdoor structure,
A boundary determination step for determining the boundary of the first outdoor structure and the boundary of the portion of the first outdoor structure in the acquired first image, and
A site identification step for identifying the type of the first outdoor structure and the site of the first outdoor structure based on the determined boundary, and
A material identification step for specifying the material of the identified part of the first outdoor structure, and
The first outdoor structure identified based on the identified type of the first outdoor structure, the identified part of the first outdoor structure and the material of the identified part of the first outdoor structure. Damage determination steps to determine the damage that occurred in the area
An artificial intelligence is trained together with the type of the first outdoor structure, the site of the first outdoor structure, and the determined damage by assigning an identifier that can identify each of the determined damages. A model generation step to generate a damaged specific model, and
The reception step that accepts the input of the second image of the second outdoor structure,
Using the learned damage identification model generated in the model generation step, the damage identification step for identifying the damage occurring at the site of the second outdoor structure and the damage identification step.
A damage identification program that lets your computer run.

Images