JP6944506B2 - Image processing method, image processing device, and image processing program - Google Patents

Description

The present invention is an image processing method capable of analyzing an image of a cracked portion without requiring an expensive device and presenting a composite image of the cracked portion and an image analysis result without complicating the user's operation. , Image processing device, and image processing program.

As social infrastructure, there are structures such as bridges and tunnels. Since these structures are damaged and the damage is progressive, it is required to be inspected regularly. It is also required to report accurate inspection results. For example, it is required to accurately grasp the properties of cracks generated on the surface of the structure and report the inspection results, and based on the report, it is necessary to carry out appropriate maintenance according to the degree of progress of the cracks. be.

In Patent Document 1, when a distant view image and a close-up image with a laser beam spot obtained by imaging a concrete wall surface with a camera equipped with a laser projectile are transmitted from a mobile personal computer, the distant view image and the close-up image are transmitted from the front of the concrete wall surface. If you use a mobile computer to perform an operation to assign the proximity image to which area of the distant view image in order to normalize it to the viewed image and further specify the proximity image to be image-analyzed, and send the analysis instruction, the specified proximity image will be displayed. A server device that analyzes an image and measures the crack width is described.

According to Patent Document 2, in a personal computer, an operation of pasting an image of a concrete structure on a CAD (computer aided design) drawing based on information on a shooting position entered in a field book during work at an inspection site is performed. , The image is associated with the CAD drawing according to the operation, and when the operation of tracing the crack on the image is performed, the crack on the image is mapped to the CAD drawing according to the operation, and the crack is mapped using the information of the CAD drawing. It is stated that the width and length of the

In Patent Document 3, a personal computer measures the distance from the camera to the concrete wall surface and the inclination angle of the concrete wall surface with respect to the optical axis of the camera, joins a plurality of images, and views the image from the front of the concrete wall surface. It is described that the state diagram is displayed by obtaining the distribution state of cracks on the concrete wall surface and the width of the cracks. Cracks that are actually cracks but cannot be recognized can be input by tracing using a mouse or a pen input device.

Patent Document 4 supports crack identification processing. When machine learning is performed using a vector machine algorithm, the center of each region image among a plurality of region images and the identification target in each region image (cracks, stains, etc.) It is described that only the region image that coincides with the center of gravity of (such as the unevenness of) is selected as the sample to be learned.

Japanese Unexamined Patent Publication No. 2004-69434 Japanese Unexamined Patent Publication No. 2005-310044 Japanese Unexamined Patent Publication No. 10-78305 Japanese Unexamined Patent Publication No. 2003-216953

Patent Documents 1 to 4 do not disclose that the position information of each image is transmitted from the user terminal together with a plurality of images obtained by dividing and imaging the structure and combined into one image by the server device. ..

Further, in the system including a mobile personal computer (a form of a user terminal) and a server device described in Patent Document 1, not only a camera provided with a laser projectile is required, but also a distant view image and a close-up image are provided. It is necessary, and further, it is necessary to specify the proximity image to be image-analyzed by the user operation of allocating the proximity image to the distant view image. Therefore, it can be said that not only an expensive device is required, but also the user's operation until the image analysis result of the crack is obtained is complicated.

Further, in order to generate a composite image of a cracked portion on a user terminal and perform image analysis of the cracked portion, expensive hardware or expensive software is required. In other words, it required an expensive device.

In Patent Document 2, when an operation of pasting an image on a CAD drawing and an operation of tracing a crack on the image are performed on a personal computer (a form of a user terminal), the width and length of the crack are determined. It only states that it should be calculated.

In the technique described in Patent Document 3, when correcting crack information, it is necessary to perform an operation of tracing one by one using a mouse or a pen input device. By using the crack information correction function by such a trace operation, it is possible to individually correct cracks of any pattern one by one, but the operation is complicated and it takes a lot of work time, so it is busy. It may not be possible for workers to actually do it.

Patent Document 4 merely describes learning the crack identification process based on the image of the structure.

As described above, various devices have been proposed or provided to help accurately grasp the properties of cracks generated on the surface of the structure and report the inspection results, but the images of the cracked parts are analyzed. In order to present the image analysis result of the crack to the user, an expensive device is required, and there is a problem that the load on the user for synthesizing a plurality of images of the cracked portion is heavy.

The present invention is an image processing method capable of analyzing an image of a cracked portion without requiring an expensive device and presenting a composite image of the cracked portion and an image analysis result without complicating the user's operation. , An image processing apparatus, and an image processing program.

In order to achieve the above-mentioned object, the server device according to the first aspect of the present invention includes a receiving unit that receives a plurality of images showing the surface of the structure and position information of the plurality of images from the user terminal. Based on a plurality of received images and position information, a composite image generator that generates a composite image showing the surface of the structure and an image analysis of the composite image or the image before synthesis are performed to detect cracks on the surface of the structure. A crack detection unit is provided, and a crack information generation unit that generates crack information indicating a crack image corresponding to the detected crack and a feature amount of the detected crack is provided.

According to this aspect, when a plurality of images showing the surface of a structure and position information of the plurality of images are transmitted from a user terminal, a composite image is generated based on the plurality of images and the position information of the images by the server device. Moreover, since the crack information is generated, it is possible to analyze the image of the cracked portion without requiring an expensive device and to present the composite image and the image analysis result of the cracked portion without complicating the user's operation. It will be possible. Further, according to this aspect, the image transmitted from the user terminal and its position information, as well as the composite image and crack information generated by the server device can be accumulated and converted into big data. It can be said that the secondary use of big data makes it possible to derive appropriate composite images and appropriate image analysis results.

The server device according to the second aspect of the present invention includes a transmission unit that transmits a composite image and crack information to a user terminal and displays the composite image and crack information on the screen of the user terminal, and the reception unit is a user. A request for correction of crack information according to a manual operation on the terminal is received from the user terminal, the crack information generation unit corrects the crack information based on the correction request, and the transmission unit transmits the corrected crack information to the user terminal. Send. According to this aspect, when a composite image and crack information are displayed on the screen of the user terminal and the user manually operates the displayed contents, a correction request corresponding to the manual operation is stored in the server. Becomes possible. Therefore, it becomes easy to improve the image analysis process in the server device so that a more appropriate image analysis result can be derived. That is, it is possible to present a more appropriate image analysis result to the user while reducing complicated user operations for correcting the image processing result.

In the server device according to the third aspect of the present invention, the receiving unit receives a correction request indicating the correction content of the crack information in a feature amount from the user terminal, and the crack information generating unit receives the correction request indicated by the feature. Correct crack information based on quantity. According to this aspect, it is possible to perform a user operation indicating a request for correction of crack information by the feature amount of the crack. For example, compared with the case where the crack information is corrected one by one based on the trace operation, the crack information can be corrected by a simple operation.

In the server device according to the fourth aspect of the present invention, the frequency distribution indicating the detection frequency of the line segment in the composite image with respect to the feature amount and the slider for instructing and inputting the correction content of the crack information by the feature amount are cracked. It is added to the information, and the crack information is corrected according to the user operation using the slider on the user terminal. According to this aspect, the crack information can be corrected by a simple operation of operating the slider while observing the detection frequency of the line segment with respect to the feature amount.

In the server device according to the fifth aspect of the present invention, the crack information generation unit groups the crack images based on the feature amount and generates the crack information according to the manual operation of selecting the group of the crack images on the user terminal. Fix it. According to this aspect, the crack information can be corrected by a simple operation of selecting a group of crack images.

The server device according to the sixth aspect of the present invention includes a machine learning unit that machine-learns the detection of cracks based on a manual operation on the user terminal. According to this aspect, since the detection of cracks is machine-learned based on the manual operation on the user terminal, it is possible to steadily reduce the complicated user operation for correcting the image analysis result.

In the server device according to the seventh aspect of the present invention, the feature amount includes at least one of the crack direction, length, width, edge strength, and edge density.

In the image processing method according to the eighth aspect of the present invention, the composite image generation unit corrects at least one of image position information, scale, tilt, and rotation angle for a plurality of images.

In the server device according to the ninth aspect of the present invention, the composite image generation unit derives correction parameters based on a plurality of images and performs correction.

The server device according to the tenth aspect of the present invention includes a scale image generation unit that generates a scale image for measuring the width of the crack indicated by the crack image on the screen of the user terminal based on the crack information. The crack information generation unit adds a scale image to the crack information.

The server device according to the eleventh aspect of the present invention includes a crack evaluation unit that evaluates the degree of cracks based on at least the crack information and determines the evaluation classification of the degree of cracks in the structure.

The server device according to the twelfth aspect of the present invention includes a form creation unit that creates a form showing the inspection result of the structure including the determined evaluation category.

The server device according to the thirteenth aspect of the present invention includes a data conversion unit that converts a composite image and crack information into CAD data in a predetermined data format.

In the server device according to the fourteenth aspect of the present invention, based on at least one of the structural information, the environmental condition information, the usage condition information, the maintenance management history information, and the disaster history information of the structure, and the crack information. , It is equipped with a prediction unit that predicts the progress of cracks.

The image processing system according to the fifteenth aspect of the present invention is an image processing system including a user terminal and a server device, and the user terminal includes a plurality of images showing the surface of a structure and position information of the plurality of images. A terminal input unit for inputting images, a terminal transmitter unit for transmitting a plurality of images and position information to the server device, a terminal receiver unit for receiving image processing results for a plurality of images from the server device, and an image processing result. The server device includes a terminal display unit for displaying the above, and the server device receives a plurality of images and position information from the user terminal, and a structure based on the received plurality of images and position information. A composite image generator that generates a composite image showing the surface, a crack detection unit that detects cracks on the surface of the structure by analyzing the composite image or the image before synthesis, and a crack image corresponding to the detected cracks. It is provided with a crack information generation unit that generates crack information indicating the detected feature amount of the crack, and a server transmission unit that transmits a composite image and crack information to a user terminal as an image processing result for a plurality of images. The terminal display unit displays a composite image and crack information.

In the image processing method according to the sixteenth aspect of the present invention, a step of receiving a plurality of images showing the surface of a structure and position information of the plurality of images from a user terminal, and a plurality of received images and images. Corresponds to the step of generating a composite image showing the surface of the structure based on the position information, the step of analyzing the composite image or the image before synthesis to detect the crack on the surface of the structure, and the detected crack. It includes a step of generating crack information indicating the crack image to be performed and the feature amount of the detected crack.

According to the present invention, it is possible to analyze an image of a cracked portion without requiring an expensive device, and to present a composite image of the cracked portion and an image analysis result without complicating the user's operation.

FIG. 1 is a perspective view showing a bridge which is an example of a structure to which the present invention is applied. FIG. 2 is a block diagram showing a configuration example of the image processing system according to the first embodiment. FIG. 3 is a diagram showing a frequency distribution and a first display example of the slider. FIG. 4 is a diagram showing a frequency distribution and a second display example of the slider. FIG. 5 is a diagram showing a frequency distribution and a third display example of the slider. FIG. 6 is a diagram showing a frequency distribution and a fourth display example of the slider. FIG. 7 is a diagram showing a display example when the cracked images are grouped based on the feature amount. FIG. 8 is a flowchart showing the flow of the first image processing example in the first embodiment. FIG. 9 is an explanatory diagram used for explaining the first image processing example in the first embodiment. FIG. 10 is an explanatory diagram used for explaining an operation of arranging a plurality of images. FIG. 11 is a diagram showing a display example of a composite image and a cracked image. FIG. 12 is a diagram schematically showing an example of a crack progress model. FIG. 13 is a diagram showing an example of a damage diagram as a form. FIG. 14 is a flowchart showing the flow of the second image processing example in the first embodiment. FIG. 15 is a flowchart showing the flow of the third image processing example in the first embodiment. FIG. 16 is an explanatory diagram used for explaining a third image processing example in the first embodiment. FIG. 17 is a flowchart showing a flow of a fourth image processing example in the first embodiment. FIG. 18 is a block diagram showing a configuration example of the image processing system according to the second embodiment. It is a 1st explanatory diagram used for the explanation of the vectorization of the crack image. It is a 2nd explanatory diagram used for the explanation of the vectorization of the crack image. It is a 3rd explanatory diagram used for the explanation of the vectorization of the crack image. It is a 4th explanatory diagram used for the explanation of the vectorization of the crack image. FIG. 5 is a fifth explanatory diagram used for explaining the vectorization of a cracked image. FIG. 6 is a sixth explanatory diagram used for explaining the vectorization of a cracked image. It is a figure which shows the display example of the scale image.

Hereinafter, an image processing method, an image processing apparatus, and a mode for carrying out an image processing program according to the present invention will be described with reference to the accompanying drawings.

[Structure example]
FIG. 1 is a perspective view showing a structure of a bridge, which is an example of a structure to which the present invention is applied. The bridge 1 shown in FIG. 1 has a plurality of main girders 3, and the main girders 3 are joined by a joint portion 3A. A floor slab 2, which is a concrete member, is placed on the upper part of the main girder 3. In addition to the floor slab 2 and the main girder 3, the bridge 1 has members such as a horizontal girder, an anti-tilt structure, and a horizontal structure (not shown).

The "structure" in the present invention is not limited to bridges. The structure may be a tunnel, a dam, a building, or the like.

[First Embodiment]
FIG. 2 is a block diagram showing a configuration example of the image processing system according to the first embodiment.

A client-server type image processing system is configured by the user terminal 10 and the server device 20. Actually, a plurality of user terminals 10 are connected to the network NW. The user terminal 10 is composed of a computer device that performs front-end processing for the user, and the server device 20 is composed of a computer device that performs back-end processing such as image processing. As the user terminal 10, for example, a computer device such as a personal computer, a tablet, or a smartphone can be used. The server device 20 may be composed of a plurality of computer devices. The server device 20 of this example includes a database 50 for convenience of explanation. The server device 20 and the database 50 may be provided as different devices.

The user terminal 10 is used for a terminal input unit 11 for inputting a plurality of images showing the surface of a structure and information necessary for image composition including position information of the plurality of images, and for image composition with the input plurality of images. A terminal transmitting unit 12 that transmits necessary information to the server device 20, a terminal receiving unit 13 that receives image processing results for a plurality of images from the server device 20, and a terminal that displays the image processing results of the server device 20. The display unit 14, the terminal operation unit 15 that accepts the user's manual operation, the terminal control unit 16 that controls each part of the user terminal 10 according to the front-end program for the user terminal 10, and the execution of the front-end program and its front-end program. It is provided with a terminal storage unit 17 that stores various information necessary for the program.

The terminal input unit 11 can be configured by, for example, a communication device for inputting an image and position information of the image by wireless communication or wired communication. The terminal input unit 11 may be configured to include a storage medium interface device for inputting an image from a storage medium such as a memory card, and an operation device for receiving an operation for inputting image position information from a person. The terminal transmitting unit 12 and the terminal receiving unit 13 can be configured by a communication device for communicating with other devices including the server device 20 via the network NW. The terminal display unit 14 can be configured by a display device such as a liquid crystal display device. The terminal operation unit 15 can be configured by an operation device such as a keyboard, a mouse, and a touch panel. The terminal control unit 16 can be configured by, for example, a CPU (central processing unit). The terminal storage unit 17 can be configured by a memory device.

The server device 20 receives from the user terminal 10 a plurality of images and information necessary for image composition including position information of the plurality of images, and a receiving unit 22 and an image processing result in the server device 20 are transmitted to the user terminal 10. Image analysis of the composite image or the image before synthesis with the transmission unit 24 for transmitting the image, the composite image generation unit 26 for generating a composite image showing the surface of the structure based on a plurality of images and information necessary for image synthesis. A crack detection unit 28 that detects cracks on the surface of the structure, a crack information generation unit 30 that generates crack information indicating a crack image corresponding to the detected cracks and a feature amount of the detected cracks, and a user. It includes a machine learning unit 32 that performs machine learning to detect cracks based on a user operation on the terminal 10. The transmission unit 24 transmits the composite image and the crack information to the user terminal 10, and displays the composite image and the crack information on the screen of the user terminal 10. The receiving unit 22 receives from the user terminal 10 a request for correcting crack information according to the user operation on the user terminal 10. The crack information generation unit 30 corrects the crack information based on the correction request. The transmission unit 24 transmits the corrected crack information to the user terminal 10. The database 50 associates the information received from the user terminal 10 (a plurality of images, information necessary for image composition, manual operation information, etc.) with the image processing result (composite image, crack information, etc.) of the server device 20. accumulate.

Further, the server device 20 includes a scale image generation unit 34 that generates a scale image for measuring the width of the crack indicated by the crack image on the screen of the user terminal 10 based on the generated crack information. The crack information generation unit 30 adds the generated scale image to the crack information. The transmission unit 24 transmits the crack information to which the scale image is added to the user terminal 10.

The server device 20 of this example includes a crack evaluation unit 36 that evaluates the degree of cracks based on at least the crack information and determines the evaluation category of the degree of cracks in the structure (hereinafter, also referred to as “rank information”). The crack information generation unit 30 adds the determined rank information to the crack information. The transmission unit 24 transmits the crack information to which the rank information is added to the user terminal 10.

Further, the server device 20 acquires CAD data by converting the composite image and crack information into CAD (computer aided design) data in a predetermined data format (of the "data conversion unit"). It is a form).

Further, the server device 20 includes a form creation unit 40 that creates a form (hereinafter, also referred to as “inspection record”) showing the inspection result of the structure including the evaluation classification of the determined crack degree. The transmission unit 24 transmits the created form to the user terminal 10.

Further, the server device 20 includes a storage unit 42 that stores a back-end program and various information necessary for executing the back-end program.

The receiving unit 22 and the transmitting unit 24 can be configured by a communication device for communicating with other devices including the user terminal 10 via the network NW. The composite image generation unit 26, the crack detection unit 28, the crack information generation unit 30, the machine learning unit 32, the scale image generation unit 34, the crack evaluation unit 36, the CAD data acquisition unit 38, and the form creation unit 40 are composed of, for example, a CPU. can do. It may be configured by a plurality of CPUs. The storage unit 42 can be configured by a memory device.

The position information of the image transmitted from the user terminal 10 to the server device 20 (hereinafter, also referred to as “image position information”) indicates the positional relationship of each image with respect to the surface of the structure (that is, a plurality of objects on the surface of the structure). There are cases where each position of the imaging region is shown) and cases where the relative positional relationship between a plurality of images is shown.

<Image correction>
The composite image generation unit 26 of the server device 20 synthesizes a plurality of images after image correction. The composite image generation unit 26 of this example can derive correction parameters based on a plurality of images and perform correction on a plurality of images. The composite image generation unit 26 of this example has a function of performing at least one of the following image corrections 1 to 4 on a plurality of images.

Image correction 1 (image position information correction): Image processing that matches the relative positional relationship between a plurality of images with the relative positional relationship between a plurality of imaged areas on an imaged surface (for example, a deck of a bridge). Is. That is, when it is determined that the inaccurate image position information is received from the user terminal 10, the inaccurate image position information is corrected to more accurate image position information. For example, based on a crack vector that vectorizes cracks in an image, if there are cracks that straddle adjacent images, but the crack images are separated and not connected between adjacent images, the cracks are adjacent. Connect between the images you want to do. This correction may be performed based on the non-crack vector obtained by vectorizing the non-crack (for example, the pattern of the structure) of the structure in the image. When the image is taken for each coffer of the floor slab 2 of the bridge 1, the image position information may be corrected based on the edge of the coffer in the image. The composite image generation unit 26 of this example has a function of correcting image position information based on the image at the time of the latest inspection, as well as an image at the time of the latest inspection and an image at the time of the past inspection (composite image or before compositing). It has a function to correct image position information based on the image). Thereby, it is possible to easily correct the inaccurate image position information to the accurate image position information. If it is determined that accurate image position information has been received from the user terminal 10, this correction does not have to be performed.

Image correction 2 (scale correction): Image processing (also referred to as “scale correction”) for matching the scales of a plurality of images. That is, when it is determined that a plurality of images in which the distances from the image pickup device 60 to the imaged surface of the structure do not match are received from the user terminal 10, a plurality of images having more matched distances to the imaged surface (““ It is also called "image of a certain distance"). If the image is associated with distance information, it may be scaled based on the distance information. For example, based on a crack vector obtained by vectorizing cracks in an image, if there are a plurality of cracks straddling adjacent images but the intervals between the cracked images do not match between the adjacent images, the plurality of cracks. Image processing that matches the spacing between images between adjacent images can be mentioned. The non-cracking of the structure in the image (for example, the pattern of the structure) may be corrected based on the vectorized non-cracking vector. When the image is taken for each coffer of the floor slab 2 of the bridge 1, the scale may be corrected based on the edge of the coffer in the image. The composite image generation unit 26 of this example has a function of performing scale correction based on the image at the time of the latest inspection, as well as an image at the time of the latest inspection and an image at the time of the past inspection (composite image or image before composition). It has a function to perform scale correction based on. When divided imaging is performed at a fixed distance, this correction does not have to be performed.

Image correction 3 (tilt correction): Image processing for correcting the tilt of each of a plurality of images. That is, when it is determined that a plurality of images in which the tilt angles of the imaged surface of the structure do not match with respect to the imaging direction of the imaging device are received from the user terminal 10, the plurality of images having more matched tilt angles of the imaged surface. (Also called "image with constant tilt angle"). When the tilt angle information is associated with the image, the tilt correction may be performed based on the tilt angle information. For example, the directions of cracks are matched based on the crack vector obtained by vectorizing the cracks in the image. The non-cracking of the structure in the image (for example, the pattern of the structure) may be corrected based on the vectorized non-cracking vector. In the case of taking an image for each coffer of the floor slab 2 of the bridge 1, the tilt correction may be performed based on the edge of the coffer in the image. The composite image generation unit 26 of this example has a function of performing tilt correction based on the image at the time of the latest inspection, as well as an image at the time of the latest inspection and an image at the time of the past inspection (composite image or image before composition). It has a function to correct the tilt based on. When divided imaging is performed at a constant tilt angle, this correction does not have to be performed.

Image correction 4 (rotation angle correction): This is an image process for correcting the rotation angle of a plurality of images with respect to the imaged surface. That is, when it is determined that a plurality of images in which the rotation angles (also referred to as "azimuths") of the imaging device with respect to the image-imposed surface of the structure do not match are received from the user terminal 10, the image-imposed surface thereof. It is corrected to a plurality of images (also referred to as "images having a constant rotation angle") having more matching rotation angles. When the rotation angle information is associated with the image, the rotation may be performed based on the rotation angle information. For example, the directions of cracks are matched based on the crack vector obtained by vectorizing the cracks in the image. The non-cracking of the structure in the image (for example, the pattern of the structure) may be corrected based on the vectorized non-cracking vector. In the case of taking an image for each coffer of the floor slab 2 of the bridge 1, the rotation angle may be corrected based on the edge of the coffer in the image. In addition to the function of correcting the rotation angle based on the image at the time of the latest inspection, the composite image generation unit 26 of this example has an image at the time of the latest inspection and an image at the time of the past inspection (composite image or image before composition). ), It has a function to correct the rotation angle. When divided imaging is performed at a constant rotation angle, this correction does not have to be performed.

<Batch correction of crack information>
The crack information correction process performed by the crack information generation unit 30 of the server device 20 of this example according to the user operation on the user terminal 10 will be described.

The crack information generation unit 30 of the server device 20 of this example has a function of collectively correcting crack information for a plurality of cracks based on the feature amount of the crack instructed by the user on the user terminal 10 (hereinafter referred to as "batch correction"). ). The receiving unit 22 of the server device 20 receives from the user terminal 10 a correction request indicating the correction content of the crack information by the feature amount of the crack. The crack information generation unit 30 of the server device 20 corrects the crack information based on the feature amount indicated by the correction request. Various modes can be mentioned as a mode of correction (collective correction) of crack information based on such a feature amount.

As a first mode of batch correction, there is a mode in which batch correction is performed according to a manual operation (hereinafter referred to as “slide operation”) using a slider displayed on the screen of the user terminal 10. The crack information generation unit 30 of this embodiment has a frequency distribution (also referred to as a “histogram”) indicating the detection frequency of a line segment (hereinafter, also referred to as a “linear pattern”) in a composite image with respect to the feature amount, and crack information based on the feature amount. A slider for instructing and inputting the correction content of the above is added to the crack information transmitted to the user terminal 10, and the crack information is corrected according to the operation using the slider on the user terminal 10.

FIG. 3 is a diagram showing a frequency distribution and a first display example of the slider. In this example, the crack images CR1, CR2, CR3, and CR4 showing the detected cracks are displayed on the composite image IMG1 displayed on the screen of the user terminal 10. Further, in the vicinity of the composite image IMG1, the frequency distribution HG1 (histogram) indicating the detection frequency of the line segment (crack candidate) in the composite image IMG1 with respect to the feature amount and the correction content of the crack information are instructed and input by the feature amount. To display the slider SL1 for. The y direction of the frequency distribution HG1 (vertical direction of the screen) indicates the feature amount of the line segment that is a candidate for cracking, and the x direction of the frequency distribution HG1 (horizontal direction of the screen) indicates the frequency of the line segment that is a candidate for cracking. The slider SL1 is a slide-type GUI (graphical user interface) for adjusting a criterion for determining whether or not a line segment in the composite image IMG1 is a crack. The slider SL1 of this example has one bar BR1, and the crack target region AR1 in the frequency distribution HG1 is determined with reference to the display position of the one bar BR1. In the vertical direction (x direction) of FIG. 3, in the frequency distribution HG1, the area above the display position of the bar BR1 is the crack target area AR1, and the line segment having the feature amount in the crack target area AR1 is ". Detected as "crack".

FIG. 4 is a diagram showing a frequency distribution and a second display example of the slider. The slider SL2 of this example has two bars BR21 and BR22, and the crack target region AR2 is determined with reference to the display positions of the two bars BR21 and BR22. In the y direction of FIG. 4 (vertical direction of the screen), the region of the frequency distribution HG1 sandwiched between the two bars BR21 and BR22 is the crack target region AR2, and the line having the feature amount in the crack target region AR2. Minutes are detected as "cracks".

FIG. 5 is a diagram showing a frequency distribution and a third display example of the slider. In this example, two sliders SL1 and SL2 can be displayed, and the crack target areas AR1 and AR2 can be selectively input based on a plurality of feature quantities.

FIG. 6 is a diagram showing a frequency distribution and a fourth display example of the slider. In this example, the check boxes BX1, BX2, and BX3 can be displayed and the type of damage can be selected and input.

Features include at least one of crack direction, length, width, edge strength, and edge density. Other features may be used. The crack information generation unit 30 adds a frequency distribution (histogram) and a slider to the crack information, and corrects the crack information according to a manual operation using the slider on the user terminal 10.

As described above, the frequency distribution indicating the detection frequency of the line segment with respect to the feature amount (the feature amount of the line segment detected by the crack detection unit 28) and the correction content of the crack information are instructed and input by the feature amount. By adding a slider to the crack information and correcting the crack information according to the slide operation using the slider on the user terminal 10, it is possible to correct the crack information with higher accuracy by a simple operation. ..

Secondly, there is an embodiment in which batch correction is performed according to a group selection operation for a cracked image displayed on the screen of the user terminal 10. The crack information generation unit 30 of this aspect groups the crack images based on the feature amount, and corrects the crack information according to a manual operation of selecting a group of the crack images on the user terminal 10.

FIG. 7 shows an example in which crack images are grouped based on a direction (a form of a feature amount) and displayed with different line types for each group. In this example, one group is formed by a plurality of crack images in the same direction and similar directions (images corresponding to line segments detected as cracks by the crack detection unit 28). The crack image of the first group G1 is displayed by a solid line, the crack image of the second group G2 is displayed by a broken line, and the crack image of the third group G3 is displayed by a dashed-dotted line. For convenience of illustration, the case where different line types are displayed for each group is illustrated, but different colors may be displayed for each group.

The user looks at the display on the screen of the user terminal 10 and performs a touch operation to enclose at least one cracked image in the group to be deleted. Then, based on the direction of the enclosed crack image, all the crack images (the crack images in the same direction and the similar direction) in the group to which the crack image belongs are deleted at once. At the same time, the crack information of the selected group is deleted by the server device 20. In the example shown in FIG. 7, the crack image of the same group included in the selection area AR surrounded by the touch operation is deleted from the screen, and the crack information of the same group is deleted.

Instead of enclosing the cracked image, the user may be allowed to directly select the group to be deleted. For example, when deleting the crack information of the second group G2 displayed by the dotted line in FIG. 7, the user terminal 10 accepts the operation of selecting the “dotted line”. When the cracked image is displayed in a different color for each group, the operation of selecting the color (for example, "red") is accepted. The operation of selecting the group identification information (for example, the group number) may be accepted.

Further, it is preferable to accept not only "deletion" of crack information but also "addition", "increase" and "decrease" operations of crack information.

As described above, by grouping the crack images based on the feature amount and correcting the crack information according to the user operation of selecting the group of the crack images on the user terminal 10, the crack with higher accuracy is performed by a simple operation. It becomes possible to correct the information.

<Machine learning>
The machine learning unit 32 of the server device 20 is composed of, for example, artificial intelligence. As the artificial intelligence, for example, an artificial neural network may be used, or another known artificial intelligence may be used. However, the present invention can be applied even when artificial intelligence is not used as the machine learning unit 32 (for example, when information is acquired by using a simple search means), and the machine learning unit 32 can be omitted. ..

As the first learning, the machine learning unit 32 machine-learns the detection of cracks based on the user operation on the user terminal 10. The machine learning unit 32 of this example has at least a history of user operations for editing crack information on the user terminal 10, an image stored in the database 50 (composite image or an image before synthesis), and a database. Based on the crack information stored in 50, the detection of cracks is machine-learned.

The machine learning unit 32 of this example learns from the editing history (history indicating the content of correction of the crack information) stored in the database 50 based on the operation history related to the crack information stored in the database 50. The feature amount of cracks to be detected from the composite image (or the image before synthesis) based on the specified edit history and the composite image and crack information corresponding to the specified edit history. Machine learning. That is, the model used for crack detection of the crack detection unit 28 is updated.

The crack information generation unit 30 of this example has a "batch correction" function of receiving a correction instruction of crack information in feature quantity units as described above. Therefore, the machine learning unit 32 of this example machine-learns the crack detection based on the correspondence relationship between the user operation history of the feature amount unit on the user terminal 10 and the composite image and the crack information. When cracks are detected by image analysis of the image before composition, machine learning is performed based on the correspondence between the operation history of the feature unit and the image before composition and the crack information.

Further, as the second learning, the machine learning unit 32 machine-learns the image correction based on the user operation on the user terminal 10. The machine learning unit 32 of this example is based on at least a history of user operations for image correction on the user terminal 10 and an image stored in the database 50 (an image before correction and an image after correction). And machine learning image correction.

The machine learning unit 32 of this example learns from the image correction history (history indicating the content of image correction) stored in the database 50 based on the operation history of the image correction stored in the database 50. The image to be corrected is specified, and the image before correction is based on the specified image correction history and the image before correction and the image after correction (which may be a composite image) corresponding to the specified image correction history. Machine learning the correspondence between image correction and the content of image correction. That is, the model used in the image correction of the composite image generation unit 26 is updated.

The composite image generation unit 26 of this example has the functions of image position information correction, scale correction, tilt correction, and rotation angle correction as described above. Therefore, the machine learning unit 32 of this example includes a history of user operations for image position information correction, scale correction, tilt correction, or rotation angle correction on the user terminal 10, and an image before correction and an image after correction (composite image). Image correction is machine-learned based on the correspondence with (may be sufficient).

The progress of artificial intelligence technology is remarkable, and artificial intelligence that can perform image analysis beyond the brains of excellent people is expected. However, when an unlearned event occurs (for example, in the initial stage of learning), a correction operation may be required. Therefore, even when artificial intelligence is adopted as the machine learning unit 32, a function of editing crack information by a simple user operation, as in the above-mentioned "batch correction", is implemented in the server device 20 together with the machine learning function. Is preferable.

<First image processing example>
FIG. 8 is a flowchart showing the flow of the first image processing example in the first embodiment.

The front-end process on the left side of the figure is started at the user terminal 10 according to the program (front-end program) stored in the terminal storage unit 17 of the user terminal 10 (step S2). Further, the back-end processing on the right side of this figure is started in the server device 20 according to the program (back-end program) stored in the storage unit 42 of the server device 20 (step S4).

In this example, as shown in FIG. 9, a plurality of images obtained by the imaging device 60 at the inspection site are input to the user terminal 10 of the office (step S6), and are entered in the field note by the user at the inspection site. The image position information is input to the user terminal 10 of the office (step S8).

The plurality of images are obtained by dividing the space between the decks of the bridge into a plurality of images by the image pickup device 60. These plurality of images are input by the terminal input unit 11 using, for example, wired communication or short-range wireless communication. You may input from a storage medium such as a memory card. A plurality of images may be uploaded from the image pickup apparatus 60 to a storage device (not shown) on the network NW, and a plurality of images may be input from the storage device via the network NW.

The field note contains information obtained by the user visually observing the deck of the bridge. Information measured using a measuring instrument (not shown) may be entered in a field note. In the following description, it is assumed that at least handwritten information (handwritten image position information) indicating the positional relationship of a plurality of images with respect to the deck of the bridge is entered in the field note.

If a CAD (computer aided design) drawing showing the floor slab of the bridge has been created, the CAD drawing is displayed on the terminal display unit 14, and the floor slab in the displayed CAD drawing is photographed. The terminal operation unit 15 receives a manual operation for fitting each image with respect to the position. In response to such a manual operation, image position information indicating the positional relationship of each image with respect to the reference position of the floor slab in the CAD drawing is input.

When the CAD drawing showing the deck of the bridge has not been created, the terminal operation unit 15 accepts the manual operation of arranging a plurality of images on the screen of the terminal display unit 14, as shown in FIG. In response to such a manual operation, image position information indicating a relative positional relationship between a plurality of images is input. In this aspect, in the server device 20, it is preferable to convert the relative positional relationship between the plurality of images into the positional relationship of each image with respect to the reference position of the deck. You may accept a manual operation of superimposing common parts between adjacent images.

Further, if necessary, information (hereinafter referred to as "correction hint information") suggestive of later image correction (step S16) is input to the user terminal 10 (step S10).

For example, "all front" when all of a plurality of images are taken from the front of the deck, "all tilt" when all of the multiple images are taken from the non-front of the deck, and among the multiple images. The terminal operation unit 15 accepts a manual operation for inputting hint information for tilt correction, such as "partially tilt" when only a part of the image is taken from the non-front of the floor slab. You may accept the input of a specific tilt angle (the tilt angle of the floor slab with respect to the imaging direction of the imaging device 60), but in this case, an approximate angle may be input, and a later step In the server device 20, it is preferable to correct the approximate angle to a detailed angle. Input of correction hint information for other image correction such as scale correction (scale correction) and rotation angle correction may be accepted. If tilt correction, scale correction, and rotation angle correction are not required, this step may be omitted.

Next, the terminal control unit 16 of the user terminal 10 generates image processing request information, and the terminal transmission unit 12 of the user terminal 10 transmits the generated image processing request information to the server device 20 (step S12). ). The image processing request information is received by the receiving unit 22 of the server device 20 and registered in the database 50 (step S14).

The image processing request information includes a plurality of images input in step S6 and image position information input in step S8. When the correction hint information is input in step S10, the correction hint information is also included in the image processing request information.

Next, the composite image generation unit 26 of the server device 20 corrects a plurality of images (step S16). For example, among the above-mentioned image corrections (image position information correction, scale correction, tilt correction, and rotation angle correction), necessary corrections are performed. When the correction hint information is input in step S10, the image correction is performed based on the correction hint information.

Next, the composite image generation unit 26 of the server device 20 generates one composite image for each floor slab coffer based on a plurality of images and image position information (step S18). The composite image is registered in the database 50.

Next, the machine learning unit 32 of the server device 20 determines whether or not the composite image is inappropriate (step S20). The machine learning unit 32 learns an event in which the composite image is appropriate and an event in which the composite image is inappropriate based on various information stored in the database 50, and the composite image this time is not available based on the learning result. It is possible to judge whether it is appropriate or not. The machine learning unit 32 can have, for example, a function of detecting an abnormality such as a pattern on a structure not being connected between original images (images before composition) as a learning result. If it is determined that the composite image is inappropriate (YES in step S20), the parameters related to image correction and image composition are changed to return to step S16, and if it is determined to be appropriate (NO in step S20). In the case of), the process proceeds to step S22.

Next, the crack detection unit 28 of the server device 20 analyzes the composite image of each deck of the floor slab, and detects the crack of the deck for each coffer (step S22).

Next, the crack information generation unit 30 of the server device 20 generates crack information indicating the crack image corresponding to the detected crack and the feature amount of the detected crack (step S24). The feature amount includes information necessary for discriminating the evaluation classification of the degree of cracks in the deck, such as the width, length, and spacing of cracks. In addition, the feature amount includes information necessary for batch editing. The generated crack information is registered in the database 50.

In this example, the frequency distribution, slider, group information, etc. required for batch editing are added to the crack information. Further, the scale image generated by the scale image generation unit 34 may be added to the crack information. Specific examples of the scale image will be described later.

The crack evaluation unit 36 evaluates the degree of cracks in the deck of the bridge based on at least the crack information, generates rank information indicating the evaluation classification of the degree of cracks, and adds the rank information to the crack information. May be good. In the following, for convenience of explanation, it is assumed that the rank information is not added to the crack information.

Next, the transmission unit 24 of the server device 20 transmits the composite image and the crack information for each floor slab to the user terminal 10 (step S26). The composite image and the crack information are received by the terminal receiving unit 13 of the user terminal 10 and displayed on the terminal display unit 14 (step S28).

For example, as shown in FIG. 11, on the screen of the terminal display unit 14, the composite image IMG2 for each floor slab and the crack information including the crack image CR are displayed. In this example, the crack image CR is superimposed on the position where the crack is detected in the composite image IMG2. Although the scale image is omitted in FIG. 11, the scale image may be further superimposed on the composite image IMG2 and displayed.

Next, the terminal operation unit 15 of the user terminal 10 determines whether or not the correction operation of the crack information has been accepted from the user (step S32), and when the correction operation is accepted (YES in step S32), the user terminal 10 The terminal transmission unit 12 of the above transmits a request for correction of crack information corresponding to the correction operation of the user to the server device 20 (step S34). The correction request is received by the receiving unit 22 of the server device 20 and registered in the database 50 (step S36).

Further, the user can see the display of the terminal display unit 14 and input the rank information indicating the evaluation classification of the degree of cracking by the terminal operation unit 15.

FIG. 12 is a diagram schematically showing an example of a crack progress model. In this progression model, the rank information is represented by five stages of "RANK1" to "RANK5". The correspondence between each rank information and the cracked state is as follows.

RANK1: No damage.

RANK2: A state in which a plurality of cracks are generated in parallel along the lateral direction (the short direction orthogonal to the traffic direction of the vehicle). This is the stage where multiple cracks due to drying shrinkage form parallel beams.

RANK3: A state in which cracks in the vertical direction (longitudinal direction parallel to the traffic direction of the vehicle) and cracks in the horizontal direction intersect each other. This is a stage in which a plurality of cracks are formed into a grid pattern due to live load, and the crack density in the grid pattern region is increased. In the latter half of the period, cracks penetrate the deck in the vertical direction (vertical direction orthogonal to the lower surface of the deck).

RANK4: A state in which the crack density of the grid-like region exceeds a specified value and the fracture surfaces of a plurality of cracks that have penetrated are smoothed. This is the stage where the deck loses its shear resistance due to the polishing action.

RANK5: A state in which omission has occurred. Falling out occurs due to a wheel load that exceeds the reduced punching shear strength.

Next, the crack information generation unit 30 of the server device 20 corrects the crack information based on the received correction request (step S38). When the rank information is input by the user terminal 10, the rank information is added to the crack information.

Next, the transmission unit 24 of the server device 20 transmits the corrected crack information to the user terminal 10 (step S40). The modified crack information is received by the terminal receiving unit 13 of the user terminal 10 and displayed on the terminal display unit 14 (step S42).

Next, when the terminal operation unit 15 of the user terminal 10 determines whether or not the input of the form creation instruction is accepted from the user (step S44) and the input of the form creation instruction is accepted (YES in step S44), The form request is transmitted to the server device 20 by the terminal transmission unit 12 of the user terminal 10 (step S46). The form request is received by the receiving unit 22 of the server device 20 (step S48).

Next, the form creation unit 40 of the server device 20 creates a form (inspection record) showing the inspection result of the deck of the bridge (step S50). The form contains rank information indicating the evaluation classification of the degree of cracks in the deck of the bridge.

Next, the transmission unit 24 of the server device 20 transmits the form to the user terminal 10 (step S52). The form is received by the terminal receiving unit 13 of the user terminal 10, and the terminal display unit 14 notifies the user of the form reception (step S54). The form can be displayed on the terminal display unit 14 according to the operation of the user, and can also be printed on a printer (not shown) connected to the network NW.

FIG. 13 is a diagram showing an example of a damage diagram as a form. In this damage diagram, for each damage that occurred on the deck of the bridge to be inspected, the damage display (crack display, water leakage display, free lime display, etc.), member name ("deck", etc.), The element number (Ds0201 etc.), the type of damage (“crack”, “water leakage”, “free lime”, etc.), and the evaluation classification (rank information) of the degree of damage are described. In FIG. 13, the five-level evaluation categories (“RANK1” to “RANK5”) shown in FIG. 12 are represented by the alphabets “a” to “e”.

<Second image processing example>
FIG. 14 is a flowchart showing the flow of the second image processing example in the first embodiment.

Steps S2 to S16 of this example are the same as the first image processing example shown in FIG.

In this example, each image before composition is image-analyzed to detect cracks (step S218), and a composite image is generated (step S220). Next, the machine learning unit 32 determines whether or not the composite image is inappropriate (step S222). The machine learning unit 32 learns an event in which the composite image is appropriate and an event in which the composite image is inappropriate based on various information stored in the database 50, and this synthesis is based on the learning result. It is possible to determine whether the image is inappropriate. The machine learning unit 32 can have, for example, a function of detecting an abnormality such as a cracked image not being connected between original images (images before composition) as a learning result. If it is determined to be inappropriate (YES in step S222), the parameters related to image correction and image composition are changed to return to step S16, and if it is determined to be appropriate (NO in step S222). , Step S24.

Steps S24 to S54 are the same as the first image processing example shown in FIG.

<Third image processing example>
FIG. 15 is a flowchart showing the flow of the third image processing example in the first embodiment.

In this example, as shown in FIG. 16, a plurality of two-viewpoint images obtained by split imaging by the imaging device 62 mounted on the robot device 70 at the inspection site are input by the user terminal 10 (step S306), and the robot The image pickup device control information according to the image pickup device control of the device 70 is input by the user terminal 10 (step S308). The image pickup device control information of this example includes image position information, distance information, tilt angle information, and rotation angle information.

The robot device 70 of this example is a suspension type robot suspended from a bridge, and FIG. 16 shows only the main parts related to the imaging thereof. The suspension type robot may be a robot that suspends from a traveling body (vehicle) on the bridge instead of suspending from the bridge. The present invention can also be applied when an unmanned flying robot such as a drone (unmanned aerial vehicle) is used instead of the suspended robot.

The image pickup device 62 of this example is a compound eye type, and acquires a two-viewpoint image (also referred to as a “stereoscopic image”) by taking an image of the imaged surface of the structure from each of the two viewpoints. The robot device 70 can calculate the distance from the image pickup device 62 to each point in the imaged surface by performing image processing on the two-viewpoint image. Further, the robot device 70 of this example includes an image pickup device control mechanism 72 that controls the rotation and movement of the image pickup device 62. The image pickup device control mechanism 72 has an XYZ movement mechanism capable of moving the image pickup device 62 in each of the X, Y, and Z directions orthogonal to each other, and the image pickup device 62 can be rotated in each of the pan direction P and the tilt direction T. It also serves as a pan-tilt mechanism. The image pickup device control mechanism 72 allows the position of the image pickup device 62 to be freely set with respect to a reference point on the surface to be imaged (surface) of the structure. That is, it is possible to freely set the position information of each of the plurality of images obtained by the plurality of divided imaging. Further, by the image pickup device control mechanism 72, the distance between the image pickup device 62 and the imaged surface of the structure, the tilt angle between the image pickup direction of the image pickup device 62 and the image imaged surface of the structure, and the image pickup direction and structure of the image pickup device 62. The rotation angle of the object with the image-imparted surface can be freely set. That is, it is possible to freely set the distance information, the tilt angle information, and the rotation angle information of each of the plurality of images obtained by the plurality of divided imaging.

The terminal input unit 11 of the user terminal 10 transmits image pickup device control information including image position information, distance information, tilt angle information, and rotation angle information together with a plurality of images via a storage medium such as a memory card or a network NW. , Can be entered.

Steps S12 to S54 are the same as the first image processing example shown in FIG.

If all the information necessary for image composition or image correction such as image position information, distance information, tilt angle information, and rotation angle information is accurately acquired by measurement, an appropriate composite image is generated. Therefore, step S20 (determination of whether or not the composite image is inappropriate) can be omitted.

<Fourth image processing example>
FIG. 17 is a flowchart showing a flow of a fourth image processing example in the first embodiment.

Steps S2, S4, S306 to S308, and S12 to S16 of this example are the same as the third image processing example shown in FIG. Similar to the third image processing example, a plurality of two-viewpoint images obtained by split imaging by the compound-eye image pickup device 62 mounted on the robot apparatus 70 are input (step S306), and the robot apparatus 70 is imaged. The image pickup device control information according to the image pickup device control of the device control mechanism 72 is input (step S308).

Steps S218 to S222 are the same as the second image processing example shown in FIG. That is, after analyzing the image before composition in step S218 and detecting cracks, the composite image is generated and registered in step S220, and it is determined in step S222 whether or not the composite image is inappropriate. Step S222 can be omitted as in S20 of the third image processing example.

Steps S24 to S54 are the same as in the first image processing example.

An image processing example (first image processing example and second image processing example) when a person operates an image pickup device to image a structure, and a robot device 70 controls the image pickup device to capture the structure. Image processing examples (third image processing example and fourth image processing example) in the case of imaging have been described separately for convenience, but the present invention is carried out in a closed configuration in either case. It is not limited to cases, but rather it is preferable to carry out in an open configuration including both cases. That is, the server device 20 of this example not only stores an image when a person operates the image pickup device, information necessary for image composition and correction (including image position information), an operation history, and the like in the database 50. , The image when the robot device 70 controls the image pickup device, the image pickup device control information (including the image position information), the operation history, and the like are stored in the database 50, and machine learning is performed in both cases. As a result, even when a person operates an image pickup device to image a structure, a composite image more appropriate than a closed configuration can be obtained based on the machine learning result of image processing when the robot device 70 controls the image pickup device. It becomes possible to generate.

[Second Embodiment]
FIG. 18 is a block diagram showing a configuration example of the image processing system according to the second embodiment. The user terminal 10 in this figure is the same as that of the first embodiment. The server device 200 of this figure has a configuration in which a prediction unit 44 for predicting the progress of cracks is added to the server device 20 of the first embodiment shown in FIG.

The prediction unit 44 predicts the progress of cracks in the bridge based on various information about the bridge stored in the database 50 and the crack information generated by the crack information generation unit 30. The transmission unit 24 transmits the prediction result of the progress of cracks to the user terminal 10.

The database 50 of this example stores bridge structural information, bridge environmental condition information, bridge usage condition information, bridge maintenance history information, and bridge disaster history information. The prediction unit 44 of this example includes at least one of bridge structural information, bridge environmental condition information, bridge usage condition information, bridge maintenance history information, and bridge disaster history information, and bridge crack information. Based on the above, the progress of cracks in the bridge is predicted.

The rank information is, for example, an evaluation classification for each inspection site of a bridge (for example, a floor slab coffer). The rank information may include "health", which is a gradual or numerical representation of the soundness of the bridge.

The structural information of the bridge indicates the structure of at least the part of the entire bridge including the floor slab to be evaluated and the members around it. For example, structural form and structural features can be mentioned. Includes structural information related to the progress of cracks in the deck, such as the relationship between loads and decks, the relationship between earthquake vibrations and decks, the relationship between rainwater and decks, and the relationship between temperature and decks. .. Chemical information such as whether or not a material that easily causes an alkaline aggregate reaction is used may be used. Information that directly represents a part where cracks are likely to occur may be used.

The environmental condition information of the bridge includes, for example, the location condition such as whether or not salt comes in.

Information on the conditions for using bridges includes, for example, the traffic volume of vehicles and the mixing rate of large vehicles.

Examples of bridge maintenance history information include spraying history information of antifreeze agents. It may include history information of repairs and reinforcements.

The disaster history information of a bridge is the history information of disasters such as typhoons and earthquakes received by the bridge.

The database 50 of this example further stores rank information indicating the evaluation classification of the degree of cracking of the bridge in association with the composite image of the bridge and the crack information. The prediction unit 44 of this example executes a simulation process for predicting how the rank information of the bridge evaluated by the crack evaluation unit 36 will change in the future.

In addition, the prediction unit 44 may simulate how the change has occurred from the past to the present. The crack information generation unit 30 can further improve the accuracy of crack information by generating crack information based on the simulation results from the past to the present and the current image (composite image or image before synthesis). can.

The machine learning unit 32 of this example can perform machine learning of crack detection based on the prediction result of the prediction unit 44 in addition to the operation history, the image (composite image or the image before composition), and the crack information. Is. The machine learning unit 32 further cracks based on at least one of bridge structural information, bridge environmental condition information, bridge usage condition information, bridge maintenance history information, and bridge disaster history information. Learning detection is preferred.

[Vectorization]
The vectorization of the cracked image will be described with reference to FIGS. 19 to 22.

The crack information generation unit 30 of the server device 20 vectorizes the crack portion extracted from the composite image or the image before synthesis to generate crack vector data (hereinafter, referred to as “crack vector” or simply “vector”). That is, the crack information generation unit 30 vectorizes the crack portion. The transmission unit 24 of the server device 20 transmits the crack information including the crack vector, which is a vectorized crack image, to the user terminal 10 together with the composite image. The vectorized crack image is superimposed and displayed on the composite image on the terminal display unit 14 of the user terminal 10.

The crack information generation unit 30 binarizes and / or thins the detected cracks as necessary at the time of vectorization. "Vectoring" is to find a line segment determined by the start point and end point for a crack. When the crack is curved, the crack is divided into multiple sections so that the distance between the curve and the line segment is less than the threshold. Then, a crack vector is generated for each of the plurality of intervals.

When generating the crack vector, for example, the feature point of the floor slab 2 is set as the origin of the coordinate system, and for the group of crack vectors (vector group), the end point at which the distance from the origin is minimized is set as the first starting point, and the following crack vector It is possible to determine the end point and the start point in sequence along the traveling direction of. In the example of FIG. 19, when the point P0 on the floor slab 2 is the origin of the coordinate system and the right and downward directions of the figure are the X-axis direction and the Y-axis direction of the coordinate system, respectively, the points P13 of the vector group C7, Of P14, P15, and P16, the point P13 where the distance d from the point P0 is the shortest is set as the start point of the crack vector C7-1, and hereinafter, the point P14 is the end point of the crack vector C7-1 (and the crack vectors C7-2 and C7). The start point of -3) and points P15 and P16 can be set as the end points of the crack vectors C7-2 and C7-3, respectively.

However, when the start point of the vector group C8 is determined by the same method, the point P17 becomes the start point of the crack vector C8-1, the point P18 becomes the start point of the crack vectors C8-2 and C8-3, and the traveling direction of the crack vector C8-3. (The direction from the point P18 to the point P20) is opposite to the traveling direction of the crack vector C8-1. Therefore, in such a case, as shown in FIG. 20, the point P19 is set as the start point of the crack vector C8A-1, and hereinafter, the point P18 is the end point of the crack vector C8A-1 (and the start point of the crack vectors C8A-2 and C8A-3). ), Points P17 and P20 may be set as the end points of the crack vectors C8A-2 and C8A-3, respectively. The aggregate of crack vectors in this case is referred to as vector group C8A.

When the crack vector is generated as described above, if the cracks are continuous inside the deck 2 but separated on the surface, they may be recognized as separated crack vectors. Therefore, a plurality of crack vectors are concatenated to generate one or a plurality of vectors.

FIG. 21 is a diagram showing an example of connecting crack vectors, in which a vector group C3 including a crack vector C3-1 (point P21 and a point P22 are start points and end points, respectively) and a crack vector C4-1 (point P23, point P24) are shown. Indicates a situation in which the vector group C4 including the start point and the end point) is extracted. Further, the angle formed by the crack vector C3-1 with the line segment connecting the points P22 and P23 is α1, and the angle formed by the line segment connecting the points P22 and P23 with the crack vector C4-1 is α2. At this time, if both the angles α1 and the angles α2 are equal to or less than the threshold value, the crack vectors C3-1 and C4-1 are connected, and the vector groups C3 and C4 are fused. Specifically, as shown in FIG. 22, a new crack vector C5-2 is generated to generate other crack vectors C5-1 (same as the crack vector C3-1) and C5-3 (same as the crack vector C4-1). ), And a new vector group including these crack vectors C5-1, C5-2, and C5-3 is designated as a vector group C5. By appropriately connecting the spatially separated vectors (on the surface of the deck 2) in this way, the connection relationship between the vectors can be accurately grasped.

The above-mentioned vector connection is an example, and other methods may be used.

Further, the crack information generation unit 30 calculates the relative angles between the plurality of vectors. That is, the angle formed by one vector and the other vector is calculated.

Further, the crack information generation unit 30 generates hierarchical structure information indicating the hierarchical structure of the crack vector. Hierarchical structure information is information that hierarchically expresses the connection relationship between vectors. The hierarchical structure information includes hierarchical identification information (also referred to as “affiliation hierarchical information”) indicating which hierarchy each vector belongs to in a vector group composed of a plurality of vectors. As the layer identification information, for example, a layer number represented by a number can be used. It may be expressed by a code other than numbers (for example, letters and symbols).

FIG. 23 is a diagram showing the vector group C1. The vector group C1 is composed of crack vectors C1-1 to C1-6, and these crack vectors have points P1 to P7 as a start point or an end point. In such a situation, each time the crack vector branches (the end point of one crack vector is the start point of a plurality of other crack vectors), the hierarchy is lowered. Specifically, the hierarchy of the crack vector C1-1 is set as the highest "level 1", and the layers of the crack vectors C1-2 and C1-3 starting from the point P2 which is the end point of the crack vector C1-1 are cracked. Let it be "level 2", which is lower than the vector C1-1. Similarly, the hierarchy of the crack vectors C1-5 and C1-6 starting from the point P4 which is the end point of the crack vector C1-3 is set to "level 3" which is lower than the crack vector C1-3. On the other hand, the point P3 which is the end point of the crack vector C1-2 is the start point of the crack vector C1-4, but the crack vector starting from the point P3 is only the crack vector C1-4 and there is no branch, so the crack vector C1 The hierarchy of -4 is the same as C1-2, "level 2". The hierarchy of each crack vector determined in this way is included in the hierarchical structure information.

FIG. 24 is a diagram showing vector group C1 (the connection relationship between the crack vectors is the same as that shown in FIG. 23). In this example, among the crack vectors to be connected, those whose angles formed with other crack vectors are equal to or less than the threshold value (crack vectors corresponding to the "trunk" in the tree structure) belong to the same hierarchy. Specifically, the crack vectors C1-1, C1-2, and C1-4 existing in the dotted line in FIG. 24 (the range indicated by the reference code Lv1) are set to "level 1" (top level), which is the same layer. .. Regarding the other crack vectors C1-3, C1-5, and C1-6, it is said that the hierarchy becomes lower each time the crack vector branches, and the crack vector C1-3 (“branch” in the tree structure). (Equivalent) is defined as "level 2", and crack vectors C1-5 and C1-6 (corresponding to "leaves" in the tree structure) are defined as "level 3". The hierarchy and type (trunk, branch, or leaf) of each crack vector determined in this way are included in the hierarchical structure information.

The hierarchical structure described with reference to FIGS. 23 and 24 is an example, and the hierarchical structure may be formed by a modified method or a different method.

Further, the crack information generation unit 30 adds various additional information (attribute information) to the vector. For example, information indicating the width of each crack (width information) is added to each vector.

As described above, since the crack information generation unit 30 of the server device 20 vectorizes and hierarchically structures the crack image, various processes are efficiently and appropriately performed using the generated crack vector and the hierarchical structure information. Will be possible.

For example, image correction can be performed using the crack vector and the hierarchical structure information. In addition, it becomes possible to determine whether or not the composite image is inappropriate by using the crack vector and the hierarchical structure information. In addition, machine learning can be performed using the crack vector and the hierarchical structure information. In addition, the crack vector and the hierarchical structure information can be used to determine the evaluation classification of the degree of crack. In addition, it becomes possible to predict the progress of cracks by using the crack vector and the hierarchical structure information.

[Example of scale image]
An example of the scale image generated by the scale image generation unit 34 will be described. The scale image generation unit 34 generates various scale images for measuring the feature amount of the crack (for example, the width of the crack) indicated by the crack image on the screen of the user terminal 10. The scale image is added to the crack information.

FIG. 25 is a diagram showing a display example of a scale image. For convenience of illustration and description, the crack images CR11 (CR11-1, CR11-2, CR11-3) showing the cracks to be measured are shown in an enlarged manner.

Scale images CS1, CS2, CS3, and CS4 are images for measuring cracks generated on the measurement surface of a structure (in this example, the deck of a bridge). These scale images CS1 to CS4 are generated by the scale image generation unit 34 of the server device 20. The scale image generation unit 34 of this example includes scale images CS1 and CS2 showing scales with scales for measuring the length of cracks, and scales with line drawings and numbers for measuring the width of cracks. Scale images CS3 and CS4 showing the above are generated. The scale images CS1 to CS4 show scales according to the distance between the imaging device and the measuring surface and the tilt angle of the measuring surface with respect to the imaging direction of the imaging device. The scale image generation unit 34 makes the type, shape, size, etc. of the scale different according to the distance and the angle. The scale image generation unit 34 of this example makes the scales of the scale images CS1 and CS2 for length measurement different according to the distance and the tilt angle, and makes the line drawings and numbers of the scale images CS3 and CS4 for width measurement different. ing.

The crack information generation unit 30 of the server device 20 adds the scale images CS1 to CS4 to the crack information. The transmission unit 24 of the server device 20 transmits the crack information to which the scale images CS1 to CS4 are added to the user terminal 10 together with the composite image of the structure. The transmitted crack information with the scale image and the composite image are received by the terminal receiving unit 13 of the user terminal 10.

The display area DA in the terminal display unit 14 of the user terminal 10 is an image display area IA for displaying a composite image and a scale image of a structure, and a scale image of a type that can be displayed in the image display area IA (that is, the user can select the scale image). Scale image selection area SA for displaying a large scale image) is included. In the scale image selection area SA, in addition to the scale images CS1 to CS4, an OK button BU1 for confirming the selection of the scale image, a cancel button BU2 for canceling the selection of the scale image, and an image display area IA are displayed. The erase button BU3 for erasing the scale image is displayed.

The terminal operation unit 15 accepts a user's selection operation for the scale images CS1 to CS4 displayed in the scale image selection area SA. The user performs a selection operation using a button, a mouse, or the like. From the plurality of scale images CS1 to CS4 displayed in the scale image selection area SA, scale images (CS2 and CS3 in this example) corresponding to the user's selection operation are displayed in the image display area IA.

The terminal display unit 14 superimposes and displays the composite image and the scale image on the image display area IA. The position and direction of the scale image are determined according to the position and direction of the cracked image in the composite image. The crack image CR11 of this example has a hierarchical structure and includes a parent crack image CR11-1 and child crack images CR11-2 and CR11-3. The scale image CS2 for length measurement is displayed along the direction of the crack image CR11-2 and at a position near the crack image CR11-2. The scale image CS3 for width measurement is a line drawing (1.4 mm line drawing in this example) having a width closest to the width of the crack image CR11-1 along a direction orthogonal to the direction of the crack image CR11-1. It is aligned with the crack image CR11-1 and displayed.

Although not shown, the scale image selection also includes a move button for moving the scale image, a rotation button for rotating the scale image, an enlargement button for enlarging the scale image, and a reduction button for reducing the scale image. It is preferable to display it in the area SA. You may instruct the movement, rotation, enlargement, reduction, etc. of the scale image by the movement operation of the mouse.

Although the case where two scale images CS2 and CS3 are displayed in the image display area IA is illustrated, the number of scale images to be displayed in the image display area IA is not limited.

Although the embodiments for carrying out the present invention have been described above, the present invention is not limited to the above-described embodiments and modifications, and various modifications can be made without departing from the gist of the present invention.

1 Bridge 2 Floor slab 3 Main girder 3A Joint 10 User terminal 11 Terminal input unit 12 Terminal transmission unit 13 Terminal reception unit 14 Terminal display unit 15 Terminal operation unit 16 Terminal control unit 17 Terminal storage unit 20, 200 Server device 22 Receiver unit 24 Transmission unit 26 Composite image generation unit 28 Crack detection unit 30 Crack information generation unit 32 Machine learning unit 34 Scale image generation unit 36 Crack evaluation unit 38 CAD data acquisition unit (data conversion unit)
40 Form creation unit 42 Storage unit 44 Prediction unit 50 Database 60, 62 Imaging device 70 Robot device 72 Imaging device control mechanism AR Selection area AR1, AR2 Crack target area BR1, BR21, BR22 Bar BU1 OK button BU2 Cancel button BU3 Erase button BX1 , BX2, BX3 check boxes C1, C3, C4, C5, C7, C8, C8A Vector groups C1-1, C1-2, C1-3, C1-4, C1-5, C3-1, C4-1, C5 -1, C5-2, C7-1, C7-2, C8-1, C8-2, C8-3, C8A-1, C8A-2 Vector CR, CR1, CR2, CR3, CR4, CR11, CR11-1 , CR11-2, CR11-3 Cracked image CS1, CS2, CS3, CS4 Scale image DA Display area G1 First group G2 Second group G3 Third group HG1, HG2 Frequency distribution IA Image display area IMG1, IMG2 synthesis Image NW network SA scale image selection area SL1, SL2 slider d distance α1, α2 angle

Claims (7)

A step of receiving a plurality of images showing the surface of a structure from a user terminal,
A step of image-synthesizing the plurality of received images to generate a composite image showing the surface of the structure.
A step of image-analyzing the composite image or the plurality of images before synthesis to detect cracks on the surface of the structure.
A step of generating crack information indicating a crack image corresponding to the detected crack and a feature amount of the detected crack, and
A step of grouping the cracked images based on the feature amount,
A step of transmitting the composite image and the crack information to the user terminal and displaying the composite image and the crack information on the screen of the user terminal.
A step of displaying the cracked image on the screen of the user terminal using a different line type or color for each group, and
A step of receiving the operation of selecting the group from the user terminal, and
The step of receiving the correction request of the crack information for the selected group from the user terminal, and
The step of correcting the crack information in response to the received correction request, and
Image processing method including. A history of the correction request received before reporting, the steps and the composite image and the cracking information corresponding to the history, based on the, to machine learning to detect the cracks,
The image processing method according to claim 1. The image processing method according to claim 1 or 2, wherein the feature amount includes at least one of the crack direction, length, width, edge strength, and edge density. A receiver that receives a plurality of images showing the surface of a structure from a user terminal, and
A composite image generation unit that synthesizes a plurality of received images to generate a composite image showing the surface of the structure, and a composite image generation unit.
A crack detection unit that detects cracks on the surface of the structure by image-analyzing the composite image or the plurality of images before synthesis.
A crack information generation unit that generates crack information indicating a crack image corresponding to the detected crack and a feature amount of the detected crack, and groups the crack image based on the feature amount.
A transmission unit that transmits the crack information including the crack image displayed using different line types or colors for each group and the composite image to the user terminal.
A selection operation receiving unit that receives an operation for selecting the group from the user terminal, and
A correction request receiving unit that receives a correction request for the crack information for the selected group from the user terminal, and a correction request receiving unit.
A crack information correction unit that corrects the crack information in response to the received correction request,
An image processing device comprising. A history of the correction request received before reporting, and the composite image and the cracking information corresponding to the history, on the basis of image processing according to claim 4, comprising a machine learning unit for machine learning to detect the cracks Device. The image processing apparatus according to claim 4, wherein the feature amount includes at least one of the crack direction, length, width, edge strength, and edge density. An image processing program that causes a computer to execute the image processing method according to claim 1.

Images