JP2023078205A - Image processing device, image processing system, operation method of image processing device and image processing program - Google Patents

Abstract

To provide an image processing device, an image processing system, an operation method of the image processing device and an image processing program which make it possible to analyze an image of a cracked location without the need for an expensive device, and to present a composite image of the cracked location and the result of the image analysis without complicating an operation performed by a user.SOLUTION: In an image processing device, a processor acquires a plurality of images showing a surface of a structure, generates a composite image showing the surface of the structure by performing image composition of the acquired images, acquires a determination result on whether the composite image is inappropriate, acquires a correction parameter for image correction and/or image composition when the acquired determination result shows that the composite image is inappropriate, generates the composite image again on the basis of the correction parameter, and detects a crack on the surface of the structure by performing image analysis on the composite image or an image before composition.SELECTED DRAWING: Figure 2

Description

INDUSTRIAL APPLICABILITY The present invention is an image processing apparatus that can analyze an image of a crack location without requiring an expensive device, and can present a composite image of the crack location and the image analysis results without complicating the user's operation. , an image processing system, an operating method of an image processing apparatus, and an image processing program.

Structures such as bridges and tunnels exist as social infrastructure. These structures are subject to damage, and the damage tends to progress, so regular inspections are required. It is also required to report accurate inspection results. For example, it is necessary to accurately grasp the properties of cracks that occur on the surface of structures and report the inspection results, and based on the reports, it is necessary to perform appropriate maintenance and management according to the progress of cracks. be.

In Patent Document 1, when a background image and a close-up image with a laser light spot obtained by imaging a concrete wall surface with a camera with a laser projectile are sent from a mobile personal computer, the background image and the close-up image are transmitted from the front of the concrete wall surface. Normalize the viewed image, and assign which area of the background image the close-up image belongs to in order to specify the close-up image for image analysis on the mobile computer. A server device for image analysis to measure crack width is described.

Patent Document 2 describes an operation in which an image of a concrete structure is pasted onto a CAD (computer aided design) drawing based on information on the shooting position entered in a field notebook during work at an inspection site on a personal computer. , the image is associated with the CAD drawing according to the operation, and if the operation of tracing the crack on the image is performed, the crack on the image is mapped to the CAD drawing according to the operation, and the crack is detected using the information of the CAD drawing. It is described to calculate the width and length of the .

In Patent Document 3, in a personal computer, the distance from a camera to a concrete wall surface and the inclination angle of the concrete wall surface with respect to the optical axis of the camera are measured, a plurality of images are joined, and an image of the concrete wall surface viewed from the front is obtained. and to obtain the distribution of cracks on the concrete wall surface and the width of the cracks to display a state diagram. A crack that cannot be recognized even though it is actually a crack can be input by a tracing operation using a mouse or a pen input device.

In Patent Document 4, when performing machine learning for crack identification processing using a support vector machine algorithm, among a plurality of area images, the center of each area image and the identification target in each area image (cracks, dirt, etc. It is described that only region images whose centroids coincide with each other (e.g., unevenness of the surface) are selected as learning target samples.

JP-A-2004-69434 JP 2005-310044 A JP-A-10-78305 Japanese Patent Application Laid-Open No. 2003-216953

Patent Literatures 1 to 4 do not disclose that a plurality of images obtained by separately imaging a structure and the position information of each image are transmitted from the user terminal and synthesized into one image by the server device. .

In addition, the system described in Patent Document 1, which consists of a mobile personal computer (a form of user terminal) and a server device, not only requires a camera provided with a laser projectile, but also provides a background image and a close-up image. Further, it is necessary to specify the proximity image to be subjected to image analysis by the user's operation for allocating the proximity image to the background image. Therefore, it can be said that not only is an expensive device required, but also the user's operation is complicated until the image analysis result of the crack is obtained.

Further, when a user terminal generates a composite image of a cracked portion and analyzes the image of the crack, expensive hardware or software is required. In other words, an expensive device was required.

In Patent Document 2, when an operation of pasting an image on a CAD drawing and an operation of tracing a crack on the image are performed on a personal computer (one form of a user terminal), the width and length of the crack can be obtained. It is only described to calculate.

In the technique described in Patent Document 3, when correcting crack information, it is necessary to perform an operation of tracing one by one using a mouse or a pen input device. If the crack information correction function by such a tracing operation is used, cracks of any pattern can be individually corrected one by one. It may not be practical for workers.

Patent Literature 4 merely describes learning a crack identification process based on an image of a structure.

As described above, various devices have been proposed or provided to assist in accurately grasping the properties of cracks occurring on the surface of structures and reporting the inspection results. In order to present the crack image analysis results to the user, there is a problem that an expensive device is required and the burden on the user for synthesizing a plurality of images of the crack location is heavy.

INDUSTRIAL APPLICABILITY The present invention is an image processing apparatus that can analyze an image of a crack location without requiring an expensive device, and can present a composite image of the crack location and the image analysis results without complicating the user's operation. , an image processing system, an operating method of an image processing apparatus, and an image processing program.

In order to achieve the object described above, a server device according to a first aspect of the present invention includes a receiving unit that receives a plurality of images showing the surface of a structure and position information of the plurality of images from a user terminal; A synthetic image generation unit that generates a synthetic image showing the surface of the structure based on the received multiple images and position information, and an image analysis of the synthetic image or the image before synthesis to detect cracks on the surface of the structure. and a crack information generation unit that generates crack information indicating a crack image corresponding to the detected crack and the feature amount of the detected crack.

According to this aspect, when a plurality of images showing the surface of a structure and the positional information of the plurality of images are transmitted from the user terminal, a composite image is generated by the server device based on the plurality of images and the positional information of the images. Furthermore, since the crack information is generated, it is possible to analyze the image of the crack location without requiring an expensive device, and to present the composite image of the crack location and the image analysis results without complicating the user's operation. be possible. In addition, according to this aspect, it is possible to store the image transmitted from the user terminal and its position information, as well as the composite image and crack information generated by the server device, into big data. It can be said that the secondary use of big data enables us to derive appropriate synthetic images and appropriate image analysis results.

The server device according to the second aspect of the present invention includes a transmission unit that transmits the composite image and the crack information to the user terminal and displays the composite image and the crack information on the screen of the user terminal. A crack information correction request is received from the user terminal according to manual operation at the terminal, the crack information generation unit corrects the crack information based on the correction request, and the transmission unit sends the corrected crack information to the user terminal. Send. According to this aspect, when the composite image and the crack information are displayed on the screen of the user terminal, and the user performs manual operation on the contents of the display, the correction request corresponding to the manual operation is stored in the server. becomes possible. Therefore, it becomes easy to improve the image analysis processing in the server device so that more appropriate image analysis results can be derived. That is, it is possible to present more appropriate image analysis results to the user while reducing complicated user operations for correcting image processing results.

In the server device according to the third aspect of the present invention, the receiving unit receives from the user terminal a correction request indicating the content of correction of crack information in the form of a feature amount, and the crack information generating unit receives the feature indicated by the correction request Modify the crack information based on the amount. According to this aspect, it is possible to perform a user operation to indicate a crack information correction request using the crack feature amount. For example, compared to the case of correcting crack information one by one based on a tracing operation, crack information can be corrected with a simple operation.

In the server device according to the fourth aspect of the present invention, the frequency distribution indicating the detection frequency of the line segment in the composite image with respect to the feature amount and the slider for instructing and inputting the correction content of the crack information with the feature amount are combined into cracks. It is added to the information and the crack information is modified according to the user's operation using the slider on the user terminal. According to this aspect, it is possible to correct the crack information by a simple operation of operating the slider while looking at the detection frequency of the line segment with respect to the feature amount.

In the server device according to the fifth aspect of the present invention, the crack information generation unit groups the crack images based on the feature amount, and generates the crack information according to the manual operation of selecting the crack image group on the user terminal. fix it. According to this aspect, it is possible to correct crack information by a simple operation of selecting a group of crack images.

A server device according to a sixth aspect of the present invention includes a machine learning unit that machine-learns crack detection based on a manual operation at a user terminal. According to this aspect, detection of cracks is machine-learned based on manual operations at the user terminal, so it is possible to steadily reduce complicated user operations for correcting image analysis results.

In the server device according to the seventh aspect of the present invention, the feature amount includes at least one of crack direction, length, width, edge strength, and edge density.

In the image processing method according to the eighth aspect of the present invention, the composite image generator corrects at least one of image position information, scale, tilt, and rotation angle for the plurality of images.

In the server device according to the ninth aspect of the present invention, the composite image generator derives correction parameters based on the plurality of images and performs correction.

A server device according to a tenth aspect of the present invention comprises a scale image generation unit that generates a scale image for measuring the width of the crack indicated by the crack image on the screen of the user terminal based on the crack information, The crack information generator adds the scale image to the crack information.

The server device according to the eleventh aspect of the present invention includes a crack evaluation unit that evaluates the degree of cracking based on at least crack information and determines the evaluation classification of the degree of cracking of the structure.

A server device according to a twelfth aspect of the present invention includes a form creating unit that creates a form indicating inspection results of structures including the determined evaluation category.

A server device according to a thirteenth aspect of the present invention includes a data conversion unit that converts the composite image and the crack information into CAD data in a predetermined data format.

In the server device according to the fourteenth aspect of the present invention, based on at least one of structural information of a structure, environmental condition information, use condition information, maintenance history information, and disaster history information, and crack information, , and a predictor for predicting crack progression.

An image processing system according to a fifteenth aspect of the present invention is an image processing system including a user terminal and a server device, wherein the user terminal comprises a plurality of images showing surfaces of structures, position information of the plurality of images, and a terminal input unit for inputting a terminal transmission unit for transmitting a plurality of images and position information to a server device; a terminal reception unit for receiving image processing results for a plurality of images from the server device; and an image processing result and a terminal display unit that displays the structure, based on the server reception unit that receives a plurality of images and position information from the user terminal, and the received plurality of images and position information. A synthetic image generation unit that generates a synthetic image showing the surface, a crack detection unit that analyzes the synthetic image or the image before synthesis to detect cracks on the surface of the structure, and a crack image corresponding to the detected crack. A crack information generation unit that generates crack information indicating the feature amount of the detected crack, and a server transmission unit that transmits a composite image and crack information to the user terminal as image processing results for a plurality of images, The terminal display unit displays the composite image and the crack information.

In the image processing method according to the sixteenth aspect of the present invention, a step of receiving, from a user terminal, a plurality of images showing a surface of a structure and position information of the plurality of images; generating a synthetic image showing the surface of the structure based on the positional information; analyzing the synthetic image or the pre-synthesis image to detect cracks on the surface of the structure; and responding to the detected cracks. and generating crack information indicating the crack image and the feature quantity of the detected crack.

According to the present invention, it is possible to analyze an image of a crack location without requiring an expensive device, and to present a composite image of the crack location and the image analysis results without complicating the user's operation.

FIG. 1 is a perspective view showing a bridge as an example of a structure to which the present invention is applied. FIG. 2 is a block diagram showing a configuration example of an image processing system according to the first embodiment. FIG. 3 is a diagram showing a first display example of the frequency distribution and the slider. FIG. 4 is a diagram showing a second display example of the frequency distribution and the slider. FIG. 5 is a diagram showing a third display example of the frequency distribution and the slider. FIG. 6 is a diagram showing a fourth display example of the frequency distribution and the slider. FIG. 7 is a diagram showing a display example when crack images are grouped based on feature amounts. FIG. 8 is a flow chart showing the flow of the first image processing example in the first embodiment. FIG. 9 is an explanatory diagram used for explaining the first image processing example in the first embodiment. FIG. 10 is an explanatory diagram used for explaining the operation of arranging a plurality of images. FIG. 11 is a diagram showing a display example of a composite image and a crack image. FIG. 12 is a diagram schematically showing an example of a crack progression model. FIG. 13 is a diagram showing an example of a damage diagram as a form. FIG. 14 is a flow chart showing the flow of the second image processing example in the first embodiment. FIG. 15 is a flow chart showing the flow of the third image processing example in the first embodiment. FIG. 16 is an explanatory diagram used for explaining the third image processing example in the first embodiment. FIG. 17 is a flow chart showing the flow of the fourth example of image processing in the first embodiment. FIG. 18 is a block diagram showing a configuration example of an image processing system according to the second embodiment. FIG. 4 is a first explanatory diagram used for explaining vectorization of crack images; FIG. 11 is a second explanatory diagram used for explaining vectorization of crack images; FIG. 11 is a third explanatory diagram used for explaining vectorization of a crack image; FIG. 11 is a fourth explanatory diagram used for explaining vectorization of a crack image; FIG. 11 is a fifth explanatory diagram used for explaining vectorization of a crack image; FIG. 11 is a sixth explanatory diagram used for explaining vectorization of a crack image; FIG. 10 is a diagram showing a display example of a scale image;

An image processing apparatus, an image processing system, an operation method of the image processing apparatus, and an image processing program according to the present invention will be described below with reference to the accompanying drawings.

[Structure example]
FIG. 1 is a perspective view showing the structure of a bridge, which is an example of a structure to which the present invention is applied. The bridge 1 shown in FIG. 1 has a plurality of main girders 3, and the main girders 3 are joined together at joints 3A. A floor slab 2 made of concrete is placed on top of the main girder 3 . The bridge 1 has members such as a lateral girder, a tilting structure, and a lateral structure (not shown) in addition to the floor slab 2 and the main girder 3 .

In addition, the "structure" in the present invention is not limited to a bridge. A structure may be a tunnel, a dam, a building, or the like.

[First embodiment]
FIG. 2 is a block diagram showing a configuration example of an image processing system according to the first embodiment.

The user terminal 10 and the server device 20 constitute a client-server type image processing system. A plurality of user terminals 10 are actually connected to the network NW. The user terminal 10 is configured by a computer device that performs front-end processing for the user, and the server device 20 is configured by a computer device that performs back-end processing such as image processing. As the user terminal 10, for example, a computer device such as a personal computer, tablet, or smart phone can be used. The server device 20 may be composed of a plurality of computer devices. The server device 20 of this example includes a database 50 for convenience of explanation. The server device 20 and the database 50 may be provided as different devices.

The user terminal 10 includes a terminal input unit 11 for inputting a plurality of images showing the surface of a structure and information necessary for image synthesis including position information of the plurality of images, and a terminal input unit 11 for inputting the plurality of input images and image synthesis. a terminal transmitting unit 12 for transmitting necessary information to the server device 20; a terminal receiving unit 13 for receiving image processing results for a plurality of images from the server device 20; and a terminal for displaying the image processing results of the server device 20. a display unit 14, a terminal operation unit 15 for accepting a user's manual operation, a terminal control unit 16 for controlling each unit of the user terminal 10 according to the front end program for the user terminal 10, a front end program, and execution of the front end program. and a terminal storage unit 17 for storing various kinds of information necessary for

The terminal input unit 11 can be configured by, for example, a communication device that inputs an image and position information of the image by wireless communication or wired communication. The terminal input unit 11 may include a storage medium interface device for inputting an image from a storage medium such as a memory card, and an operation device for receiving an input operation of image position information from a person. The terminal transmission unit 12 and the terminal reception unit 13 can be configured by communication devices for communicating with other devices including the server device 20 via the network NW. The terminal display unit 14 can be configured by a display device such as a liquid crystal display device. The terminal operation unit 15 can be configured by an operation device such as a keyboard, mouse, touch panel, or the like. The terminal control unit 16 can be configured by, for example, a CPU (central processing unit). The terminal storage unit 17 can be configured by a memory device.

The server device 20 includes a receiving unit 22 that receives a plurality of images and information necessary for image composition including position information of the plurality of images from the user terminal 10, and an image processing result in the server device 20 to the user terminal 10. a transmission unit 24 for transmitting a plurality of images and information necessary for image synthesis, a synthetic image generation unit 26 for generating a synthetic image showing the surface of a structure based on the information necessary for image synthesis, and an image analysis of the synthetic image or the image before synthesis a crack detection unit 28 for detecting cracks on the surface of the structure by doing so, a crack information generation unit 30 for generating crack information indicating a crack image corresponding to the detected crack and the feature amount of the detected crack, and a user and a machine learning unit 32 that machine-learns detection of cracks based on user operations on the terminal 10 . The transmission unit 24 transmits the composite image and the crack information to the user terminal 10 and causes the screen of the user terminal 10 to display the composite image and the crack information. The receiving unit 22 receives from the user terminal 10 a correction request for crack information according to a user operation on the user terminal 10 . The crack information generator 30 corrects the crack information based on the correction request. The transmission unit 24 transmits the corrected crack information to the user terminal 10 . The database 50 associates information received from the user terminal 10 (multiple images, information necessary for image synthesis, manual operation information, etc.) with image processing results (composite image, crack information, etc.) in the server device 20. accumulate.

The server device 20 also includes a scale image generation unit 34 that generates a scale image for measuring the crack width indicated by the crack image on the screen of the user terminal 10 based on the generated crack information. The crack information generator 30 adds the generated scale image to the crack information. The transmission unit 24 transmits the crack information to which the scale image is added to the user terminal 10 .

The server device 20 of this example includes a crack evaluation unit 36 that evaluates the degree of cracking based on at least the crack information and determines the evaluation classification of the crack degree of the structure (hereinafter also referred to as "rank information"). The crack information generator 30 adds the determined rank information to the crack information. The transmitting unit 24 transmits the crack information to which the rank information is added to the user terminal 10 .

Furthermore, the server device 20 has a CAD data acquisition unit 38 (“data conversion unit”) that acquires CAD data by converting the composite image and the crack information into CAD (computer aided design) data in a predetermined data format. is one form).

Further, the server device 20 includes a form creation unit 40 that creates a form (hereinafter also referred to as an "inspection report") showing the inspection result of the structure including the determined evaluation category of the degree of cracking. The transmission unit 24 transmits the created form to the user terminal 10 .

The server device 20 also includes a storage unit 42 that stores a backend program and various information necessary for executing the backend program.

The receiving unit 22 and the transmitting unit 24 can be configured by communication devices for communicating with other devices including the user terminal 10 via the network NW. The synthetic image generation unit 26, the crack detection unit 28, the crack information generation unit 30, the machine learning unit 32, the scale image generation unit 34, the crack evaluation unit 36, the CAD data acquisition unit 38, and the form creation unit 40 are configured by, for example, a CPU. can do. A plurality of CPUs may be used. The storage unit 42 can be configured by a memory device.

Image position information (hereinafter also referred to as “image position information”) transmitted from the user terminal 10 to the server device 20 indicates the positional relationship of each image with respect to the surface of the structure. The position of each image area is indicated), and the relative positional relationship between a plurality of images is indicated.

<Image Correction>
The composite image generator 26 of the server device 20 synthesizes a plurality of images after correcting the images. The composite image generator 26 of this example can derive correction parameters based on a plurality of images and perform correction on the plurality of images. The composite image generator 26 of this example has a function of performing at least one of the following image corrections 1 to 4 on a plurality of images.

Image correction 1 (image position information correction): Image processing that matches the relative positional relationship between multiple images with the relative positional relationship between multiple imaged areas on the imaged surface (for example, the floor slab of a bridge). is. That is, when it is determined that inaccurate image position information has been received from the user terminal 10, the inaccurate image position information is corrected to more accurate image position information. For example, based on a crack vector obtained by vectorizing a crack in an image, if there is a crack that straddles adjacent images but the crack image is separated and not connected between the adjacent images, the crack is Make connections between images. This correction may be performed based on a non-cracking vector obtained by vectorizing the non-cracking of the structure (for example, the pattern of the structure) in the image. When imaging each coffer of the floor slab 2 of the bridge 1, the image position information may be corrected based on the edge of the coffer in the image. In addition to the function of correcting the image position information based on the latest inspection image, the synthetic image generation unit 26 of this example has the latest inspection image and past inspection images (composite image or pre-synthesis image). It has a function to correct the image position information based on the image). This makes it possible to easily correct inaccurate image position information to accurate image position information. If it is determined that accurate image position information has been received from the user terminal 10, this correction need not be performed.

Image correction 2 (scale correction): image processing (also referred to as “scaling”) for matching the scales of a plurality of images. That is, when it is determined that a plurality of images in which the distances from the imaging device 60 to the surface to be imaged of the structure do not match have been received from the user terminal 10, a plurality of images in which the distances to the surface to be imaged are more consistent (" (also called "image at a fixed distance"). If distance information is associated with the image, scaling may be performed based on the distance information. For example, based on a crack vector obtained by vectorizing cracks in an image, if there are multiple cracks across adjacent images but the intervals between the crack images do not match between the adjacent images, the cracks An example of image processing is to match the intervals between adjacent images. Correction may be performed based on a non-cracking vector obtained by vectorizing the non-cracking of the structure (for example, the pattern of the structure) in the image. When imaging each coffer of the floor slab 2 of the bridge 1, scale correction may be performed based on the edge of the coffer in the image. In addition to the function of performing scale correction based on the latest inspection image, the synthetic image generation unit 26 of this example has the latest inspection image and the past inspection image (composite image or image before synthesis). It has a function to perform scale correction based on When divided imaging is performed at a constant distance, this correction need not be performed.

Image correction 3 (tilt correction): Image processing for correcting the tilt of each of a plurality of images. That is, when it is determined that a plurality of images in which the tilt angles of the surface to be captured of the structure with respect to the imaging direction of the imaging device do not match have been received from the user terminal 10, a plurality of images in which the tilt angles of the surface to be captured are more consistent. (Also called “image with a constant tilt angle”). When tilt angle information is associated with an image, tilt correction may be performed based on the tilt angle information. For example, the directions of cracks are matched based on crack vectors obtained by vectorizing cracks in the image. Correction may be performed based on a non-cracking vector obtained by vectorizing the non-cracking of the structure (for example, the pattern of the structure) in the image. When imaging each coffer of the floor slab 2 of the bridge 1, tilt correction may be performed based on the edge of the coffer in the image. In addition to the function of performing tilt correction based on the latest inspection image, the synthetic image generation unit 26 of this example has the latest inspection image and the past inspection image (composite image or image before synthesis). It has a function of performing tilt correction based on This correction does not need to be performed when divided imaging is performed at a constant tilt angle.

Image correction 4 (rotational angle correction): Image processing for correcting the rotational angles of each of a plurality of images with respect to the surface to be captured. That is, when it is determined that a plurality of images having mismatched rotation angles (also referred to as "azimuth angles") around the imaging direction of the imaging device with respect to the imaging plane of the structure are received from the user terminal 10, A plurality of images with more consistent rotation angles (also referred to as “images with constant rotation angles”) are corrected. When rotation angle information is associated with an image, rotation may be performed based on the rotation angle information. For example, the directions of cracks are matched based on crack vectors obtained by vectorizing cracks in the image. Correction may be performed based on a non-cracking vector obtained by vectorizing the non-cracking of the structure (for example, the pattern of the structure) in the image. When imaging each coffer of the floor slab 2 of the bridge 1, the rotation angle correction may be performed based on the edge of the coffer in the image. In addition to the function of correcting the rotation angle based on the latest inspection image, the synthetic image generation unit 26 of this example has the function of performing the rotation angle correction based on the latest inspection image and the past inspection image (composite image or pre-synthesis image). ) to correct the rotation angle. When divided imaging is performed at a constant rotation angle, this correction need not be performed.

<Batch correction of crack information>
The crack information correction process performed by the crack information generation unit 30 of the server device 20 of the present example in accordance with the user's operation on the user terminal 10 will be described.

The crack information generation unit 30 of the server device 20 of the present example has a function of collectively correcting crack information for a plurality of cracks based on the feature amount of cracks specified by the user on the user terminal 10 (hereinafter referred to as "batch correction" ). The receiving unit 22 of the server device 20 receives, from the user terminal 10, a correction request indicating the content of correction of the crack information in terms of crack feature amounts. The crack information generator 30 of the server device 20 modifies the crack information based on the feature amount indicated by the correction request. There are various modes of correcting crack information (batch correction) based on such feature amounts.

As a first collective correction mode, there is a mode in which batch correction is performed according to manual operation (hereinafter referred to as “slide operation”) using a slider displayed on the screen of the user terminal 10 . The crack information generation unit 30 of this embodiment generates a frequency distribution (also referred to as a "histogram") indicating the detection frequency of line segments (hereinafter also referred to as "linear patterns") in the composite image with respect to the feature amount, and crack information with the feature amount. is added to the crack information transmitted to the user terminal 10, and the crack information is corrected according to the operation using the slider on the user terminal 10.

FIG. 3 is a diagram showing a first display example of the frequency distribution and the slider. In this example, crack images CR1, CR2, CR3, and CR4 representing the detected cracks are displayed on the composite image IMG1 displayed on the screen of the user terminal 10. FIG. Furthermore, in the vicinity of the synthesized image IMG1, a frequency distribution HG1 (histogram) indicating the frequency of detection of line segments (crack candidates) in the synthesized image IMG1 with respect to the feature quantity, and the content of correction of the crack information with the feature quantity are input. to display the slider SL1. The y direction (vertical direction of the screen) of the frequency distribution HG1 indicates the feature quantity of the line segment that is the crack candidate, and the x direction (horizontal direction of the screen) of the frequency distribution HG1 indicates the frequency of the line segment that is the crack candidate. The slider SL1 is a sliding GUI (graphical user interface) for adjusting criteria for determining whether a line segment in the composite image IMG1 is a crack. The slider SL1 of this example has one bar BR1, and the crack target area AR1 in the frequency distribution HG1 is determined based on the display position of the one bar BR1. In the vertical direction (x direction) of FIG. 3, the area above the display position of the bar BR1 in the frequency distribution HG1 is the crack target area AR1. detected as cracks.

FIG. 4 is a diagram showing a second display example of the frequency distribution and the slider. The slider SL2 of this example has two bars BR21 and BR22, and the crack target area AR2 is determined based on the display positions of the two bars BR21 and BR22. In the y direction of FIG. 4 (vertical direction of the screen), the area sandwiched between the two bars BR21 and BR22 in the frequency distribution HG1 is the crack target area AR2. Minutes are detected as "cracks".

FIG. 5 is a diagram showing a third display example of the frequency distribution and the slider. In this example, two sliders SL1 and SL2 are displayed, and crack target areas AR1 and AR2 can be selected and input based on a plurality of feature amounts.

FIG. 6 is a diagram showing a fourth display example of the frequency distribution and the slider. In this example, check boxes BX1, BX2, and BX3 are displayed, and the type of damage can be selected and input.

The feature quantity includes at least one of crack direction, length, width, edge strength, and edge density. Other feature quantities may be used. The crack information generator 30 adds a frequency distribution (histogram) and a slider to the crack information, and corrects the crack information according to manual operation using the slider on the user terminal 10 .

As described above, the frequency distribution indicating the detection frequency of the line segment with respect to the feature quantity (which is the feature quantity of the line segment detected by the crack detection unit 28), and the correction content of the crack information using the feature quantity. By adding the slider to the crack information and correcting the crack information according to the slide operation using the slider on the user terminal 10, it is possible to correct the crack information to higher accuracy with a simple operation. .

Secondly, there is a mode in which collective correction is performed in accordance with a group selection operation for cracked images displayed on the screen of the user terminal 10 . The crack information generation unit 30 of this aspect groups crack images based on the feature amount, and corrects the crack information in accordance with a manual operation of selecting a group of crack images on the user terminal 10 .

FIG. 7 shows an example in which crack images are grouped based on the direction (one form of feature quantity) and displayed with different line types for each group. In this example, a plurality of crack images (images corresponding to line segments detected as cracks by the crack detector 28) in the same direction and similar directions form one group. The crack images of the first group G1 are displayed with solid lines, the crack images of the second group G2 are displayed with dashed lines, and the crack images of the third group G3 are displayed with dashed lines. For convenience of illustration, the case where different line types are displayed for each group is exemplified, but different colors may be used for each group.

The user looks at the display on the screen of the user terminal 10 and performs a touch operation to enclose at least one cracked image in the group to be deleted. Then, based on the direction of the enclosed crack image, all crack images in the group to which the crack image belongs (which are crack images with the same orientation and similar orientation) are collectively erased. At the same time, the server device 20 deletes the crack information of the selected group. In the example shown in FIG. 7, the crack images of the same group included in the selection area AR surrounded by the touch operation are erased from the screen, and the crack information of the same group is deleted.

Instead of enclosing the cracked image, the user may directly select the group to be deleted. For example, when deleting the crack information of the second group G2 indicated by the dotted line in FIG. 7, the user terminal 10 receives an operation to select "dotted line". When the crack images are displayed in different colors for each group, an operation of selecting a color (for example, "red") is accepted. An operation of selecting group identification information (for example, group number) may be accepted.

In addition, it is preferable to accept operations of "addition", "increase" and "decrease" of crack information, not limited to "delete" of crack information.

As described above, the crack images are grouped based on the feature amount, and the crack information is corrected in accordance with the user operation of selecting the group of crack images on the user terminal 10, so that the crack can be formed with a simple operation and with higher accuracy. information can be corrected.

<Machine learning>
The machine learning unit 32 of the server device 20 is configured by artificial intelligence, for example. As artificial intelligence, for example, an artificial neural network may be used, or other known artificial intelligence may be used. However, the present invention can also be applied when artificial intelligence is not used as the machine learning unit 32 (for example, when information is acquired using a simple search means), and the machine learning unit 32 can be omitted. .

As the first learning, the machine learning unit 32 machine-learns detection of cracks based on the user's operation on the user terminal 10 . The machine learning unit 32 of this example includes at least a history of user operations for editing crack information on the user terminal 10, an image stored in the database 50 (a composite image or an image before composite), and a database The crack detection is machine-learned based on the crack information stored in 50 .

The machine learning unit 32 of this example learns from the editing history (a history indicating the content of correction of crack information) stored in the database 50 based on the operation history related to the crack information stored in the database 50. A feature amount of a crack to be detected from a synthesized image (or an image before synthesis) based on the specified editing history and the synthesized image and crack information corresponding to the specified editing history. machine learning. That is, the model used for crack detection by the crack detector 28 is updated.

Incidentally, the crack information generation unit 30 of this example has a function of "batch correction" of receiving correction instructions for crack information in units of feature amounts as described above. Therefore, the machine learning unit 32 of this example machine-learns crack detection based on the correspondence relationship between the history of user operations in units of feature amounts in the user terminal 10, the composite image, and the crack information. When an image before synthesis is analyzed to detect cracks, machine learning is performed based on the correspondence relationship between the operation history of each feature amount, the image before synthesis, and the crack information.

As second learning, the machine learning unit 32 machine-learns image correction based on user operations on the user terminal 10 . The machine learning unit 32 of this example is based on at least the history of user operations for image correction on the user terminal 10 and the images stored in the database 50 (images before correction and images after correction). machine learning for image correction.

The machine learning unit 32 of this example learns from the image correction history (a history indicating the content of image correction) stored in the database 50 based on the image correction operation history stored in the database 50. The image before correction is specified based on the specified image correction history and the image before correction and the image after correction (which may be a composite image) corresponding to the specified image correction history. and the content of image correction are machine-learned. That is, the model used for image correction by the composite image generator 26 is updated.

It should be noted that the composite image generation unit 26 of this example has the functions of image position information correction, scale correction, tilt correction, and rotation angle correction, as described above. Therefore, the machine learning unit 32 of this example includes a history of user operations for image position information correction, scale correction, tilt correction, or rotation angle correction on the user terminal 10, an image before correction and an image after correction (composite image machine learning for image correction based on the correspondence relationship with

Advances in artificial intelligence technology have been remarkable, and expectations are high for artificial intelligence that can perform image analysis that surpasses the brains of talented people. However, when an unlearned event occurs (for example, at the early stage of learning), corrective operation may be required. Therefore, even when artificial intelligence is adopted as the machine learning unit 32, a function of editing crack information with a simple user operation, such as the above-mentioned "batch correction", is implemented in the server device 20 together with the machine learning function. is preferred.

<First example of image processing>
FIG. 8 is a flow chart showing the flow of the first image processing example in the first embodiment.

The front-end processing on the left side of the figure is started in the user terminal 10 according to a program (front-end program) stored in the terminal storage unit 17 of the user terminal 10 (step S2). Further, the back-end processing on the right side of the drawing is started in the server device 20 according to the program (back-end program) stored in the storage unit 42 of the server device 20 (step S4).

In this example, as shown in FIG. 9, a plurality of images obtained by the imaging device 60 at the inspection site are input to the user terminal 10 in the office (step S6), and entered in a field notebook by the user at the inspection site. Image position information is input to the user terminal 10 in the office (step S8).

A plurality of images are obtained by capturing images of the coffers of the floor slab of the bridge divided into a plurality of images by the image capturing device 60 . These multiple images are input through the terminal input unit 11 using, for example, wired communication or short-range wireless communication. You may input from storage media, such as a memory card. A plurality of images may be uploaded from the imaging device 60 to a storage device (not shown) on the network NW, and the plurality of images may be input from the storage device via the network NW.

The field notebook contains information obtained by the user's visual inspection of the floor slab of the bridge. Information measured using a measuring instrument (not shown) may be entered in a field notebook. In the following description, it is assumed that at least handwritten information (handwritten image position information) indicating the positional relationship of a plurality of images with respect to the floor slab of a bridge is entered in a field notebook.

When a CAD (computer aided design) drawing showing the floor slab of the bridge has been created, the CAD drawing is displayed on the terminal display unit 14, and the coffers of the floor slab in the displayed CAD drawing are photographed. The terminal operation unit 15 accepts a manual operation for fitting each image to a position. According to such manual operation, image position information indicating the positional relationship of each image with respect to the reference position of the floor slab in the CAD drawing is input.

When a CAD drawing representing the floor slab of the bridge has not yet been created, the terminal operation unit 15 accepts a manual operation of arranging a plurality of images on the screen of the terminal display unit 14 as shown in FIG. Image position information indicating the relative positional relationship between a plurality of images is input according to such manual operation. In this aspect, it is preferable that the server device 20 converts the relative positional relationship between the plurality of images into the positional relationship of each image with respect to the reference position of the floor slab. A manual operation of superimposing common portions of adjacent images may be accepted.

Further, if necessary, information (hereinafter referred to as "correction hint information") that suggests subsequent image correction (step S16) is input to the user terminal 10 (step S10).

For example, if all of the multiple images are captured from the front of the floor slab, "all front", if all of the multiple images are captured from the non-front of the floor slab, "all tilt", and among the plurality of images The terminal operation unit 15 accepts a manual operation for inputting hint information for tilt correction, such as "partial tilt" when only a part of the floor slab is imaged from the non-front side. An input of a specific tilt angle (the tilt angle of the floor slab with respect to the imaging direction of the imaging device 60) may be accepted, but in this case, there is a possibility that an approximate angle is input, and the subsequent steps are performed. In the server device 20, it is preferable to correct the general angle to a detailed angle. Input of correction hint information for other image corrections such as scale correction (enlargement/reduction) and rotation angle correction may be accepted. If tilt correction, scale correction, and rotation angle correction are unnecessary, this step may be omitted.

Next, the terminal control unit 16 of the user terminal 10 generates image processing request information, and the terminal transmission unit 12 of the user terminal 10 transmits the generated image processing request information to the server device 20 (step S12). ). The image processing request information is received by the receiving unit 22 of the server device 20 and registered in the database 50 (step S14).

The image processing request information includes the plurality of images input in step S6 and the image position information input in step S8. When correction hint information is input in step S10, the correction hint information is also included in the image processing request information.

Next, the multiple images are corrected by the composite image generator 26 of the server device 20 (step S16). For example, among the aforementioned image corrections (image position information correction, scale correction, tilt correction, and rotation angle correction), necessary corrections are performed. When correction hint information is input in step S10, image correction is performed based on the correction hint information.

Next, one composite image is generated for each coffer of the floor slab based on the plurality of images and the image position information by the composite image generation unit 26 of the server device 20 (step S18). A composite image is registered in the database 50 .

Next, the machine learning unit 32 of the server device 20 determines whether or not the synthesized image is inappropriate (step S20). Based on various information stored in the database 50, the machine learning unit 32 learns events in which the synthesized image is appropriate and events in which the synthesized image is inappropriate. It can be judged whether it is appropriate or not. The machine learning unit 32 can have, as a learning result, a function of detecting an abnormality such as a pattern on a structure not being connected between original images (images before synthesis), for example. If it is determined that the synthesized image is inappropriate (YES in step S20), the parameters relating to image correction and image synthesis are changed and the process returns to step S16. case), the process proceeds to step S22.

Next, the crack detector 28 of the server device 20 performs image analysis on the composite image for each coffer of the floor slab to detect cracks in the floor slab for each coffer (step S22).

Next, the crack information generation unit 30 of the server device 20 generates crack information indicating the crack image corresponding to the detected crack and the feature amount of the detected crack (step S24). The feature quantity includes information necessary for discriminating the evaluation classification of the degree of cracking in the floor slab, such as crack width, length, and interval. In addition, the feature amount includes information necessary for batch editing. The generated crack information is registered in the database 50 .

In this example, the frequency distribution, slider, group information, etc. required for collective editing are added to the crack information. Also, the scale image generated by the scale image generation unit 34 may be added to the crack information. A specific example of the scale image will be described later.

Based on at least the crack information, the crack evaluation unit 36 evaluates the degree of cracks in the floor slab of the bridge, generates rank information indicating the evaluation classification of the degree of cracking, and adds the rank information to the crack information. good too. For convenience of explanation, it is assumed below that the rank information is not added to the crack information.

Next, the transmission unit 24 of the server device 20 transmits the composite image for each coffer of the floor slab and the crack information to the user terminal 10 (step S26). The composite image and crack information are received by the terminal receiver 13 of the user terminal 10 and displayed on the terminal display 14 (step S28).

For example, as shown in FIG. 11, the screen of the terminal display unit 14 displays a composite image IMG2 for each coffer of the floor slab and crack information including a crack image CR. In this example, the crack image CR is superimposed on the position where the crack is detected in the synthetic image IMG2. Although the scale image is omitted in FIG. 11, the scale image may be superimposed on the composite image IMG2 and displayed.

Next, the terminal operation unit 15 of the user terminal 10 determines whether or not an operation for correcting crack information has been received from the user (step S32). terminal transmission unit 12 transmits a request for correction of crack information corresponding to the user's correction operation to server device 20 (step S34). The correction request is received by the receiving unit 22 of the server device 20 and registered in the database 50 (step S36).

Also, the user can view the display on the terminal display unit 14 and input the rank information indicating the evaluation classification of the degree of cracking through the terminal operation unit 15 .

FIG. 12 is a diagram schematically showing an example of a crack progression model. In this progression model, rank information is expressed in five stages from "RANK1" to "RANK5". Correspondence between each rank information and the crack state is as follows.

RANK 1: No damage.

RANK 2: A state in which a plurality of cracks are generated in parallel along the lateral direction (transverse direction orthogonal to the vehicle traffic direction). This is the stage where multiple cracks due to drying shrinkage form parallel beams.

RANK 3: A crack in the longitudinal direction (longitudinal direction parallel to the direction of travel of the vehicle) and a crack in the lateral direction intersect each other. This is the stage where multiple cracks form a grid due to the live load and the crack density in the grid region increases. In the latter half of the period, cracks penetrate the floor slab in the vertical direction (perpendicular to the bottom surface of the floor slab).

RANK 4: A state in which the crack density in the lattice-like region exceeds a specified value, and the fracture surfaces of a plurality of penetrating cracks are smoothed. This is the stage where the floor slab loses shear resistance due to the polishing action.

RANK 5: A state in which omission has occurred. Wheel loads exceeding the reduced punch shear strength cause pull-out.

Next, the crack information generator 30 of the server device 20 corrects the crack information based on the received correction request (step S38). When the rank information is input by the user terminal 10, the rank information is added to the crack information.

Next, the modified crack information is transmitted to the user terminal 10 by the transmission unit 24 of the server device 20 (step S40). The corrected crack information is received by the terminal receiver 13 of the user terminal 10 and displayed on the terminal display 14 (step S42).

Next, the terminal operation unit 15 of the user terminal 10 determines whether input of a form creation instruction has been received from the user (step S44). The terminal transmission unit 12 of the user terminal 10 transmits the form request to the server device 20 (step S46). The form request is received by the receiving unit 22 of the server device 20 (step S48).

Next, the form creating unit 40 of the server device 20 creates a form (inspection report) indicating the inspection result of the floor slab of the bridge (step S50). The form includes rank information indicating the evaluation classification of the degree of cracking in the floor slab of the bridge.

Next, the form is transmitted to the user terminal 10 by the transmission unit 24 of the server device 20 (step S52). The form is received by the terminal receiving section 13 of the user terminal 10, and the terminal display section 14 notifies the user of receipt of the form (step S54). The form can be displayed on the terminal display unit 14 according to the user's operation, and can also be printed by a printer (not shown) connected to the network NW.

FIG. 13 is a diagram showing an example of a damage diagram as a form. This damage diagram shows damage indications (crack indications, water leakage indications, free lime indications, etc.), component names (“floor slabs,” etc.), Element number (Ds0201 etc.), type of damage ("crack", "leakage", "free lime" etc.), evaluation classification (rank information) of degree of damage are described. In FIG. 13, the five-level evaluation categories (“RANK1” to “RANK5”) shown in FIG. 12 are represented by alphabetical characters “a” to “e”.

<Second Image Processing Example>
FIG. 14 is a flow chart showing the flow of the second image processing example in the first embodiment.

Steps S2 to S16 in this example are the same as in the first image processing example shown in FIG.

In this example, each image before synthesis is analyzed to detect cracks (step S218), and a synthesized image is generated (step S220). Next, the machine learning unit 32 determines whether the synthesized image is inappropriate (step S222). Based on various information stored in the database 50, the machine learning unit 32 learns events in which the synthesized image is appropriate and events in which the synthesized image is inappropriate. It can be determined whether the image is inappropriate. The machine learning unit 32 can have, as a learning result, a function of detecting an abnormality such as a crack image not connecting between original images (images before synthesis), for example. If it is determined to be inappropriate (YES in step S222), the parameters related to image correction and image synthesis are changed and the process returns to step S16, and if it is determined to be appropriate (NO in step S222). , the process proceeds to step S24.

Steps S24 to S54 are the same as in the first image processing example shown in FIG.

<Third Image Processing Example>
FIG. 15 is a flow chart showing the flow of the third image processing example in the first embodiment.

In this example, as shown in FIG. 16, a plurality of two-viewpoint images obtained by split imaging by the imaging device 62 mounted on the robot device 70 at the inspection site are input by the user terminal 10 (step S306), and the robot Imaging device control information in accordance with imaging device control of the device 70 is input by the user terminal 10 (step S308). The imaging device control information of this example includes image position information, distance information, tilt angle information, and rotation angle information.

The robot device 70 of this example is a suspended robot suspended from a bridge, and FIG. 16 shows only the essential parts related to its imaging. The suspended robot may be a robot that suspends from a running body (vehicle) on a bridge instead of suspending from the bridge. The present invention can also be applied to a case where an unmanned flying robot such as a drone (unmanned flying object) is used instead of the suspended robot.

The imaging device 62 of this example is of a compound-eye type, and obtains a two-viewpoint image (also referred to as a “stereoscopic image”) by capturing images of the surface of the structure to be imaged from two viewpoints. The robot device 70 can calculate the distance from the imaging device 62 to each point in the plane to be imaged by performing image processing on the two-viewpoint images. The robot device 70 of this example also includes an imaging device control mechanism 72 that controls the rotation and movement of the imaging device 62 . The imaging device control mechanism 72 includes an XYZ moving mechanism capable of moving the imaging device 62 in each of the mutually orthogonal X, Y, and Z directions, and capable of rotating the imaging device 62 in each of the pan direction P and tilt direction T. It also serves as a pan-tilt mechanism. The imaging device control mechanism 72 can freely set the position of the imaging device 62 with respect to the reference point of the surface to be imaged (surface) of the structure. In other words, it is possible to freely set position information for each of a plurality of images obtained by a plurality of times of divided imaging. Further, the imaging device control mechanism 72 controls the distance between the imaging device 62 and the surface to be imaged of the structure, the inclination angle between the imaging direction of the imaging device 62 and the imaging surface of the structure, the imaging direction of the imaging device 62 and the structure. It is possible to freely set the rotation angle of the object with respect to the surface to be imaged. In other words, it is possible to freely set the distance information, the tilt angle information, and the rotation angle information of each of a plurality of images obtained by a plurality of times of divided imaging.

The terminal input unit 11 of the user terminal 10 transmits imaging device control information including image position information, distance information, tilt angle information, and rotation angle information together with a plurality of images via a storage medium such as a memory card or a network NW. , can be entered.

Steps S12 to S54 are the same as in the first image processing example shown in FIG.

If all information necessary for image composition or image correction, such as image position information, distance information, tilt angle information, and rotation angle information, is accurately obtained by measurement, an appropriate composite image is generated. Therefore, step S20 (determination of whether or not the synthesized image is inappropriate) can be omitted.

<Fourth example of image processing>
FIG. 17 is a flow chart showing the flow of the fourth example of image processing in the first embodiment.

Steps S2, S4, S306-S308, and S12-S16 in this example are the same as in the third image processing example shown in FIG. As in the third image processing example, a plurality of two-viewpoint images obtained by split imaging are input by the compound-eye imaging device 62 mounted on the robot device 70 (step S306), and the imaging of the robot device 70 is performed. Imaging device control information according to the imaging device control of the device control mechanism 72 is input (step S308).

Steps S218 to S222 are the same as in the second image processing example shown in FIG. That is, after detecting cracks by analyzing the pre-composite image in step S218, a composite image is generated and registered in step S220, and whether or not the composite image is inappropriate is determined in step S222. Step S222 can be omitted, like S20 in the third image processing example.

Steps S24 to S54 are the same as in the first image processing example.

An image processing example (a first image processing example and a second image processing example) in which a person operates an imaging device to capture an image of a structure and a robot device 70 controlling the imaging device to capture an image of a structure. For the sake of convenience, the image processing examples (the third image processing example and the fourth image processing example) in the case of imaging have been separately described, but the present invention is implemented in a closed configuration in either case. It is preferable to implement in an open configuration including both cases without being limited to the case. That is, the server device 20 of this example not only stores in the database 50 images when a person operates the imaging device, information (including image position information) necessary for synthesizing and correcting images, operation history, and the like. , images when the robot device 70 controls the imaging device, imaging device control information (including image position information), operation history, etc. are stored in the database 50, and machine learning is performed for both cases. As a result, even when a person operates the imaging device to capture an image of a structure, a composite image that is more suitable than a closed configuration can be obtained based on the machine learning results of image processing when the imaging device is controlled by the robot device 70. can be generated.

[Second embodiment]
FIG. 18 is a block diagram showing a configuration example of an image processing system according to the second embodiment. The user terminal 10 in this figure is the same as in the first embodiment. A server device 200 of this figure has a configuration in which a prediction unit 44 for predicting progress of cracks is added to the server device 20 of the first embodiment shown in FIG.

The prediction unit 44 predicts progress of cracks in the bridge based on the various information about the bridge stored in the database 50 and the crack information generated by the crack information generation unit 30 . The transmission unit 24 transmits the predicted progress of cracking to the user terminal 10 .

The database 50 of this example stores bridge structural information, bridge environmental condition information, bridge usage condition information, bridge maintenance history information, and bridge disaster history information. The prediction unit 44 of this example includes at least one of bridge structure information, bridge environmental condition information, bridge usage condition information, bridge maintenance history information, and bridge disaster history information, and bridge crack information. Predict the progression of cracks in bridges based on

The rank information is, for example, an evaluation division for each inspection part of a bridge (for example, a floor slab coffer). The rank information may include a "soundness" representing the soundness of the bridge in stages or numerically.

The structural information of a bridge indicates the structure of at least a portion of the entire bridge including the floor slab and surrounding members that are subject to evaluation. Examples include structural form and structural features. Including structural information related to the progression of floor slab cracking, such as the relationship between load and floor slab, the relationship between earthquake vibration and floor slab, the relationship between rainwater and floor slab, the relationship between temperature and floor slab, etc. . Chemical information such as whether or not a material that easily causes an alkali-aggregate reaction is used may be used. Information that directly represents a portion where cracks are likely to occur may be used.

The environmental condition information of the bridge includes, for example, location conditions such as whether or not salinity will fly.

The bridge use condition information includes, for example, the traffic volume of vehicles and the ratio of large vehicles.

Bridge maintenance management history information includes, for example, antifreeze spraying history information. Repair and retrofit history information may also be included.

The bridge disaster history information is history information of disasters such as typhoons and earthquakes that the bridge has suffered.

The database 50 of the present example further stores rank information indicating evaluation categories of the degree of cracking of the bridge in association with the composite image of the bridge and the crack information. The prediction unit 44 of this example executes simulation processing for predicting how the rank information of the bridge evaluated by the crack evaluation unit 36 will change in the future.

Also, the prediction unit 44 may simulate how things have changed from the past to the present. The crack information generation unit 30 generates crack information based on the simulation results from the past to the present and the current image (composite image or image before composition), thereby further improving the accuracy of the crack information. can.

The machine learning unit 32 of this example can machine-learn crack detection based on the prediction result of the prediction unit 44 in addition to the operation history, image (composite image or image before synthesis), and crack information. is. The machine learning unit 32 further determines cracks based on at least one of bridge structural information, bridge environmental condition information, bridge usage condition information, bridge maintenance management history information, and bridge disaster history information. Learning to detect is preferred.

[Vectorization]
Vectorization of a crack image will be described with reference to FIGS. 19 to 22. FIG.

The crack information generation unit 30 of the server device 20 vectorizes the crack portion extracted from the synthesized image or the image before synthesis to generate crack vector data (hereinafter referred to as "crack vector" or simply "vector"). That is, the crack information generator 30 vectorizes the crack portion. The transmission unit 24 of the server device 20 transmits crack information including a crack vector, which is a vectorized crack image, to the user terminal 10 together with the composite image. The vectorized crack image is superimposed on the composite image and displayed on the terminal display unit 14 of the user terminal 10 .

The crack information generation unit 30 binarizes and/or thins the detected cracks as necessary during vectorization. "Vectorization" is to find a line segment determined by the start and end points of a crack, and if the crack is curved, divide the crack into multiple sections so that the distance between the curve and the line segment is less than the threshold. and generate a crack vector for each of a plurality of sections.

When generating the crack vectors, for example, the characteristic point of the floor slab 2 is set as the origin of the coordinate system, and for a group of crack vectors (vector group), the end point with the minimum distance from the origin is set as the first starting point. The end point and the start point can be determined sequentially along the running direction of . In the example of FIG. 19, when the point P0 on the floor slab 2 is the origin of the coordinate system, and the rightward and downward directions of the drawing are the X-axis direction and the Y-axis direction of the coordinate system, points P13, P13, and P13 of the vector group C7 The point P13 at which the distance d from the point P0 is the shortest among P14, P15, and P16 is defined as the starting point of the crack vector C7-1, and hereinafter, the point P14 is defined as the end point of the crack vector C7-1 (and the crack vectors C7-2, C7 -3), and points P15 and P16 can be the end points of crack vectors C7-2 and C7-3, respectively.

However, when the starting point of the vector group C8 is determined by a similar method, the point P17 becomes the starting point of the crack vector C8-1, the point P18 becomes the starting point of the crack vectors C8-2 and C8-3, and the running direction of the crack vector C8-3. (Direction from point P18 to point P20) runs counter to the running direction of crack vector C8-1. Therefore, in such a case, as shown in FIG. 20, the point P19 is the starting point of the crack vector C8A-1, and the point P18 is the end point of the crack vector C8A-1 (and the starting point of the crack vectors C8A-2 and C8A-3). ), and the points P17 and P20 may be the end points of the crack vectors C8A-2 and C8A-3, respectively. A set of crack vectors in this case is denoted as vector group C8A.

When generating crack vectors as described above, if the cracks are continuous inside the floor slab 2 but separated on the surface, they may be recognized as separate crack vectors. Therefore, one or more vectors are generated by concatenating a plurality of crack vectors.

FIG. 21 is a diagram showing an example of connection of crack vectors. A vector group C3 including a crack vector C3-1 (points P21 and P22 are the start and end points, respectively), and a crack vector C4-1 (points P23 and P24). indicates a situation in which a vector group C4 containing a starting point and an end point, respectively, is extracted. The angle formed by the crack vector C3-1 with the line segment connecting the points P22 and P23 is α1, and the angle formed by the line segment connecting the points P22 and P23 with the crack vector C4-1 is α2. At this time, if both the angles α1 and α2 are less than the threshold value, the crack vectors C3-1 and C4-1 are connected and the vector groups C3 and C4 are merged. Specifically, as shown in FIG. 22, a new crack vector C5-2 is generated and other crack vectors C5-1 (same as crack vector C3-1) and C5-3 (same as crack vector C4-1) ), and a new vector group including these crack vectors C5-1, C5-2, and C5-3 is defined as vector group C5. In this way, by appropriately connecting the vectors that are spatially separated (on the surface of the floor slab 2), it is possible to accurately grasp the connection relationship between the vectors.

Note that the vector concatenation described above is an example, and other methods may be used.

Further, the crack information generator 30 calculates relative angles between the vectors. That is, the angle formed by one vector and another vector is calculated.

In addition, the crack information generation unit 30 generates hierarchical structure information indicating the hierarchical structure of crack vectors. Hierarchical structure information is information that hierarchically expresses the connection relationship between vectors. The hierarchical structure information includes hierarchical identification information (also referred to as “hierarchy belonging information”) indicating to which hierarchy each vector belongs in a vector group composed of a plurality of vectors. For the layer identification information, for example, a layer number represented by numbers can be used. Codes other than numbers (for example, letters and symbols) may be used.

FIG. 23 is a diagram showing vector group C1. The vector group C1 is composed of crack vectors C1-1 to C1-6, which start or end at points P1 to P7. In such a situation, each time the crack vector branches (the end point of a certain crack vector is the start point of a plurality of other crack vectors), the hierarchy is said to be lower. Specifically, the hierarchy of the crack vector C1-1 is the highest "level 1", and the hierarchy of the crack vectors C1-2 and C1-3 starting from the point P2, which is the end point of the crack vector C1-1, is the crack Let it be "level 2" which is lower than the vector C1-1. Similarly, the hierarchy of crack vectors C1-5 and C1-6 starting from point P4, which is the end point of crack vector C1-3, is "level 3", which is lower than crack vector C1-3. On the other hand, the point P3, which is the end point of the crack vector C1-2, is the starting point of the crack vector C1-4. The hierarchy of -4 is the same "level 2" as C1-2. The hierarchy of each crack vector determined in this way is included in the hierarchical structure information.

FIG. 24 is a diagram showing vector group C1 (the connection relationship between crack vectors is the same as that shown in FIG. 23). In this example, among the crack vectors to be connected, those whose angles with other crack vectors are equal to or less than a threshold value (crack vectors corresponding to the "trunk" in the tree structure) belong to the same hierarchy. Specifically, the crack vectors C1-1, C1-2, and C1-4 existing within the dotted line in FIG. . As for the other crack vectors C1-3, C1-5, and C1-6, the hierarchy becomes lower each time the crack vector branches. equivalent) is "level 2", and crack vectors C1-5 and C1-6 (corresponding to "leaves" in the tree structure) are "level 3". The hierarchy and type (trunk, branch, or leaf) of each crack vector thus determined are included in the hierarchical structure information.

Note that the hierarchical structuring described with reference to FIGS. 23 and 24 is an example, and the hierarchical structuring may be performed by a modified method or a different method.

Further, the crack information generator 30 adds various types of additional information (attribute information) to the vector. For example, information indicating the width of each crack (width information) is added to each vector.

As described above, since the crack image is vectorized and hierarchically structured by the crack information generating unit 30 of the server device 20, various processes are efficiently and appropriately performed using the generated crack vector and hierarchical structure information. becomes possible.

For example, image correction can be performed using crack vectors and hierarchical structure information. Also, it becomes possible to determine whether or not the synthesized image is inappropriate using the crack vector and the hierarchical structure information. In addition, machine learning can be performed using crack vectors and hierarchical structure information. In addition, it becomes possible to discriminate the evaluation classification of the degree of cracking using the crack vector and the hierarchical structure information. In addition, it becomes possible to predict the progression of cracks using crack vectors and hierarchical structure information.

[Scale image example]
An example of a scale image generated by the scale image generator 34 will be described. The scale image generation unit 34 generates various scale images for measuring the crack feature amount (for example, crack width) indicated by the crack image on the screen of the user terminal 10 . A scale image is added to the crack information.

FIG. 25 is a diagram showing a display example of a scale image. For convenience of illustration and explanation, crack images CR11 (CR11-1, CR11-2, CR11-3) showing cracks to be measured are shown in enlarged form.

The scale images CS1, CS2, CS3, and CS4 are images for measuring cracks occurring in the measurement surface of the structure (the floor slab of the bridge in this example). These scale images CS1 to CS4 are generated by the scale image generator 34 of the server device 20. FIG. The scale image generator 34 of this example includes scale images CS1 and CS2 showing scales with graduations for measuring crack lengths, and scales with line drawings and numbers for measuring crack widths. to generate scale images CS3 and CS4. The scale images CS1 to CS4 show scales corresponding to the distance between the imaging device and the measurement plane and the tilt angle of the measurement plane with respect to the imaging direction of the imaging device. The scale image generator 34 varies the type, shape, size, etc. of the scale according to the distance and angle. The scale image generation unit 34 of this example varies the scales of the scale images CS1 and CS2 for length measurement, and varies the line drawings and numbers of the scale images CS3 and CS4 for width measurement, according to the distance and the tilt angle. ing.

The crack information generator 30 of the server device 20 adds the scale images CS1 to CS4 to the crack information. The transmission unit 24 of the server device 20 transmits the crack information to which the scale images CS1 to CS4 are added to the user terminal 10 together with the composite image of the structure. The transmitted crack information with the scale image and the synthesized image are received by the terminal reception unit 13 of the user terminal 10 .

The display area DA in the terminal display unit 14 of the user terminal 10 includes an image display area IA that displays a composite image and a scale image of a structure, and a scale image of a type that can be displayed in the image display area IA (that is, a user selectable and a scale image selection area SA for displaying a scale image). In the scale image selection area SA, in addition to the scale images CS1 to CS4, an OK button BU1 for confirming the selection of the scale image, a cancel button BU2 for canceling the selection of the scale image, and an image display area IA. An erase button BU3 is displayed for erasing the scale image.

The terminal operation unit 15 receives a user's selection operation for the scale images CS1 to CS4 displayed in the scale image selection area SA. A user performs a selection operation using a button, a mouse, or the like. Scale images (CS2 and CS3 in this example) corresponding to the user's selection operation are displayed in the image display area IA from among the plurality of scale images CS1 to CS4 displayed in the scale image selection area SA.

The terminal display unit 14 superimposes and displays the composite image and the scale image in the image display area IA. The position and orientation of the scale image are determined according to the position and orientation of the crack image in the composite image. The crack image CR11 in this example has a hierarchical structure and includes a parent crack image CR11-1 and child crack images CR11-2 and CR11-3. The scale image CS2 for length measurement is displayed along the direction of the crack image CR11-2 and at a position near the crack image CR11-2. The scale image CS3 for width measurement is a line drawing along a direction perpendicular to the direction of the crack image CR11-1 and having a width closest to the width of the crack image CR11-1 (a line drawing of 1.4 mm in this example). It is displayed aligned with the crack image CR11-1.

Although not shown, a move button for moving the scaled image, a rotate button for rotating the scaled image, an enlarge button for enlarging the scaled image, a reduce button for reducing the scaled image, etc. are also used for scale image selection. It is preferable to display it in the area SA. Movement, rotation, enlargement, reduction, etc. of the scale image may be instructed by moving the mouse.

Although two scale images CS2 and CS3 are displayed in the image display area IA, the number of scale images displayed in the image display area IA is not limited.

Although the embodiments for carrying out the present invention have been described above, the present invention is not limited to the above-described embodiments and modifications, and various modifications are possible without departing from the gist of the present invention.

1 Bridge 2 Floor slab 3 Main girder 3A Joint 10 User terminal 11 Terminal input unit 12 Terminal transmission unit 13 Terminal reception unit 14 Terminal display unit 15 Terminal operation unit 16 Terminal control unit 17 Terminal storage unit 20, 200 Server device 22 Reception unit 24 transmission unit 26 composite image generation unit 28 crack detection unit 30 crack information generation unit 32 machine learning unit 34 scale image generation unit 36 crack evaluation unit 38 CAD data acquisition unit (data conversion unit)
40 Form creation unit 42 Storage unit 44 Prediction unit 50 Databases 60, 62 Imaging device 70 Robot device 72 Imaging device control mechanism AR Selection areas AR1, AR2 Crack target areas BR1, BR21, BR22 Bar BU1 OK button BU2 Cancel button BU3 Erase button BX1 , BX2, BX3 Check boxes C1, C3, C4, C5, C7, C8, C8A Vector groups C1-1, C1-2, C1-3, C1-4, C1-5, C3-1, C4-1, C5 -1, C5-2, C7-1, C7-2, C8-1, C8-2, C8-3, C8A-1, C8A-2 Vector CR, CR1, CR2, CR3, CR4, CR11, CR11-1 , CR11-2, CR11-3 Crack images CS1, CS2, CS3, CS4 Scaled image DA Display area G1 First group G2 Second group G3 Third group HG1, HG2 Frequency distribution IA Image display areas IMG1, IMG2 Synthesis Image NW Network SA Scale image selection area SL1, SL2 Slider d Distance α1, α2 Angle

Claims (12)

An image processing device comprising a processor,
The processor
Acquiring multiple images showing the surface of the structure,
synthesizing the plurality of acquired images to generate a synthetic image showing the surface of the structure;
Acquiring a judgment result as to whether or not the synthesized image is inappropriate;
if the acquired determination result indicates that the synthesized image is inappropriate, acquiring modification parameters for image correction and/or image synthesis, and recreating the synthesized image based on the modification parameters; generate and
An image processing device that analyzes the synthesized image or the image before synthesis to detect cracks on the surface of the structure. The modification parameters include a parameter indicating relative positional relationship between the plurality of images, a parameter indicating enlargement or reduction of the plurality of images, a parameter indicating tilt of each of the plurality of images, and a parameter indicating the tilt of each of the plurality of images. 2. The image processing apparatus according to claim 1, comprising at least one parameter indicating a rotation angle with respect to each surface to be imaged. 3. The image processing apparatus according to claim 1, wherein the processor acquires correction hint information that is information suggestive of the image correction, and acquires the correction parameter based on the correction hint information. 4. The image processing apparatus according to any one of claims 1 to 3, wherein the processor obtains the correction parameters based on information indicating cracks in the image and/or information indicating non-cracking of the structure. 5. The processor according to any one of claims 1 to 4, wherein the processor obtains the modified parameters based on the plurality of images acquired during a most recent inspection and/or the plurality of images acquired during a previous inspection. Image processing device. The image processing apparatus according to any one of claims 1 to 5, wherein the processor acquires the judgment result obtained by a machine learning technique. 7. The image processing apparatus according to any one of claims 1 to 6, wherein said processor generates crack information indicating a crack image corresponding to said detected crack and a feature amount of said detected crack. 8. The image processing apparatus according to claim 7, wherein the processor groups the cracked images based on the feature amount, and modifies the cracked information according to an operation of selecting the group of the cracked images. 9. The image processing apparatus according to claim 7, wherein the processor displays the crack images using different line types or colors for each group. An image processing system comprising the image processing device according to any one of claims 1 to 9 and a user terminal,
The image processing system, wherein the processor acquires the plurality of images from the user terminal. A method of operating an image processing device comprising a processor, comprising:
The processor
Acquiring multiple images showing the surface of the structure,
synthesizing the plurality of acquired images to generate a synthetic image showing the surface of the structure;
Acquiring a judgment result as to whether or not the synthesized image is inappropriate;
if the acquired determination result indicates that the synthesized image is inappropriate, acquiring modification parameters for image correction and/or image synthesis, and recreating the synthesized image based on the modification parameters; generate and
A method of operation comprising image analysis of said composite image or said image prior to composite to detect cracks in the surface of said structure. An image processing program that causes an image processing apparatus to execute the operating method according to claim 11 .

Images