JP6626970B2 - Server device, image processing system, and image processing method - Google Patents

Description

  The present invention provides a server device that analyzes an image of a crack without requiring an expensive device, and that can present a composite image of a crack and an image analysis result without complicating a user's operation, The present invention relates to an image processing system and an image processing method.

  Structures such as bridges and tunnels exist as social infrastructure. These structures are subject to damage, and the damage tends to progress, so that periodic inspections are required. In addition, it is required to report accurate inspection results. For example, it is required to accurately understand the nature of cracks that have occurred on the surface of structures and report inspection results, and it is necessary to perform appropriate maintenance based on the progress of cracks based on the reports. is there.

  Patent Document 1 discloses that when a distant view image and a close-up image with a laser beam spot obtained by imaging a concrete wall surface with a camera equipped with a laser projectile are transmitted from a mobile personal computer, the distant view image and the close-up image are viewed from the front of the concrete wall surface. Normalizing to the viewed image, and further performing an operation of assigning which area of the distant view image the proximity image belongs to in order to specify the proximity image of the image analysis target and transmitting an analysis instruction, the specified proximity image is transmitted. A server device that measures the width of a crack by image analysis is described.

  Patent Document 2 discloses that an operation of pasting an image of a concrete structure on a CAD (computer aided design) drawing based on information on a photographing position written in a field note at the time of work at an inspection site is performed using a personal computer. When an operation is performed to associate an image with a CAD drawing according to the operation and trace the crack on the image, the crack on the image is mapped to the CAD drawing according to the operation, and the crack is generated using the information of the CAD drawing. To calculate the width and length of the.

  Patent Document 3 discloses a personal computer in which the distance from a camera to a concrete wall surface and the angle of inclination of the concrete wall surface with respect to the optical axis of the camera are measured, and a plurality of images are joined and viewed from the front of the concrete wall surface. It is described that the distribution state of the cracks on the concrete wall surface and the width of the cracks are obtained and a state diagram is displayed. Cracks that are not recognized even though they are actually cracks can be input by a trace operation using a mouse or a pen input device.

  Patent Document 4 discloses that when performing machine learning for crack identification processing using a support vector machine algorithm, the center of each area image among a plurality of area images and the identification target (crack, dirt, etc.) in each area image are disclosed. It is described that only a region image whose center of gravity coincides with the center of gravity of the learning target sample is selected as a learning target sample.

JP-A-2004-69434 JP 2005-310044 A JP-A-10-78305 JP-A-2003-216953

  Patent Documents 1 to 4 do not disclose transmitting position information of each image together with a plurality of images obtained by dividing and imaging a structure from a user terminal and synthesizing them into one image by a server device. .

  Further, the system including a mobile personal computer (which is an embodiment of a user terminal) and a server device described in Patent Literature 1 requires not only a camera provided with a laser projectile but also a distant view image and a close image. It is necessary, and further, it is necessary to specify a close image to be analyzed by a user operation of assigning a close image to a distant view image. Therefore, it can be said that not only an expensive device is required, but also the operation of the user until obtaining the image analysis result of the crack is complicated.

  In addition, when a composite image of a crack location is generated and a crack image is analyzed on a user terminal, expensive hardware or expensive software is required. That is, an expensive device was required.

  Patent Document 2 discloses that when a personal computer (which is an embodiment of a user terminal) performs an operation of pasting an image on a CAD drawing and an operation of tracing a crack on the image, the width and length of the crack are reduced. It only states that it is calculated.

  In the technique described in Patent Document 3, when crack information is corrected, an operation of tracing one by one using a mouse or a pen input device is required. If the function of correcting crack information by such a trace operation is used, cracks of any pattern can be individually corrected one by one, but the operation is complicated and requires a long working time, so that it is busy. In some cases, it may not be possible for workers.

  Patent Literature 4 merely describes that a process of identifying a crack is learned based on an image of a structure.

  As described above, various devices have been proposed or provided to assist in accurately grasping the properties of cracks generated on the surface of the structure and reporting the inspection results, but by analyzing images of the cracked portions. In order to present the result of the image analysis of the crack to the user, there is a problem that an expensive device is required, and a user's load for synthesizing a plurality of images of the cracked portion is large.

  The present invention provides a server device that analyzes an image of a crack without requiring an expensive device, and that can present a composite image of a crack and an image analysis result without complicating a user's operation, It is an object to provide an image processing system and an image processing method.

  In order to achieve the above-described object, a server device according to a first aspect of the present invention includes a receiving unit that receives, from a user terminal, a plurality of images indicating a surface of a structure, and position information of the plurality of images. A composite image generation unit that generates a composite image showing the surface of the structure based on the received multiple images and position information, and detects cracks on the surface of the structure by analyzing the composite image or the image before the composition. And a crack information generating unit that generates crack information indicating a crack image corresponding to the detected crack and a characteristic amount of the detected crack.

  According to this aspect, when a plurality of images indicating the surface of the structure and the position information of the plurality of images are transmitted from the user terminal, a composite image is generated by the server device based on the plurality of images and the position information of the images. In addition, since the crack information is generated, it is possible to analyze the image of the crack location without requiring an expensive device, and to present the composite image and the image analysis result of the crack location without complicating the operation of the user. Will be possible. Further, according to this aspect, the image transmitted from the user terminal and its position information, and further, the synthesized image and the crack information generated by the server device can be accumulated and converted into big data. It can be said that the secondary use of big data makes it possible to derive an appropriate composite image and an appropriate image analysis result.

  The server device according to the second aspect of the present invention includes a transmission unit that transmits the composite image and the crack information to the user terminal and displays the composite image and the crack information on the screen of the user terminal. A request for correcting crack information according to the manual operation at the terminal is received from the user terminal, the crack information generating unit corrects the crack information based on the correction request, and the transmitting unit transmits the corrected crack information to the user terminal. Send. According to this aspect, the composite image and the crack information are displayed on the screen of the user terminal, and when the user performs a manual operation on the content of the display, a correction request corresponding to the manual operation is stored in the server. Becomes possible. Therefore, it is easy to improve the image analysis processing in the server device so that a more appropriate image analysis result can be derived. That is, it is possible to present a more appropriate image analysis result to the user while reducing complicated user operations for correcting the image processing result.

  In the server device according to the third aspect of the present invention, the receiving unit receives, from the user terminal, a correction request indicating the content of the correction of the crack information in the form of a feature, and the crack information generating unit outputs the characteristic indicated by the correction request. Modify the crack information based on the amount. According to this aspect, it is possible to perform a user operation indicating a request for correcting crack information with the feature amount of the crack. For example, it is possible to correct the crack information by a simple operation as compared with a case where the crack information is corrected one by one based on the trace operation.

  In the server device according to the fourth aspect of the present invention, the frequency distribution indicating the detection frequency of the line segment in the composite image with respect to the feature amount, and the slider for instructing and inputting the details of the correction of the crack information by the feature amount, In addition to the information, the crack information is corrected in accordance with a user operation using a slider on the user terminal. According to this aspect, it is possible to correct the crack information by a simple operation of operating the slider while watching the detection frequency of the line segment with respect to the feature amount.

  In the server device according to the fifth aspect of the present invention, the crack information generation unit groups the crack images based on the feature amount, and generates the crack information according to a manual operation of selecting a group of the crack images on the user terminal. Fix it. According to this aspect, it is possible to correct the crack information by a simple operation of selecting a group of crack images.

  The server device according to the sixth aspect of the present invention includes a machine learning unit that machine-learns crack detection based on a manual operation at a user terminal. According to this aspect, since the detection of the crack is machine-learned based on the manual operation on the user terminal, it is possible to steadily reduce the complicated user operation for correcting the image analysis result.

  In the server device according to the seventh aspect of the present invention, the feature quantity includes at least one of a crack direction, a length, a width, an edge strength, and an edge density.

  In the image processing method according to the eighth aspect of the present invention, the composite image generation unit corrects at least one of image position information, scale, tilt, and rotation angle for a plurality of images.

  In the server device according to the ninth aspect of the present invention, the composite image generation unit derives a correction parameter based on a plurality of images and performs correction.

  The server device according to the tenth aspect of the present invention includes a scale image generation unit that generates a scale image for measuring the width of a crack indicated by the crack image on a screen of the user terminal based on the crack information, The crack information generation unit adds the scale image to the crack information.

  The server device according to the eleventh aspect of the present invention includes a crack evaluation unit that evaluates the degree of crack based on at least the crack information and determines an evaluation category of the degree of crack of the structure.

  The server device according to the twelfth aspect of the present invention includes a form creation unit that creates a form showing the inspection result of the structure including the determined evaluation category.

  The server device according to the thirteenth aspect of the present invention includes a data conversion unit that converts the composite image and the crack information into CAD data in a predetermined data format.

  In the server device according to the fourteenth aspect of the present invention, based on at least one of structural information of a structure, environmental condition information, use condition information, maintenance management history information, and disaster history information, and the crack information. And a prediction unit for predicting the progress of cracks.

  An image processing system according to a fifteenth aspect of the present invention is an image processing system including a user terminal and a server device, wherein the user terminal includes a plurality of images indicating a surface of a structure, and position information of the plurality of images. A terminal input unit that inputs a plurality of images and position information to the server device, a terminal receiving unit that receives image processing results for the plurality of images from the server device, and an image processing result. A server display unit that receives a plurality of images and position information from a user terminal, and a server device based on the received plurality of images and position information. A composite image generation unit that generates a composite image showing the surface, a crack detection unit that detects a crack on the surface of the structure by performing image analysis on the composite image or an image before synthesis, and a crack detection unit that responds to the detected crack. A crack information generation unit that generates crack information indicating a crack image and a feature amount of a detected crack, and a server transmission unit that transmits a composite image and crack information to a user terminal as an image processing result for a plurality of images, And the terminal display unit displays the composite image and the crack information.

  In the image processing method according to a sixteenth aspect of the present invention, a step of receiving, from a user terminal, a plurality of images showing a surface of a structure and position information of the plurality of images, Generating a composite image showing the surface of the structure based on the position information; analyzing the composite image or an image before synthesis to detect cracks on the surface of the structure; and responding to the detected cracks Generating crack information indicating the cracked image to be cracked and the detected feature amount of the crack.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to analyze the image of a cracked part without requiring an expensive apparatus, and to present the composite image and the image analysis result of the cracked part without complicating the operation of the user.

FIG. 1 is a perspective view showing a bridge which is an example of a structure to which the present invention is applied. FIG. 2 is a block diagram illustrating a configuration example of the image processing system according to the first embodiment. FIG. 3 is a diagram showing a frequency distribution and a first display example of a slider. FIG. 4 is a diagram showing a frequency distribution and a second display example of the slider. FIG. 5 is a diagram showing a third distribution example of the frequency distribution and the slider. FIG. 6 is a diagram showing a frequency distribution and a fourth display example of the slider. FIG. 7 is a diagram illustrating a display example in the case where cracked images are grouped based on feature amounts. FIG. 8 is a flowchart showing the flow of the first example of image processing in the first embodiment. FIG. 9 is an explanatory diagram used for describing a first example of image processing in the first embodiment. FIG. 10 is an explanatory diagram used for describing an operation of arranging a plurality of images. FIG. 11 is a diagram illustrating a display example of a composite image and a crack image. FIG. 12 is a diagram schematically illustrating an example of a crack progressing model. FIG. 13 is a diagram illustrating an example of a damage diagram as a form. FIG. 14 is a flowchart illustrating the flow of the second image processing example according to the first embodiment. FIG. 15 is a flowchart illustrating the flow of the third image processing example according to the first embodiment. FIG. 16 is an explanatory diagram used for describing a third image processing example in the first embodiment. FIG. 17 is a flowchart illustrating the flow of the fourth image processing example according to the first embodiment. FIG. 18 is a block diagram illustrating a configuration example of an image processing system according to the second embodiment. It is a 1st explanatory view used for description of vectorization of a crack image. It is a 2nd explanatory view used for explanation of vectorization of a crack image. It is the 3rd explanatory view used for description of vectorization of a crack image. It is the 4th explanatory view used for explanation of vectorization of a crack image. It is the 5th explanatory view used for description of vectorization of a crack image. It is a 6th explanatory view used for explanation of vectorization of a crack image. It is a figure showing the example of a display of a scale image.

  Hereinafter, embodiments for implementing a server device, an image processing system, and an image processing method according to the present invention will be described with reference to the accompanying drawings.

[Example of structure]
FIG. 1 is a perspective view showing a structure of a bridge which is an example of a structure to which the present invention is applied. The bridge 1 shown in FIG. 1 has a plurality of main girders 3, and the main girders 3 are joined at a joint 3A. On the upper part of the main girder 3, a floor slab 2 which is a member made of concrete is cast. The bridge 1 has, in addition to the floor slab 2 and the main girder 3, members (not shown) such as a horizontal girder, an inclined structure, and a horizontal structure.

  The “structure” in the present invention is not limited to a bridge. The structure may be a tunnel, a dam, a building or the like.

[First Embodiment]
FIG. 2 is a block diagram illustrating a configuration example of the image processing system according to the first embodiment.

  The user terminal 10 and the server device 20 constitute a client-server type image processing system. Actually, a plurality of user terminals 10 are connected to the network NW. The user terminal 10 is configured by a computer device that performs front-end processing for the user, and the server device 20 is configured by a computer device that performs back-end processing such as image processing. As the user terminal 10, for example, a computer device such as a personal computer, a tablet, and a smartphone can be used. The server device 20 may be composed of a plurality of computer devices. The server device 20 of this example is configured to include a database 50 for convenience of explanation. The server device 20 and the database 50 may be provided as different devices.

  The user terminal 10 includes a terminal input unit 11 for inputting a plurality of images showing the surface of the structure and information necessary for image synthesis including positional information of the plurality of images. A terminal transmitting unit 12 for transmitting necessary information to the server device 20, a terminal receiving unit 13 for receiving image processing results for a plurality of images from the server device 20, and a terminal for displaying the image processing results of the server device 20 A display unit 14, a terminal operation unit 15 for receiving a user's manual operation, a terminal control unit 16 for controlling each unit of the user terminal 10 according to a front end program for the user terminal 10, a front end program and execution of the front end program And a terminal storage unit 17 for storing various information required for the terminal.

  The terminal input unit 11 can be configured by, for example, a communication device that inputs an image and position information of the image by wireless communication or wired communication. The terminal input unit 11 may include a storage medium interface device that inputs an image from a storage medium such as a memory card, and an operation device that receives an input operation of position information of an image from a person. The terminal transmission unit 12 and the terminal reception unit 13 can be configured by a communication device for communicating with another device including the server device 20 via the network NW. The terminal display unit 14 can be configured by a display device such as a liquid crystal display device. The terminal operation unit 15 can be configured by operation devices such as a keyboard, a mouse, and a touch panel. The terminal control unit 16 can be configured by, for example, a CPU (central processing unit). The terminal storage unit 17 can be configured by a memory device.

  The server device 20 receives, from the user terminal 10, a plurality of images and information necessary for image synthesis including positional information of the plurality of images, a receiving unit 22, and transmits an image processing result of the server device 20 to the user terminal 10. Transmitting section 24, a composite image generating section 26 for generating a composite image showing the surface of a structure based on a plurality of images and information necessary for image composition, and image analysis of the composite image or the image before the composition. A crack detection unit 28 that detects cracks on the surface of the structure, and a crack information generation unit 30 that generates crack information indicating a crack image corresponding to the detected crack and a characteristic amount of the detected crack. And a machine learning unit 32 that machine-learns crack detection based on a user operation on the terminal 10. The transmitting unit 24 transmits the synthesized image and the crack information to the user terminal 10 and causes the screen of the user terminal 10 to display the synthesized image and the crack information. The receiving unit 22 receives, from the user terminal 10, a request for correcting crack information according to a user operation on the user terminal 10. The crack information generation unit 30 corrects the crack information based on the correction request. The transmitting unit 24 transmits the corrected crack information to the user terminal 10. The database 50 associates information (a plurality of images, information necessary for image synthesis, manual operation information, etc.) received from the user terminal 10 with image processing results (synthesized images, crack information, etc.) in the server device 20. accumulate.

  In addition, the server device 20 includes a scale image generation unit 34 that generates a scale image for measuring the width of a crack indicated by the crack image on the screen of the user terminal 10 based on the generated crack information. The crack information generation unit 30 adds the generated scale image to the crack information. The transmitting unit 24 transmits the crack information to which the scale image has been added to the user terminal 10.

  The server device 20 of the present example includes a crack evaluation unit 36 that evaluates the degree of cracking based on at least the crack information and determines an evaluation category (hereinafter, also referred to as “rank information”) of the degree of cracking of the structure. The crack information generating unit 30 adds the determined rank information to the crack information. The transmitting unit 24 transmits the crack information to which the rank information has been added to the user terminal 10.

  Further, the server device 20 converts the composite image and the crack information into CAD (computer aided design) data in a predetermined data format, thereby obtaining CAD data. Which is one form).

  Further, the server device 20 includes a form creation unit 40 that creates a form (hereinafter, also referred to as an “inspection record”) indicating the inspection result of the structure including the determined classification of the degree of cracking. The transmitting unit 24 transmits the created form to the user terminal 10.

  In addition, the server device 20 includes a storage unit 42 that stores a back-end program and various information necessary for executing the back-end program.

  The receiving unit 22 and the transmitting unit 24 can be configured by a communication device for communicating with another device including the user terminal 10 via the network NW. The synthetic image generation unit 26, the crack detection unit 28, the crack information generation unit 30, the machine learning unit 32, the scale image generation unit 34, the crack evaluation unit 36, the CAD data acquisition unit 38, and the form creation unit 40 are configured by, for example, a CPU. can do. It may be constituted by a plurality of CPUs. The storage unit 42 can be configured by a memory device.

  The position information of an image transmitted from the user terminal 10 to the server device 20 (hereinafter also referred to as “image position information”) indicates a positional relationship of each image with respect to the surface of the structure (that is, a plurality of objects on the surface of the structure). There is a case where the position of each of the imaging regions is indicated) and a case where the relative positional relationship between a plurality of images is indicated.

<Image correction>
The composite image generation unit 26 of the server device 20 composites a plurality of images after image correction. The composite image generation unit 26 of the present example can derive a correction parameter based on a plurality of images and perform correction on the plurality of images. The composite image generation unit 26 of the present example has a function of performing at least one of the following image corrections 1 to 4 on a plurality of images.

  Image correction 1 (image position information correction): Image processing for matching the relative positional relationship between a plurality of images with the relative positional relationship between a plurality of imaged regions on an imaged surface (for example, a bridge slab). It is. That is, when it is determined that incorrect image position information has been received from the user terminal 10, the incorrect image position information is corrected to more accurate image position information. For example, based on a crack vector obtained by vectorizing a crack in an image, if there is a crack straddling between adjacent images, but the crack image is separated between adjacent images and is not connected, the crack is adjacent. To connect the images. This correction may be performed based on a non-crack vector obtained by vectorizing a non-crack (for example, a pattern of a structure) of a structure in the image. When an image is taken for each crest of the floor slab 2 of the bridge 1, the image position information may be corrected based on the crimper edge in the image. The composite image generation unit 26 of the present example has a function of performing image position information correction based on the latest image at the time of inspection, as well as an image at the time of the latest inspection and an image at the time of the past inspection (the composite image or the image before synthesis). Image) based on the image. This makes it possible to easily correct incorrect image position information to accurate image position information. If it is determined that accurate image position information has been received from the user terminal 10, this correction need not be performed.

  Image correction 2 (scale reduction): Image processing for matching the scales of a plurality of images (also referred to as “enlargement / reduction”). That is, when it is determined that a plurality of images having different distances from the imaging device 60 to the imaging surface of the structure have been received from the user terminal 10, a plurality of images whose distances to the imaging surface are more consistent ("" (Also referred to as "an image at a fixed distance"). When distance information is associated with an image, the image may be enlarged or reduced based on the distance information. For example, based on a crack vector obtained by vectorizing cracks in an image, if there are multiple cracks straddling between adjacent images, but the intervals between the cracked images do not match between the adjacent images, the plurality of cracks Image processing for matching the interval between images between adjacent images is exemplified. The non-crack of the structure in the image (for example, the pattern of the structure) may be corrected based on the vectorized non-crack vector. When an image is taken for each crest of the floor slab 2 of the bridge 1, the scale correction may be performed based on the crimper edges in the image. The composite image generation unit 26 of the present example has a function of performing scale correction based on the latest inspection image, a latest inspection image and a past inspection image (a composite image or an image before combination). Has the function of performing scale correction based on When the divided imaging is performed at a fixed distance, the main correction need not be performed.

  Image correction 3 (tilt correction): Image processing for correcting the tilt of each of a plurality of images. That is, when it is determined that a plurality of images in which the inclination angles of the imaging surface of the structure with respect to the imaging direction of the imaging device do not match are received from the user terminal 10, the plurality of images in which the inclination angles of the imaging surfaces are more consistent. (Also referred to as an “image with a constant tilt angle”). When tilt angle information is associated with an image, tilt correction may be performed based on the tilt angle information. For example, the direction of the crack is matched based on a crack vector obtained by vectorizing the crack in the image. The non-crack of the structure in the image (for example, the pattern of the structure) may be corrected based on the vectorized non-crack vector. When an image is taken for each crest of the floor slab 2 of the bridge 1, a tilt correction may be performed based on the crimper edge in the image. The composite image generation unit 26 of the present example has a function of performing a tilt correction based on the latest image at the time of inspection, an image at the latest inspection and an image at the time of the past inspection (a composite image or an image before combination). Has a function of performing tilt correction based on When the divided imaging is performed at a fixed inclination angle, the main correction need not be performed.

  Image correction 4 (rotation angle correction): Image processing for correcting the rotation angle of each of a plurality of images with respect to the imaging surface. That is, when it is determined that a plurality of images in which the rotation angles (also referred to as “azimuth angles”) of the imaging device with respect to the imaging surface with respect to the imaging surface of the structure do not match are received from the user terminal 10, The images are corrected to a plurality of images whose rotation angles are more consistent (also referred to as “images with a constant rotation angle”). When rotation angle information is associated with an image, rotation may be performed based on the rotation angle information. For example, the direction of the crack is matched based on a crack vector obtained by vectorizing the crack in the image. The non-crack of the structure in the image (for example, the pattern of the structure) may be corrected based on the vectorized non-crack vector. When an image is taken for each crest of the floor slab 2 of the bridge 1, the rotation angle may be corrected based on the crimper edge in the image. The composite image generation unit 26 of the present example has a function of performing the rotation angle correction based on the latest image at the time of inspection, and also has the latest image at the time of inspection and the image at the time of the past inspection (the composite image or the image before synthesis). ) Has the function of correcting the rotation angle based on the above. When the divided imaging is performed at a fixed rotation angle, the main correction need not be performed.

<Batch correction of crack information>
A description will be given of a process of correcting crack information performed by the crack information generation unit 30 of the server device 20 according to the present embodiment in response to a user operation on the user terminal 10.

  The crack information generation unit 30 of the server device 20 of the present embodiment corrects the crack information for a plurality of cracks at once based on the feature amount of the crack designated by the user at the user terminal 10 (hereinafter referred to as “collective correction”). ). The receiving unit 22 of the server device 20 receives, from the user terminal 10, a correction request in which the content of the correction of the crack information is indicated by the characteristic amount of the crack. The crack information generation unit 30 of the server device 20 corrects the crack information based on the feature indicated by the correction request. Various modes can be cited as modes for correcting (collective correction) of crack information based on such feature amounts.

  As a first batch correction mode, there is a mode in which batch correction is performed in response to a manual operation using a slider displayed on the screen of the user terminal 10 (hereinafter, referred to as “slide operation”). The crack information generation unit 30 according to the present embodiment includes a frequency distribution (also referred to as a “histogram”) indicating the detection frequency of a line segment (hereinafter also referred to as a “linear pattern”) in the composite image with respect to the feature amount, and the crack information by the feature amount. Is added to the crack information transmitted to the user terminal 10, and the crack information is corrected according to the operation of the user terminal 10 using the slider.

  FIG. 3 is a diagram showing a frequency distribution and a first display example of a slider. In this example, crack images CR1, CR2, CR3, and CR4 indicating the detected cracks are displayed on the composite image IMG1 displayed on the screen of the user terminal 10. Further, in the vicinity of the composite image IMG1, a frequency distribution HG1 (histogram) indicating the frequency of detection of a line segment (a crack candidate) in the composite image IMG1 with respect to the characteristic amount, and a correction content of the crack information are input by the characteristic amount. Is displayed. The y direction (vertical direction of the screen) of the frequency distribution HG1 indicates the feature amount of the line segment that is a crack candidate, and the x direction (the horizontal direction of the screen) of the frequency distribution HG1 indicates the frequency of the line segment that is a crack candidate. The slider SL1 is a slide-type GUI (graphical user interface) for adjusting a criterion for determining whether or not a line segment in the composite image IMG1 is cracked. The slider SL1 of this example has one bar BR1, and the crack target area AR1 in the frequency distribution HG1 is determined based on the display position of the one bar BR1. In the vertical direction (x direction) of FIG. 3, in the frequency distribution HG1, an area above the display position of the bar BR1 is a crack target area AR1, and a line segment having a feature amount in the crack target area AR1 is “ It is detected as "crack".

  FIG. 4 is a diagram showing a frequency distribution and a second display example of the slider. The slider SL2 of this example has two bars BR21 and BR22, and the crack target area AR2 is determined based on the display positions of the two bars BR21 and BR22. In the y direction of FIG. 4 (vertical direction of the screen), a region between the two bars BR21 and BR22 in the frequency distribution HG1 is a crack target region AR2, and a line having a feature amount in the crack target region AR2. Minutes are detected as "cracks".

  FIG. 5 is a diagram showing a third distribution example of the frequency distribution and the slider. In this example, two sliders SL1 and SL2 are displayed, and the crack target areas AR1 and AR2 can be selectively input based on a plurality of feature amounts.

  FIG. 6 is a diagram showing a frequency distribution and a fourth display example of the slider. In this example, check boxes BX1, BX2, and BX3 are displayed, and the type of damage can be selected and input.

  The feature quantity includes at least one of a crack direction, a length, a width, an edge strength, and an edge density. Other feature values may be used. The crack information generation unit 30 adds a frequency distribution (histogram) and a slider to the crack information, and corrects the crack information according to a manual operation of the user terminal 10 using the slider.

  As described above, the frequency distribution indicating the detection frequency of the line segment with respect to the characteristic amount (which is the characteristic amount of the line segment detected by the crack detection unit 28), and the instruction for inputting the correction content of the crack information by the characteristic amount By adding a slider to the crack information and correcting the crack information according to the sliding operation using the slider on the user terminal 10, it is possible to correct the crack information with higher accuracy by a simple operation. .

  Second, there is a mode in which batch correction is performed in response to a group selection operation on a cracked image displayed on the screen of the user terminal 10. The crack information generation unit 30 according to the present embodiment groups crack images based on the feature amount, and corrects the crack information according to a manual operation of selecting a group of crack images on the user terminal 10.

  FIG. 7 shows an example in which cracked images are grouped based on directions (one form of feature amount) and displayed with different line types for each group. In this example, one group is formed of a plurality of crack images in the same direction and similar directions (images corresponding to line segments detected as cracks by the crack detection unit 28). The crack image of the first group G1 is displayed by a solid line, the crack image of the second group G2 is displayed by a broken line, and the crack image of the third group G3 is displayed by an alternate long and short dash line. In addition, for convenience of illustration, the case where the line type is displayed for each group is illustrated, but the display may be performed in a different color for each group.

  The user looks at the display on the screen of the user terminal 10 and performs a touch operation surrounding at least one cracked image in the group to be deleted. Then, based on the direction of the enclosed cracked image, all the cracked images in the group to which the cracked image belongs (cracked images in the same direction and a similar direction) are collectively erased. At the same time, the server 20 deletes the crack information of the selected group. In the example shown in FIG. 7, the crack image of the same group included in the selection area AR surrounded by the touch operation is deleted from the screen, and the crack information of the same group is deleted.

  Instead of the operation surrounding the cracked image, the user may directly select the group to be deleted. For example, when deleting the crack information of the second group G2 indicated by the dotted line in FIG. 7, the user terminal 10 accepts the operation of selecting the “dotted line”. When a cracked image is displayed in a different color for each group, an operation of selecting a color (for example, “red”) is accepted. An operation of selecting group identification information (for example, a group number) may be accepted.

  In addition, it is preferable to accept not only “deletion” of crack information but also “add”, “increase”, and “decrease” operations of crack information.

  As described above, the crack images are grouped based on the feature amounts, and the crack information is corrected in accordance with the user operation of selecting the group of the crack images on the user terminal 10, so that the crack is more highly accurate with a simple operation. The information can be corrected.

<Machine learning>
The machine learning unit 32 of the server device 20 is configured by, for example, artificial intelligence. As the artificial intelligence, for example, an artificial neural network may be used, or another known artificial intelligence may be used. However, the present invention can be applied to a case where artificial intelligence is not used as the machine learning unit 32 (for example, a case where information is acquired using a simple search unit), and the machine learning unit 32 can be omitted. .

  The machine learning unit 32 performs machine learning on crack detection based on a user operation on the user terminal 10 as first learning. The machine learning unit 32 of this example includes at least a history of a user operation for editing crack information on the user terminal 10, an image stored in the database 50 (a synthesized image or an image before synthesis), and a database Machine detection of cracks is machine-learned based on the crack information stored in 50.

  The machine learning unit 32 of this example learns from the editing history (a history indicating the content of the correction of the crack information) stored in the database 50 based on the operation history regarding the crack information stored in the database 50. An editing history to be specified, and a characteristic amount of a crack to be detected from a synthesized image (or an image before synthesis) based on the specified editing history and a synthesized image and crack information corresponding to the specified editing history. Machine learning. That is, the model used for crack detection by the crack detection unit 28 is updated.

  Note that the crack information generation unit 30 of the present example has a function of “batch correction” for receiving a crack information correction instruction in units of feature amounts as described above. Therefore, the machine learning unit 32 of the present example machine-learns crack detection based on the correspondence between the user operation history of the feature amount unit on the user terminal 10 and the synthesized image and the crack information. When cracks are detected by image analysis of an image before combining, machine learning is performed based on the correspondence between the operation history of the feature amount unit and the image before combining and the crack information.

  Further, the machine learning unit 32 performs the machine learning of the image correction based on the user operation on the user terminal 10 as the second learning. The machine learning unit 32 of this example is based on at least a history of a user operation for image correction on the user terminal 10 and images stored in the database 50 (an image before correction and an image after correction). Machine learning of image correction.

  The machine learning unit 32 of the present example learns from the image correction history (a history indicating the content of the image correction) stored in the database 50 based on the operation history of the image correction stored in the database 50. The image correction history to be specified is specified, and the image before correction is determined based on the specified image correction history and the image before correction and the corrected image (may be a composite image) corresponding to the specified image correction history. Machine learning of the correspondence between the image and the contents of the image correction. That is, the model used in the image correction of the composite image generation unit 26 is updated.

  Note that the composite image generation unit 26 of this example has functions of image position information correction, scale correction, tilt correction, and rotation angle correction as described above. Therefore, the machine learning unit 32 of this example includes a user operation history for image position information correction, scale reduction, tilt correction, or rotation angle correction at the user terminal 10 and an image before correction and an image after correction (synthesized image). Machine learning of image correction based on the correspondence relationship with

  2. Description of the Related Art Artificial intelligence technology has advanced remarkably, and artificial intelligence capable of performing image analysis beyond the brains of excellent humans is expected. However, when an unlearned event occurs (for example, in the initial stage of learning), a correction operation may be required. Therefore, even when artificial intelligence is employed as the machine learning unit 32, a function of editing crack information by a simple user operation, such as the above-described “batch correction”, is implemented in the server device 20 together with the machine learning function. Is preferred.

<First Example of Image Processing>
FIG. 8 is a flowchart showing the flow of the first example of image processing in the first embodiment.

  The front end process on the left side of the figure is started by the user terminal 10 according to a program (front end program) stored in the terminal storage unit 17 of the user terminal 10 (step S2). Further, the back-end processing on the right side of the figure is started by the server device 20 according to a program (back-end program) stored in the storage unit 42 of the server device 20 (step S4).

  In this example, as shown in FIG. 9, a plurality of images obtained by the imaging device 60 at the inspection site are input to the user terminal 10 in the office (step S6), and are entered in the field book by the user at the inspection site. The image position information is input to the office user terminal 10 (step S8).

  The plurality of images are obtained by dividing and photographing the space between the decks of the bridge into a plurality by the imaging device 60. The plurality of images are input by the terminal input unit 11 using, for example, wired communication or short-range wireless communication. It may be input from a storage medium such as a memory card. A plurality of images may be uploaded from the imaging device 60 to a storage device (not shown) on the network NW, and the plurality of images may be input from the storage device via the network NW.

  In the field note, information obtained by the user by looking at the floor slab of the bridge is entered. Information measured using a measuring device (not shown) may be written in a field note. In the following description, it is assumed that at least handwritten information (handwritten image position information) indicating the positional relationship between the plurality of images with respect to the floor slab of the bridge has been entered in the field notebook.

  If a CAD (computer aided design) drawing representing the floor slab of the bridge has been created, the CAD drawing is displayed on the terminal display unit 14, and the crest of the floor slab in the displayed CAD drawing is captured. A manual operation for fitting each image to the position is received by the terminal operation unit 15. In response to such a manual operation, image position information indicating the positional relationship of each image with respect to the reference position of the floor slab in the CAD drawing is input.

  When a CAD drawing representing a floor slab of a bridge has not been created, a manual operation of arranging a plurality of images on the screen of the terminal display unit 14 is received by the terminal operation unit 15 as shown in FIG. In response to such a manual operation, image position information indicating a relative positional relationship between a plurality of images is input. In this aspect, it is preferable that the server device 20 converts the relative positional relationship between the plurality of images into the positional relationship of each image with respect to the reference position of the floor slab. A manual operation of superimposing a common part between adjacent images may be accepted.

  In addition, information (hereinafter, referred to as “correction hint information”) indicating the later image correction (step S16) is input to the user terminal 10 as needed (step S10).

  For example, when all of the plurality of images are imaged from the front of the floor slab, "all front" is obtained. When only a part of the image is taken from the non-front side of the floor slab, the terminal operation unit 15 accepts a manual operation for inputting hint correction hint information such as “partial tilt”. An input of a specific tilt angle (which is a tilt angle of the floor slab with respect to the imaging direction of the imaging device 60) may be accepted, but in this case, an approximate angle may be input, and a subsequent process may be performed. In the server device 20, it is preferable to correct the approximate angle to a detailed angle. Input of correction hint information for other image correction such as scale correction (enlargement / reduction) and rotation angle correction may be received. If the tilt correction, the scale correction, and the rotation angle correction are unnecessary, this step may be omitted.

  Next, the image processing request information is generated by the terminal control unit 16 of the user terminal 10, and the generated image processing request information is transmitted to the server device 20 by the terminal transmission unit 12 of the user terminal 10 (step S12). ). The image processing request information is received by the receiving unit 22 of the server device 20 and registered in the database 50 (Step S14).

  The image processing request information includes the plurality of images input in step S6 and the image position information input in step S8. When the correction hint information is input in step S10, the correction hint information is also included in the image processing request information.

  Next, a plurality of images are corrected by the composite image generation unit 26 of the server device 20 (Step S16). For example, necessary correction is performed among the above-described image corrections (image position information correction, scale correction, tilt correction, and rotation angle correction). When correction hint information is input in step S10, image correction is performed based on the correction hint information.

  Next, the composite image generation unit 26 of the server device 20 generates one composite image for each floor space, based on the plurality of images and the image position information (step S18). The composite image is registered in the database 50.

  Next, the machine learning unit 32 of the server device 20 determines whether or not the synthesized image is inappropriate (step S20). The machine learning unit 32 learns an event in which the composite image is appropriate and an event in which the composite image is inappropriate based on various types of information stored in the database 50. It can be determined whether or not it is appropriate. The machine learning unit 32 can have, as a learning result, a function of detecting an abnormality such as a pattern on a structure not being connected between original images (images before synthesis). If it is determined that the synthesized image is inappropriate (YES in step S20), the parameters related to image correction and image synthesis are changed, and the process returns to step S16. If it is determined that the synthesized image is appropriate (NO in step S20). ), The process proceeds to step S22.

  Next, the crack detection unit 28 of the server device 20 performs image analysis of the composite image of each floor space, and detects cracks of the floor space for each space (step S22).

  Next, the crack information generation unit 30 of the server device 20 generates crack information indicating a crack image corresponding to the detected crack and a characteristic amount of the detected crack (step S24). The feature quantity includes information necessary for determining the evaluation category of the degree of cracking of the floor slab, such as the width, length, and interval of the crack. The feature amount includes information necessary for batch editing. The generated crack information is registered in the database 50.

  In this example, the frequency distribution, slider, group information, and the like necessary for batch editing are added to the crack information. Further, the scale image generated by the scale image generation unit 34 may be added to the crack information. A specific example of the scale image will be described later.

  The crack evaluation unit 36 evaluates the degree of cracks in the floor slab of the bridge based on at least the crack information, generates rank information indicating the evaluation category of the degree of crack, and adds the rank information to the crack information. Is also good. In the following, it is assumed that rank information has not been added to the crack information for convenience of explanation.

  Next, the transmission unit 24 of the server device 20 transmits the composite image and the crack information for each coffer of the floor slab to the user terminal 10 (step S26). The composite image and the crack information are received by the terminal receiving unit 13 of the user terminal 10 and displayed on the terminal display unit 14 (Step S28).

  For example, as shown in FIG. 11, on the screen of the terminal display unit 14, a composite image IMG2 for each floor space and crack information including a crack image CR are displayed. In this example, the crack image CR is superimposed on the position where the crack is detected in the composite image IMG2. Although the scale image is omitted in FIG. 11, the scale image may be further superimposed and displayed on the composite image IMG2.

  Next, it is determined by the terminal operation unit 15 of the user terminal 10 whether or not a correction operation of crack information has been received from the user (step S32). If the correction operation has been received (YES in step S32), the user terminal 10 The terminal transmitting unit 12 transmits a request for correcting crack information corresponding to the user's correction operation to the server device 20 (step S34). The correction request is received by the receiving unit 22 of the server device 20 and registered in the database 50 (Step S36).

  In addition, the user can input the rank information indicating the evaluation classification of the degree of cracking through the terminal operation unit 15 while viewing the display on the terminal display unit 14.

  FIG. 12 is a diagram schematically illustrating an example of a crack progressing model. In this progress model, rank information is represented by five levels of “RANK1” to “RANK5”. The correspondence between each rank information and the crack state is as follows.

  RANK 1: No damage.

  RANK2: A state in which a plurality of cracks are generated in parallel along the lateral direction (the short direction orthogonal to the vehicle traffic direction). This is a stage where a plurality of cracks due to drying shrinkage become parallel beams.

  RANK3: A state in which a crack in a vertical direction (a longitudinal direction parallel to a traffic direction of a vehicle) and a crack in a lateral direction cross each other. This is a stage where the plurality of cracks are formed into a lattice by the live load, and the crack density in the lattice-like region increases. In the latter half, cracks penetrate in the vertical direction of the floor slab (vertical direction perpendicular to the lower surface of the floor slab).

  RANK4: A state in which the crack density of the lattice region exceeds a specified value and the fracture surfaces of a plurality of penetrating cracks are smoothed. This is the stage where the floor slab loses shear resistance due to the rubbing action.

  RANK5: A state in which dropout has occurred. Wheel load exceeding the reduced punching shear strength causes dropout.

  Next, the crack information generation unit 30 of the server device 20 corrects the crack information based on the received correction request (step S38). When rank information is input at the user terminal 10, the rank information is added to the crack information.

  Next, the corrected crack information is transmitted to the user terminal 10 by the transmission unit 24 of the server device 20 (step S40). The corrected crack information is received by the terminal receiving unit 13 of the user terminal 10 and displayed on the terminal display unit 14 (Step S42).

  Next, it is determined whether or not an input of a form creation instruction has been received from the user by the terminal operation unit 15 of the user terminal 10 (step S44), and if the input of the form creation instruction has been received (YES in step S44), The form request is transmitted to the server device 20 by the terminal transmission unit 12 of the user terminal 10 (Step S46). The form request is received by the receiving unit 22 of the server device 20 (Step S48).

  Next, the form creation unit 40 of the server device 20 creates a form (inspection record) showing the inspection result of the floor slab of the bridge (step S50). The form includes rank information indicating an evaluation category of the degree of cracking in the floor slab of the bridge.

  Next, the form is transmitted to the user terminal 10 by the transmission unit 24 of the server device 20 (step S52). The form is received by the terminal receiving unit 13 of the user terminal 10, and the user is notified of the form reception by the terminal display unit 14 (step S54). The form can be displayed on the terminal display unit 14 according to a user operation, and can also be printed by a printer (not shown) connected to the network NW.

  FIG. 13 is a diagram illustrating an example of a damage diagram as a form. This damage diagram shows damage indications (crack indications, water leak indications, free lime indications, etc.), material names (such as "slabs") for each damage that occurred on the floor slab of the bridge to be inspected. Element numbers (such as Ds0201), types of damage (such as "crack", "water leakage", and "free lime"), and evaluation categories (rank information) of the degree of damage are described. In FIG. 13, the five-grade evaluation categories (“RANK1” to “RANK5”) shown in FIG. 12 are represented by alphabetical letters “a” to “e”.

<Second image processing example>
FIG. 14 is a flowchart illustrating the flow of the second image processing example according to the first embodiment.

  Steps S2 to S16 in this example are the same as those in the first image processing example shown in FIG.

  In this example, each image before synthesis is subjected to image analysis to detect cracks (step S218), and generate a synthesized image (step S220). Next, the machine learning unit 32 determines whether or not the synthesized image is inappropriate (step S222). The machine learning unit 32 learns an event in which the synthesized image is appropriate and an event in which the synthesized image is inappropriate based on various types of information stored in the database 50, and based on the learning result, the current synthesis. It can be determined whether the image is inappropriate. The machine learning unit 32 can have, as a learning result, a function of detecting an abnormality, for example, such that a cracked image is not connected between original images (images before synthesis). If determined to be inappropriate (YES in step S222), parameters related to image correction and image composition are changed, and the process returns to step S16. If determined to be appropriate (NO in step S222). The process proceeds to step S24.

  Steps S24 to S54 are the same as those in the first image processing example shown in FIG.

<Third image processing example>
FIG. 15 is a flowchart illustrating the flow of the third image processing example according to the first embodiment.

  In the present example, as shown in FIG. 16, a plurality of two-view images obtained by dividing and imaging by the imaging device 62 mounted on the robot device 70 at the inspection site are input by the user terminal 10 (step S306), and the robot The imaging device control information according to the imaging device control of the device 70 is input by the user terminal 10 (step S308). The imaging device control information of this example includes image position information, distance information, tilt angle information, and rotation angle information.

  The robot device 70 of this example is a suspension type robot suspended from a bridge, and FIG. 16 shows only a main part related to imaging. It should be noted that the suspension type robot may be a robot that suspends on a running body (vehicle) on the bridge instead of hanging on the bridge. The present invention is also applicable to a case where an unmanned flying robot such as a drone (unmanned flying object) is used instead of the suspension robot.

  The imaging device 62 of the present example is of a compound-eye type, and acquires a two-viewpoint image (also referred to as a “stereoscopic image”) by imaging the imaging surface of the structure from each of two viewpoints. The robot device 70 can calculate the distance from the imaging device 62 to each point in the imaging target surface by performing image processing on the two viewpoint images. Further, the robot device 70 of the present example includes an imaging device control mechanism 72 that controls rotation and movement of the imaging device 62. The imaging device control mechanism 72 can rotate the imaging device 62 in each of the pan direction P and the tilt direction T, and an XYZ movement mechanism that can move the imaging device 62 in each of the X, Y, and Z directions orthogonal to each other. Double pan / tilt mechanism. By the imaging device control mechanism 72, the position of the imaging device 62 with respect to the reference point on the imaging surface (surface) of the structure can be set freely. That is, position information of each of a plurality of images obtained by a plurality of divided imagings can be set freely. The imaging device control mechanism 72 also controls the distance between the imaging device 62 and the imaging surface of the structure, the inclination angle between the imaging direction of the imaging device 62 and the imaging surface of the structure, and the imaging direction and structure of the imaging device 62. The rotation angle of the object with respect to the imaging surface can be freely set. That is, it is possible to freely set the distance information, the tilt angle information, and the rotation angle information of each of the plurality of images obtained by the plurality of divided imagings.

  The terminal input unit 11 of the user terminal 10 transmits the imaging device control information including image position information, distance information, tilt angle information, and rotation angle information together with a plurality of images via a storage medium such as a memory card or a network NW. , Can be entered.

  Steps S12 to S54 are the same as those in the first image processing example shown in FIG.

  If all information necessary for image synthesis or image correction, such as image position information, distance information, tilt angle information, and rotation angle information, is accurately obtained by measurement, an appropriate synthesized image is generated. Therefore, step S20 (determination of whether or not the synthesized image is inappropriate) can be omitted.

<Fourth image processing example>
FIG. 17 is a flowchart illustrating the flow of the fourth image processing example according to the first embodiment.

  Steps S2, S4, S306 to S308, and S12 to S16 of this example are the same as those of the third image processing example shown in FIG. Similarly to the third image processing example, a plurality of two-viewpoint images obtained by dividing and imaging by the compound eye type imaging device 62 mounted on the robot device 70 are input (step S306), and the imaging of the robot device 70 is performed. Image capturing device control information according to the image capturing device control of the device control mechanism 72 is input (step S308).

  Steps S218 to S222 are the same as those in the second image processing example shown in FIG. That is, after analyzing the image before synthesis in Step S218 to detect a crack, a generation and registration of a synthesized image are performed in Step S220, and it is determined in Step S222 whether the synthesized image is inappropriate. Step S222 can be omitted similarly to S20 of the third image processing example.

  Steps S24 to S54 are the same as in the first image processing example.

  Note that an image processing example in which a person operates the imaging apparatus to image a structure (a first image processing example and a second image processing example), and that the robot apparatus 70 controls the imaging apparatus to generate a structure. The image processing examples (third image processing example and fourth image processing example) in the case of imaging are described separately for the sake of convenience, but the present invention is implemented in a closed configuration in any one of the cases. It is preferable that the present invention is not limited to the case, but rather implemented in an open configuration including both cases. That is, the server device 20 of the present embodiment not only stores, in the database 50, an image when a person operates the imaging device, information (including image position information) necessary for image synthesis and correction, an operation history, and the like. The image when the robot device 70 controls the imaging device, the imaging device control information (including image position information), the operation history, and the like are stored in the database 50, and machine learning is performed in both cases. Accordingly, even when a person operates the imaging device to image a structure, a synthesized image that is more appropriate than a closed configuration is obtained based on the machine learning result of image processing when the robot device 70 controls the imaging device. Can be generated.

[Second embodiment]
FIG. 18 is a block diagram illustrating a configuration example of an image processing system according to the second embodiment. The user terminal 10 in the figure is the same as in the first embodiment. The server device 200 of this figure has a configuration in which a prediction unit 44 for predicting the progress of cracks is added to the server device 20 of the first embodiment shown in FIG.

  The prediction unit 44 predicts the progress of cracks in the bridge based on various information related to the bridge stored in the database 50 and the crack information generated by the crack information generation unit 30. The transmitting unit 24 transmits the prediction result of the progress of the crack to the user terminal 10.

  The database 50 in this example stores bridge structural information, bridge environmental condition information, bridge use condition information, bridge maintenance management history information, and bridge disaster history information. The prediction unit 44 of this example includes at least one of bridge structural information, bridge environmental condition information, bridge use condition information, bridge maintenance management history information, and bridge disaster history information, and bridge crack information. Based on these, the progress of cracks in the bridge is predicted.

  The rank information is, for example, an evaluation section for each inspection site of the bridge (for example, a space between floor slabs). The rank information may include a “health degree” that indicates the health of the bridge in a stepwise or numerical manner.

  The structural information of the bridge indicates the structure of at least the portion including the floor slab to be evaluated and its peripheral members in the entire bridge. For example, there are structural forms and structural characteristics. Includes structural information related to the progression of floor slabs, such as the relationship between load and floor slab, the relationship between earthquake vibration and floor slab, the relationship between rainwater and floor slab, and the relationship between temperature and floor slab . Chemical information such as whether or not a material that easily causes an alkali-aggregate reaction may be used. Information that directly indicates a portion where cracks are likely to occur may be used.

  The environmental condition information of the bridge includes, for example, location conditions such as whether or not the salt comes.

  The bridge use condition information includes, for example, the traffic volume of vehicles and the mixing ratio of large vehicles.

  The bridge maintenance management history information includes, for example, spraying history information of an antifreeze agent. Repair and reinforcement history information may be included.

  The disaster history information of a bridge is history information of a disaster such as a typhoon or an earthquake received by the bridge.

  The database 50 of the present example further stores rank information indicating an evaluation category of the degree of cracking of the bridge in association with the composite image of the bridge and the crack information. The prediction unit 44 in this example executes a simulation process for predicting how the rank information of the bridge evaluated by the crack evaluation unit 36 will change in the future.

  Further, the prediction unit 44 may simulate how it has changed from the past to the present. By generating the crack information based on the simulation result from the past to the present and the current image (the synthesized image or the image before the synthesis) by the crack information generation unit 30, the accuracy of the crack information can be further improved. it can.

  The machine learning unit 32 of this example can machine-learn crack detection based on the prediction result of the prediction unit 44 in addition to the operation history, the image (the synthesized image or the image before the synthesis), and the crack information. It is. The machine learning unit 32 further performs a crack based on at least one of bridge structural information, bridge environmental condition information, bridge use condition information, bridge maintenance management history information, and bridge disaster history information. Learning detection is preferred.

[Vectorization]
Vectorization of a cracked image will be described with reference to FIGS.

  The crack information generation unit 30 of the server device 20 generates a vector of a crack (hereinafter, referred to as a “crack vector” or simply “vector”) by vectorizing the crack portion extracted from the synthesized image or the image before the synthesis. That is, the crack information generation unit 30 converts the cracked portion into a vector. The transmission unit 24 of the server device 20 transmits the crack information including the crack vector, which is the vectorized crack image, to the user terminal 10 together with the composite image. On the terminal display unit 14 of the user terminal 10, the vectorized cracked image is displayed superimposed on the composite image.

  When vectorizing, the crack information generation unit 30 binarizes and / or thins the detected cracks as necessary. `` Vectorization '' means finding a line segment defined by a start point and an end point for a crack.If the crack is curved, the crack is divided into a plurality of sections so that the distance between the curve and the line segment is equal to or less than a threshold. Then, a crack vector is generated for each of the plurality of sections.

  When generating the crack vector, for example, the feature point of the floor slab 2 is set as the origin of the coordinate system, and for a group of crack vectors (vector group), the end point at which the distance from the origin is the minimum is set as the first start point. The end point and the start point can be sequentially determined along the traveling direction. In the example of FIG. 19, when the point P0 on the floor slab 2 is the origin of the coordinate system, and the rightward and downward directions in the figure are the X-axis direction and the Y-axis direction of the coordinate system, respectively, the points P13, Among the points P14, P15, and P16, the point P13 at which the distance d from the point P0 is the shortest is set as the start point of the crack vector C7-1, and the point P14 is hereinafter referred to as the end point of the crack vector C7-1 (and the crack vectors C7-2, C7). -3) and the points P15 and P16 can be the end points of the crack vectors C7-2 and C7-3, respectively.

  However, when the starting point of the vector group C8 is determined in the same manner, the point P17 is the starting point of the crack vector C8-1, the point P18 is the starting point of the crack vectors C8-2 and C8-3, and the running direction of the crack vector C8-3. (The direction from the point P18 to the point P20) is opposite to the traveling direction of the crack vector C8-1. Therefore, in such a case, as shown in FIG. 20, the point P19 is set as the starting point of the crack vector C8A-1, and the point P18 is hereinafter referred to as the end point of the crack vector C8A-1 (and the starting point of the crack vectors C8A-2 and C8A-3). ) And points P17 and P20 may be the end points of the crack vectors C8A-2 and C8A-3, respectively. The aggregate of the crack vectors in this case is referred to as a vector group C8A.

  When the crack vector is generated as described above, if the crack is continuous inside the floor slab 2 but separated on the surface, the crack may be recognized as a separated crack vector. Therefore, a plurality of crack vectors are connected to generate one or a plurality of vectors.

  FIG. 21 is a diagram showing an example of the connection of the crack vectors. The vector group C3 includes the crack vector C3-1 (the points P21 and P22 are respectively the start point and the end point), and the crack vector C4-1 (the points P23 and P24). Indicate the vector group C4 including the start point and the end point, respectively. The angle formed by the crack vector C3-1 and the line connecting the points P22 and P23 is α1, and the angle formed by the line connecting the points P22 and P23 and the crack vector C4-1 is α2. At this time, if both the angle α1 and the angle α2 are equal to or smaller than the threshold value, the crack vectors C3-1 and C4-1 are connected, and the vector groups C3 and C4 are fused. Specifically, as shown in FIG. 22, a new crack vector C5-2 is generated and other crack vectors C5-1 (same as crack vector C3-1) and C5-3 (same as crack vector C4-1). ), And a new vector group including these crack vectors C5-1, C5-2, and C5-3 is referred to as a vector group C5. In this manner, by appropriately connecting the spatially separated vectors (at the surface of the floor slab 2), the connection relationship between the vectors can be accurately grasped.

  Note that the above-described vector concatenation is an example, and another method may be used.

  Further, the crack information generation unit 30 calculates a relative angle between a plurality of vectors. That is, the angle between one vector and another vector is calculated.

  Further, the crack information generation unit 30 generates hierarchical structure information indicating a hierarchical structure of the crack vector. The hierarchical structure information is information that hierarchically expresses a connection relationship between vectors. The hierarchical structure information includes hierarchical identification information (also referred to as “belonging hierarchical information”) indicating to which hierarchical layer each vector belongs in a vector group composed of a plurality of vectors. As the layer identification information, for example, a layer number represented by a numeral can be used. It may be represented by a code other than a number (for example, an alphabetic character or a symbol).

  FIG. 23 is a diagram illustrating the vector group C1. The vector group C1 is composed of crack vectors C1-1 to C1-6, and these crack vectors have points P1 to P7 as start points or end points. In such a situation, it is assumed that the hierarchy becomes lower each time the crack vector branches (the end point of a certain crack vector is the start point of another plurality of crack vectors). Specifically, the hierarchy of the crack vector C1-1 is set to the highest level “level 1”, and the hierarchy of the crack vectors C1-2 and C1-3 starting from the point P2 which is the end point of the crack vector C1-1 is a crack. It is assumed that “level 2” is lower than the vector C1-1. Similarly, the hierarchy of the crack vectors C1-5 and C1-6 starting from the point P4, which is the end point of the crack vector C1-3, is "level 3" lower than the crack vector C1-3. On the other hand, the point P3 which is the end point of the crack vector C1-2 is the start point of the crack vector C1-4, but the crack vector starting from the point P3 is only the crack vector C1-4 and has no branch, so the crack vector C1 The layer of -4 is "level 2" which is the same as C1-2. The hierarchy of each crack vector determined in this way is included in the hierarchical structure information.

  FIG. 24 is a diagram showing a vector group C1 (the connection relationship between the crack vectors is the same as that shown in FIG. 23). In the present example, among the connected crack vectors, the one that forms an angle with another crack vector that is equal to or smaller than a threshold (a crack vector corresponding to a “trunk” in a tree structure) belongs to the same hierarchy. Specifically, the crack vectors C1-1, C1-2, and C1-4 existing within the dotted line (range indicated by the reference numeral Lv1) in FIG. 24 are set to “Level 1” (the highest level) which is the same hierarchy. . In addition, for the other crack vectors C1-3, C1-5, and C1-6, it is assumed that the hierarchy becomes lower each time the crack vector branches, and the crack vector C1-3 (the "branch" in the tree structure). Equivalent) is defined as “level 2”, and the crack vectors C1-5 and C1-6 (corresponding to “leaves” in the tree structure) are defined as “level 3”. The hierarchy and type (stem, branch, or leaf) of each crack vector determined in this way are included in the hierarchical structure information.

  The hierarchical structure described with reference to FIGS. 23 and 24 is an example, and the hierarchical structure may be modified or modified.

  Further, the crack information generation unit 30 adds various types of additional information (attribute information) to the vector. For example, information (width information) indicating the width of each crack is added to each vector.

  As described above, since the crack image is vectorized and hierarchically structured by the crack information generation unit 30 of the server device 20, various processes are efficiently and appropriately performed using the generated crack vector and hierarchical structure information. It becomes possible.

  For example, image correction can be performed using a crack vector and hierarchical structure information. Further, it is possible to determine whether or not the composite image is inappropriate using the crack vector and the hierarchical structure information. Further, machine learning can be performed using the crack vector and the hierarchical structure information. In addition, it is possible to determine the evaluation category of the degree of crack using the crack vector and the hierarchical structure information. Further, it is possible to predict the progress of the crack using the crack vector and the hierarchical structure information.

[Example of scale image]
An example of a scale image generated by the scale image generation unit 34 will be described. The scale image generation unit 34 generates various scale images for measuring the characteristic amount of the crack (for example, the width of the crack) indicated by the crack image on the screen of the user terminal 10. The scale image is added to the crack information.

  FIG. 25 is a diagram illustrating a display example of a scale image. For convenience of illustration and description, a crack image CR11 (CR11-1, CR11-2, CR11-3) showing a crack to be measured is shown in an enlarged manner.

  The scale images CS1, CS2, CS3, and CS4 are images for measuring a crack generated on a measurement surface of a structure (in this example, a floor slab of a bridge). These scale images CS1 to CS4 are generated by the scale image generation unit 34 of the server device 20. The scale image generation unit 34 of the present example includes scale images CS1 and CS2 indicating a scale with a scale for measuring the length of a crack, and a scale with a line drawing and a number for measuring the width of the crack. Are generated as scale images CS3 and CS4. The scale images CS1 to CS4 indicate scales according to the distance between the imaging device and the measurement surface and the inclination angle of the measurement surface with respect to the imaging direction of the imaging device. The scale image generation unit 34 changes the type, shape, size, and the like of the scale according to the distance and the angle. The scale image generation unit 34 of this example changes the scales of the scale images CS1 and CS2 for length measurement and the line drawings and numbers of the scale images CS3 and CS4 for width measurement according to the distance and the tilt angle. ing.

  The crack information generation unit 30 of the server device 20 adds the scale images CS1 to CS4 to the crack information. The transmission unit 24 of the server device 20 transmits the crack information to which the scale images CS1 to CS4 are added to the user terminal 10 together with the composite image of the structure. The transmitted crack information with the scale image and the synthesized image are received by the terminal receiving unit 13 of the user terminal 10.

  The display area DA in the terminal display unit 14 of the user terminal 10 includes an image display area IA that displays a composite image and a scale image of a structure, and a scale image of a type that can be displayed in the image display area IA (that is, a user can select the scale image). And a scale image selection area SA for displaying a large scale image. In the scale image selection area SA, in addition to the scale images CS1 to CS4, an OK button BU1 for confirming the selection of the scale image, a cancel button BU2 for canceling the selection of the scale image, and an image display area IA are displayed. An erase button BU3 for erasing the scaled image is displayed.

  The terminal operation unit 15 receives a user's selection operation on the scale images CS1 to CS4 displayed in the scale image selection area SA. The user performs a selection operation using a button, a mouse, or the like. From the plurality of scale images CS1 to CS4 displayed in the scale image selection area SA, scale images (CS2 and CS3 in this example) corresponding to the user's selection operation are displayed in the image display area IA.

  The terminal display unit 14 displays the composite image and the scale image in a superimposed manner on the image display area IA. The position and direction of the scale image are determined according to the position and direction of the crack image in the composite image. The crack image CR11 of the present example has a hierarchical structure and includes a parent crack image CR11-1 and child crack images CR11-2 and CR11-3. The scale image CS2 for length measurement is displayed along the direction of the crack image CR11-2 and at a position near the crack image CR11-2. The scale image CS3 for width measurement has a line drawing (a line drawing of 1.4 mm in this example) having the width closest to the width of the crack image CR11-1 along a direction orthogonal to the direction of the crack image CR11-1. It is displayed in alignment with the cracked image CR11-1.

  Although not shown, a scale button for moving the scale image, a rotation button for rotating the scale image, an enlargement button for enlarging the scale image, a reduction button for reducing the scale image, and the like are also scale image selections. It is preferable to display in the area SA. Movement, rotation, enlargement, reduction, and the like of the scale image may be instructed by a mouse movement operation.

  Although the case where two scale images CS2 and CS3 are displayed in the image display area IA has been illustrated, the number of scale images displayed in the image display area IA is not limited.

  Although the embodiments for carrying out the present invention have been described above, the present invention is not limited to the above-described embodiments and modifications, and various modifications can be made without departing from the gist of the present invention.

Reference Signs List 1 bridge 2 floor slab 3 main girder 3A joint 10 user terminal 11 terminal input unit 12 terminal transmission unit 13 terminal reception unit 14 terminal display unit 15 terminal operation unit 16 terminal control unit 17 terminal storage unit 20, 200 server device 22 reception unit 24 Transmitting unit 26 Synthetic image generating unit 28 Crack detecting unit 30 Crack information generating unit 32 Machine learning unit 34 Scale image generating unit 36 Crack evaluating unit 38 CAD data obtaining unit (data converting unit)
40 form creation unit 42 storage unit 44 prediction unit 50 database 60, 62 imaging device 70 robot device 72 imaging device control mechanism AR selection region AR1, AR2 crack target regions BR1, BR21, BR22 bar BU1 OK button BU2 cancel button BU3 erase button BX1 , BX2, BX3 check boxes C1, C3, C4, C5, C7, C8, C8A vector groups C1-1, C1-2, C1-3, C1-4, C1-5, C3-1, C4-1, C5 -1, C5-2, C7-1, C7-2, C8-1, C8-2, C8-3, C8A-1, C8A-2 Vector CR, CR1, CR2, CR3, CR4, CR11, CR11-1 , CR11-2, CR11-3 Cracked images CS1, CS2, CS3, CS4 Scaled image DA Display area G1 First group G2 Second group G3 Third group HG1, HG2 Frequency distribution IA Image display area IMG1, IMG2 Composite image NW Network SA Scale image selection area SL1, SL2 Slider d Distance α1, α2 Angle

Claims (18)

A plurality of images showing the surface of the structure, and position information of the plurality of images, a receiving unit that receives from the user terminal,
Based on the received plurality of images and the position information, a composite image generation unit that generates a composite image showing the surface of the structure,
A crack detection unit that analyzes the synthesized image or the image before synthesis to detect cracks on the surface of the structure,
A crack information generating unit that generates a crack image corresponding to the detected crack and crack information indicating the feature amount of the detected crack,
A transmission unit that transmits the composite image and the crack information to the user terminal and displays the composite image and the crack information on a screen of the user terminal,
The receiving unit receives, from the user terminal, a correction request indicating the correction content of the crack information according to the user operation on the user terminal, using the feature amount.
The crack information generating unit corrects the crack information based on the feature amount indicated in the correction request,
The transmitting unit transmits the corrected crack information to the user terminal,
Adding, to the crack information, a frequency distribution indicating a detection frequency of a line segment in the composite image with respect to the feature amount, and a slider for inputting a correction content of the crack information with the feature amount; A server device that corrects the crack information in accordance with the user operation using the slider in (1) . A history of the correction request received by the receiving unit, based on the composite image and the crack information corresponding to the history, based on the machine learning unit that machine learning of the detection of the crack,
The server device according to claim 1 . A plurality of images showing the surface of the structure, and position information of the plurality of images, a receiving unit that receives from the user terminal,
Based on the received plurality of images and the position information, a composite image generation unit that generates a composite image showing the surface of the structure,
A crack detection unit that analyzes the synthesized image or the image before synthesis to detect cracks on the surface of the structure,
A crack information generating unit that generates a crack image corresponding to the detected crack and crack information indicating the feature amount of the detected crack,
A transmission unit that transmits the composite image and the crack information to the user terminal and displays the composite image and the crack information on a screen of the user terminal,
The receiving unit receives a correction request for the crack information from the user terminal according to a user operation on the user terminal,
The crack information generating unit corrects the crack information based on the correction request,
The transmitting unit transmits the corrected crack information to the user terminal,
A server device , comprising: a machine learning unit that machine-learns the detection of the crack based on the history of the correction request received by the receiving unit, the synthesized image corresponding to the history, and the crack information . The receiving unit receives, from the user terminal, the correction request indicating the correction content of the crack information in the feature amount,
The crack information generating unit corrects the crack information based on the feature amount indicated in the correction request,
The server device according to claim 3 . Adding, to the crack information, a frequency distribution indicating a detection frequency of a line segment in the composite image with respect to the feature amount, and a slider for inputting a correction content of the crack information with the feature amount; Correcting the crack information according to the user operation using the slider in,
The server device according to claim 4 . The crack information generating unit groups the crack images based on the feature amount, and corrects the crack information according to a user operation of selecting a group of the crack images on the user terminal.
The server device according to any one of claims 1, 2, 4, and 5 . The feature amount includes at least one of the direction of the crack, length, width, edge strength, and edge density,
The server device according to claim 1. The composite image generation unit, for the plurality of images, performs at least one correction of the position information, scale, tilt and rotation angle,
The server device according to any one of claims 1 to 7. The composite image generating unit derives a correction parameter based on the plurality of images, and performs the correction,
The server device according to claim 8. Based on the crack information, comprising a scale image generating unit that generates a scale image for measuring the width of the crack indicated by the crack image on the screen of the user terminal,
The crack information generating unit adds the scale image to the crack information,
The server device according to any one of claims 1 to 9. Based on at least the crack information, a degree of the crack is evaluated, comprising a crack evaluation unit that determines an evaluation category of the degree of crack of the structure,
The server device according to any one of claims 1 to 10. A form creation unit that creates a form showing an inspection result of the structure including the determined evaluation section,
The server device according to claim 11. A data conversion unit that converts the composite image and the crack information into CAD data in a predetermined data format;
The server device according to claim 1. Based on at least one of the structural information of the structure, environmental condition information, use condition information, maintenance history information, and disaster history information, and the crack information, a prediction unit that predicts the progress of the crack. Prepare,
14. The server device according to claim 1. An image processing system including a user terminal and a server device,
The user terminal,
A plurality of images showing the surface of the structure, a terminal input unit for inputting the position information of the plurality of images,
For the server device, a terminal transmission unit that transmits the plurality of images and the position information,
From the server device, a terminal receiving unit that receives an image processing result for the plurality of images,
A terminal display unit for displaying the image processing result,
With
The server device,
From the user terminal, a receiving unit that receives the plurality of images and the position information,
Based on the received plurality of images and the position information, based on the synthesized image generating unit that generates a synthesized image showing the surface of the structure,
A crack detection unit that analyzes the synthesized image or the image before synthesis to detect cracks on the surface of the structure,
A crack information generation unit that generates a crack image corresponding to the detected crack and crack information indicating the feature amount of the detected crack,
A transmission unit that transmits the composite image and the crack information to the user terminal as the image processing result for the plurality of images,
With
The terminal display unit displays the composite image and the crack information ,
The receiving unit receives, from the user terminal, a correction request indicating the correction content of the crack information according to the user operation on the user terminal, using the feature amount.
The crack information generating unit corrects the crack information based on the feature amount indicated in the correction request,
The transmitting unit transmits the corrected crack information to the user terminal,
The server device adds, to the crack information, a frequency distribution indicating a detection frequency of a line segment in the composite image with respect to the feature amount, and a slider for instructing and inputting the content of correction of the crack information with the feature amount. And correcting the crack information in accordance with the user operation using the slider in the user terminal,
Image processing system. An image processing system including a user terminal and a server device,
The user terminal,
A plurality of images showing the surface of the structure, a terminal input unit for inputting the position information of the plurality of images,
For the server device, a terminal transmission unit that transmits the plurality of images and the position information,
From the server device, a terminal receiving unit that receives an image processing result for the plurality of images,
A terminal display unit for displaying the image processing result,
With
The server device,
From the user terminal, a receiving unit that receives the plurality of images and the position information,
Based on the received plurality of images and the position information, based on the synthesized image generating unit that generates a synthesized image showing the surface of the structure,
A crack detection unit that analyzes the synthesized image or the image before synthesis to detect cracks on the surface of the structure,
A crack information generation unit that generates a crack image corresponding to the detected crack and crack information indicating the feature amount of the detected crack,
A transmission unit that transmits the composite image and the crack information to the user terminal as the image processing result for the plurality of images,
With
The terminal display unit displays the composite image and the crack information ,
The receiving unit receives a correction request for the crack information from the user terminal according to a user operation on the user terminal,
The crack information generating unit corrects the crack information based on the correction request,
The transmitting unit transmits the corrected crack information to the user terminal,
The server device, a history of the correction request received by the receiving unit, based on the synthesized image and the crack information corresponding to the history, based on the machine learning unit that machine learning of the detection of the crack,
Image processing system. A plurality of images showing the surface of the structure, and the position information of the plurality of images, receiving from a user terminal,
Based on the received plurality of images and the position information, generating a composite image showing the surface of the structure,
Image analysis of the synthesized image or the image before synthesis to detect cracks on the surface of the structure,
Generating crack information indicating the crack image corresponding to the detected crack and the feature amount of the detected crack,
Transmitting the composite image and the crack information to the user terminal to display the composite image and the crack information on a screen of the user terminal;
Receiving from the user terminal a correction request indicating the correction content of the crack information according to a user operation at the user terminal, the correction amount being indicated by the feature amount;
Correcting the crack information based on the feature amount indicated in the correction request;
Transmitting the modified crack information to the user terminal;
Adding, to the crack information, a frequency distribution indicating a detection frequency of a line segment in the composite image with respect to the feature amount, and a slider for inputting a correction content of the crack information with the feature amount; Correcting the crack information in response to the user operation using the slider in,
An image processing method including: A plurality of images showing the surface of the structure, and the position information of the plurality of images, receiving from a user terminal,
Based on the received plurality of images and the position information, generating a composite image showing the surface of the structure,
Image analysis of the synthesized image or the image before synthesis to detect cracks on the surface of the structure,
Generating crack information indicating the crack image corresponding to the detected crack and the feature amount of the detected crack,
Transmitting the composite image and the crack information to the user terminal to display the composite image and the crack information on a screen of the user terminal;
Receiving a request for correction of the crack information from the user terminal according to a user operation at the user terminal,
Correcting the crack information based on the correction request;
Transmitting the modified crack information to the user terminal;
Based on the received history of the correction request, the composite image and the crack information corresponding to the history, based on machine learning of the crack detection,
An image processing method including:

Images