WO2024029026A1 - Information processing system, program, information processing method, and server - Google Patents

Abstract

[Problem] To provide an information processing system and a program that make it possible to prevent a cracked object from being detected in a split state. [Solution] According to an embodiment of the present invention, an information processing system is provided that comprises: a crack estimation unit that estimates, with respect to an original image containing one or more objects in a structure, a cracked region in an object; a crack coloring unit that colors the estimated cracked region in the original image at least according to the color of surroundings of the cracked region; and an object region estimation unit that estimates, with respect to an image resulting from the coloring, an object region where the object is present.

Description

Information processing system and program, information processing method, server

The present invention relates to an information processing system and program for detecting cracks, an information processing method, and a server.

A crack detection device that detects cracks in the walls of buildings and civil engineering structures is disclosed in Patent Document 1. According to the crack detection device disclosed in Patent Document 1, as an example of preprocessing for crack detection, an image region representing a tile (tile region) is extracted from a target image, and an image region other than the tile region is extracted. A process of excluding it from the crack detection target range is executed (see paragraph 0022 of Cited Document 1).

Patent No. 6894339

However, if the process of extracting the tile area is performed before the crack detection process, tiles with cracks may be mistakenly treated as two tiles that were originally one tile due to the cracks. Otherwise, one part of the divided tile may be erroneously detected as missing. In this case, the crack may be recognized as a boundary (joint) between divided tiles, and the crack may not be detected correctly.

The present invention was made in view of this background, and particularly provides an information processing system and a program that can prevent objects such as tiles with cracks from being detected in a divided state. One purpose is to provide an information processing method and server.

According to one aspect of the present invention, there is provided a crack estimating unit that estimates a crack area in an object with respect to an original image in which one or more objects in a structure are reflected; Accordingly, the present invention is characterized by comprising a crack coloring unit that colors a crack area in the original image, and an object area estimating unit that estimates an object area where the object exists in the colored image. An information processing system is provided.

According to the present invention, it is possible to provide an information processing system, a program, an information processing method, and a server that can prevent an object such as a cracked tile from being detected in a divided state. can.

1 is a diagram showing the overall configuration of an embodiment of the present invention. 1 is a diagram showing a system configuration of an information processing system according to an embodiment of the present invention. 3 is a block diagram showing the hardware configuration of the server in FIG. 2. FIG. 3 is a block diagram showing the hardware configuration of the terminal in FIG. 2. FIG. FIG. 3 is a block diagram showing the hardware configuration of the aircraft shown in FIG. 2. FIG. 3 is a block diagram showing the functions of the server and terminal in FIG. 2. FIG. It is a figure explaining the shape analysis process of the crack area|region by a crack shape analysis part. 7 is a flowchart illustrating a process for implementing a crack area detection method by the information processing system according to the present embodiment. This is an example of an image in which a crack area is to be detected. FIG. 3 is a diagram conceptually showing a state in which an image to be detected is divided into a plurality of regions in a grid pattern. This is an image showing an estimated crack area on a divided area where it is estimated that the crack area exists. Figure (a) is a diagram showing a crack area estimated in an image and an enlarged area including the crack area, and Figure (b) is a diagram showing a state in which the crack area is colored. . FIG. 3 is a diagram showing a target object area estimated by a target object area estimation unit. FIG. 7 is a diagram showing a state in which the original crack region before coloring is superimposed on the estimated object region. FIG. 12 is a diagram showing an enlarged view of the object region shown in FIG. 11; FIG. 6 is a diagram showing one original detection target image reconstructed from each divided region in which the target object region has been estimated.

The contents of the embodiments of the present invention will be listed and explained. An information processing system, program, information processing method, and server according to an embodiment of the present invention has the following configuration.
[Item 1]
a crack estimation unit that estimates a crack area in one or more objects in a structure with respect to an original image in which the objects are reflected;
a crack coloring unit that colors the crack area in the original image according to at least the estimated color around the crack area;
an object area estimation unit that estimates an object area in which the object exists in the colored image;
An information processing system characterized by:
[Item 2]
further comprising a superimposing unit that specifies a crack region in the target object region by at least superimposing the estimated position of the target object region and the crack region;
The information processing system according to item 1, characterized in that:
[Item 3]
The crack estimating unit re-performs the estimation of the crack area on the original image associated with the estimated object area, and identifies the crack area in the object area.
The information processing system according to item 1, characterized in that:
[Item 4]
further comprising a crack shape analysis unit that performs a crack shape analysis on the crack region in the specified object region;
The information processing system according to item 2 or 3, characterized in that:
[Item 5]
The crack estimation unit divides the original image into two or more divided images and estimates a crack area for each divided image.
The information processing system according to any one of items 1 to 3, characterized in that:
[Item 6]
The crack coloring unit generates an enlarged area by enlarging the estimated crack area, and colors the crack area with a statistically determined color based on color information in the enlarged area.
The information processing system according to any one of items 1 to 3, characterized in that:
[Item 7]
The crack coloring unit acquires color information of the object and colors the crack area with a color that matches or substantially matches the color information.
The information processing system according to any one of items 1 to 3, characterized in that:
[Item 8]
A program that causes a computer having a processing unit to perform information processing,
The program causes the processing unit to
estimating a crack area in one or more objects in a structure with respect to an original image in which the objects are reflected;
Coloring the crack area in the original image according to at least the estimated color of the surrounding area of the crack area;
estimating an object area where the object exists in the colored image;
A program to run.
[Item 9]
estimating a crack area in one or more objects in a structure with respect to an original image in which one or more objects in the structure are reflected, by a crack estimating unit;
Coloring the crack area in the original image according to at least the estimated color of the surrounding area of the crack area using a crack coloring unit;
estimating, by a target object area estimating unit, a target object area in which the target object exists in the colored image;
An information processing method that executes on a computer.
[Item 10]
a crack estimation unit that estimates a crack area in one or more objects in a structure with respect to an original image in which the objects are reflected;
a crack coloring unit that colors the crack area in the original image according to at least the estimated color around the crack area;
an object area estimation unit that estimates an object area in which the object exists in the colored image;
A server characterized by:

<Details of embodiment>
An information processing system according to an embodiment of the present invention will be described below. In the accompanying drawings, the same or similar elements are given the same or similar reference numerals and names, and redundant description of the same or similar elements may be omitted in the description of each embodiment. Furthermore, features shown in each embodiment can be applied to other embodiments as long as they do not contradict each other.

<Overview of this embodiment>
As shown in FIG. 1, the information processing system according to the present embodiment detects cracks existing in a wall surface of a structure such as a building or a civil engineering structure based on an image taken of such a wall surface. It is something. For example, the wall surface of the structure may be imaged by the user himself or herself by operating a camera, or by a camera mounted on an unmanned flying vehicle 4 as shown in FIG. 1 that flies autonomously or by remote control. You may operate it to take an image.

As will be described later, the information processing system in this embodiment simply detects the presence or absence of a crack in an image to be inspected by performing processing to estimate an object area where an object without cracks exists. In addition to detection, it also identifies the area in the image to be inspected where an object (for example, a wall tile or panel that forms a partitioned area) is present, and also determines which part of which object is present. It also makes it possible to detect the presence of cracks.

<System configuration>
As shown in FIG. 2, the information processing system in this embodiment includes a server 1, a terminal 2, and an unmanned flying vehicle 4. The server 1, the terminal 2, and the unmanned aircraft 4 may be communicably connected to each other via the network NW. Note that the illustrated configuration is an example, and the configuration is not limited to this. For example, the unmanned flying object 4 may not be connected to the network NW. In that case, the unmanned aerial vehicle 4 may be operated by a transmitter (so-called radio) operated by a user, or the image data acquired by the camera of the unmanned aerial vehicle 4 may be stored in an auxiliary storage device (e.g. The configuration may be such that the data is stored in a memory card such as an SD card, a USB memory, etc.), and later read out from the auxiliary storage device by the user and stored in the server 1 or terminal 2, for operational purposes or image data. The unmanned aerial vehicle 4 may be connected to the network NW only for one of the storage purposes.

<Hardware configuration of server 1>
FIG. 2 is a diagram showing the hardware configuration of the server 1 in this embodiment. Note that the illustrated configuration is an example, and other configurations may be used.

The server 1 includes at least a processor 10, a memory 11, a storage 12, a transmitting/receiving section 13, an input/output section 14, etc., which are electrically connected to each other via a bus 15. The server 1 may be a general-purpose computer, such as a workstation or a personal computer, or may be logically implemented by cloud computing.

The processor 10 is an arithmetic device that controls the overall operation of the server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and developed in the memory 11 to perform various information processing.

The memory 11 includes a main memory configured with a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory configured with a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). . The memory 11 is used as a work area for the processor 10, and also stores a BIOS (Basic Input/Output System) executed when the server 1 is started, various setting information, and the like.

The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12. Further, a storage unit 130, which will be described later, may be provided in a part of the storage area.

The transmitting/receiving unit 13 is a communication interface through which the server 1 communicates with an external device (not shown), the unmanned aircraft 4, etc. via a communication network. The transmitter/receiver 13 may further include a short-range communication interface for Bluetooth (registered trademark) and BLE (Bluetooth Low Energy), a USB (Universal Serial Bus) terminal, and the like.

The input/output unit 14 is information input devices such as a keyboard and mouse, and output devices such as a display.

The bus 15 is commonly connected to each of the above elements and transmits, for example, address signals, data signals, and various control signals.

<Terminal 2>
The terminal 2 shown in FIG. 4 also includes a processor 20, a memory 21, a storage 22, a transmitting/receiving section 23, an input/output section 24, etc., which are electrically connected to each other through a bus 25. Since the functions of each element can be configured in the same manner as the server 1 described above, a detailed explanation of each element will be omitted.

<Unmanned Aerial Vehicle 4>
FIG. 5 is a block diagram showing the hardware configuration of the unmanned aerial vehicle 4. As shown in FIG. Flight controller 41 may include one or more processors, such as a programmable processor (eg, a central processing unit (CPU)).

Additionally, the flight controller 41 has a memory 411 and can access the memory. Memory 411 stores logic, code, and/or program instructions executable by the flight controller to perform one or more steps. Further, the flight controller 41 may include sensors 412 such as an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (eg, lidar), and the like.

The memory 411 may include, for example, a separable medium or external storage device such as an SD card or random access memory (RAM). Data acquired from cameras/sensors 42 may be communicated directly to and stored in memory 411. For example, still image/video data taken with a camera or the like may be recorded in the built-in memory or external memory, but the data is not limited to this. 2 may be recorded. The camera 42 is installed on the unmanned aerial vehicle 4 via a gimbal 43.

Flight controller 41 includes a control module (not shown) configured to control the state of unmanned aerial vehicle 4 . For example, the control module adjusts the spatial position, velocity, and/or acceleration of the unmanned air vehicle 4 with six degrees of freedom (translational movements x, y, and z, and rotational movements θ _x , θ _y , and θ _z ). For this purpose, the propulsion mechanism (motor 45, etc.) of the unmanned aerial vehicle 4 is controlled via an ESC 44 (Electric Speed Controller). A propeller 46 is rotated by a motor 45 supplied with power from a battery 48, thereby generating lift of the unmanned flying vehicle 4. The control module can control one or more of the states of the mounting section and sensors.

Flight controller 41 is a transceiver configured to transmit and/or receive data from one or more external devices (e.g., a transceiver 49, terminal, display, or other remote controller). It is possible to communicate with the unit 47. Transceiver 49 may use any suitable communication means, such as wired or wireless communication.

For example, the transmitter/receiver 47 uses one or more of a local area network (LAN), wide area network (WAN), infrared rays, wireless, WiFi, point-to-point (P2P) network, telecommunications network, cloud communication, etc. can do.

The transmitting/receiving unit 47 transmits and/or receives one or more of data acquired by the sensors 42, processing results generated by the flight controller 41, predetermined control data, user commands from a terminal or a remote controller, etc. be able to.

Sensors 42 according to this embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (eg, lidar), or a vision/image sensor (eg, camera).

<Function of server 1>
FIG. 6 is a block diagram illustrating functions implemented in the server 1 and the terminal 2. In this embodiment, the server 1 includes an image acquisition section 115, a processing section 120, and a storage section 130. The processing unit 120 includes a crack estimation unit 121 , a crack coloring unit 122 , an object area estimation unit 123 , a superimposition unit 124 , and a crack shape analysis unit 125 . The storage unit 130 also includes an information/image storage unit 131, a crack estimation learning model 132, and an object area estimation learning model 133. The various functional units are illustrated as functional units in the processor 10 of the server 1, but some or all of the various functional units may be implemented in the processor 10 of the server 1, the processor 20 of the terminal 2, or the controller of the unmanned aircraft 4. The configuration may be implemented in any of the processor 10, the processor 20, and the controller 41 depending on the capabilities of the processor 41 and the like.

The communication unit 110 communicates with the terminal 2 and the unmanned aerial vehicle 4 via the network NW. The communication unit 110 also functions as a reception unit that receives various requests, data, etc. from the terminal 2, the unmanned aircraft 4, and the like.

The image acquisition unit 115 acquires images captured by a digital camera mounted on the unmanned aerial vehicle 4 or a digital camera used by a user, for example, by wireless communication via a communication interface or wired communication via a USB terminal. acquired from a digital camera. The image acquisition unit 115 may be configured to acquire images via a storage medium such as a USB memory or an SD memory.

The processing unit 120 includes functional units 121 to 125 that perform a series of processes for detecting cracks in the image acquired by the image acquisition unit 115 and detecting which parts of which objects have cracks. We are prepared.

The crack estimating unit 121 performs a process of estimating a crack area existing in an original image in which one or more objects in a structure (tiles or panels on the wall of a building or civil engineering structure, etc.) are reflected. Execute. The crack estimating unit 121 of this embodiment estimates a crack area using the crack estimation learning model 132 in the storage unit 130. Details of the crack estimation learning model 132 will be described later.

The crack estimating unit 121 may perform a process of estimating a crack area for the entire original image, or may perform a process of dividing the original image into a plurality of areas and then divide each divided area. A process for estimating the crack area may also be executed. The process of dividing the original image into multiple regions and estimating the crack area can be done once for the entire original image, since by dividing the area for crack area estimation into smaller parts, the calculation amount for the process can also be divided into smaller parts. The calculation load on the crack estimating unit 121 can be reduced compared to when the process of estimating the crack area is executed. When the crack estimating unit 121 executes the process of dividing the original image into a plurality of regions, the crack estimation unit 121 executes each process described below regarding each divided region, and then reconstructs the divided regions and returns the original image to the original image. Generate an image corresponding to one image of .

The crack coloring unit 122 executes a process of coloring the crack area estimated by the crack estimation unit 121 in the image according to the color of surrounding objects, etc., and displays the colored crack area. Generate an image.

As an example of the coloring process by the crack coloring unit 122, the crack coloring unit 122 defines an enlarged area in the image that includes the estimated crack area and is wider than the crack area, and includes the area included in the enlarged area. The color is statistically determined based on the color information of objects (objects such as wall tiles and panels, joints between these objects, etc.), and the cracked area is colored with the determined color. The process of statistically determining a color based on the color information of objects within the enlarged area is, for example, weighting the color information of various colors included within the enlarged area according to the area occupied by those colors within the enlarged area. This can be done by averaging. In the image in which the cracked area is colored, the cracked area blends in with the surrounding objects, and it appears that there is almost no cracked area on the wall surface.

As another example of the coloring process by the crack coloring unit 122, the crack coloring unit 122 acquires color information of an object such as a wall tile or panel, and matches or substantially matches the color information. Color the cracked area with color. The process of acquiring the color information of the object may be performed, for example, by determining the color information having the most frequent value among the color information in the entire image as the colored color, or by determining the color information in the predetermined area adjacent to the cracked area (for example, , pixels within a specific number from the crack area) by determining the color information having the mode or average value of the color information as the colored color. Alternatively, if the color information of an object such as a wall tile or panel is known, the user may specify or preset the color information via the input/output unit 24 of the terminal 2 (see FIG. 4), for example. By doing so, it is also possible for the crack coloring section 122 to acquire color information.

The target object area estimating unit 123 calculates an object area, which is an area where a target object (such as a wall tile or panel) is present, from the image in which the crack area is colored, which is generated by the crack coloring unit 122. Execute the process of estimating. The object area estimation unit 123 of this embodiment estimates the object area using the object area estimation learning model 133 in the storage unit 130. Details of the object area estimation learning model 133 will be described later.

The object area estimation process performed by the object area estimation unit 123 detects the area in the image where the object (wall tile, panel, etc.) exists, and further calculates the position, shape, etc. of each object in that area. is detected. In other words, the position, shape, etc. of each object are individually recognized by this object area estimation process. To explain this using an example where the target object is a tile, the target area estimation process detects the area in the image where the tile exists, and also calculates the size of each tile separated by joints in that area. Position and shape are also detected individually.

The superimposing unit 124 adds the cracks estimated by the crack estimating unit 121 to the target object area estimated by the target object area estimating unit 123 in the image with the crack area colored. A process of superimposing the crack areas and identifying the crack area in the object area is executed. As described above, the object area estimation process by the object area estimation unit 123 detects the area in the image where the object (wall tiles, panels, etc.) exists and the position and shape of each object. There is. Therefore, according to the superimposing unit 124, by superimposing the crack area estimated by the crack estimating unit 121 on the area where the crack area exists in the image (the area colored by the crack coloring unit 122), Instead of simply detecting the position where a cracked area exists on a wall surface, it is possible to specify in which object, such as a tile, on a wall surface a cracked area exists. If a cracked area exists in a single object, the superimposing unit 124 identifies that object as an object in which a cracked area exists, and also specifies that the cracked area extends over multiple objects. If there are cracked areas, those multiple objects are identified as objects in which cracked areas exist.

The crack shape analysis unit 125 executes a process of analyzing the shape (length, width, etc.) of the crack area based on the crack area estimated by the crack estimation unit 121. Although various known methods can be used to analyze the shape of the crack region, the crack shape analysis unit 125 of the present embodiment analyzes the shape of the crack location by a method using a Hessian matrix, for example. It is configured to perform analysis processing.

Here, with reference to FIG. 7, a method of analyzing the shape of a crack region using a Hessian matrix will be described.

For each pixel (x, y) of the image, if the brightness value f(x, y) is in the height direction and (x, y) is a continuous variable, the image can be interpreted as a three-dimensional curved surface. The Hessian matrix for a certain pixel (x, y) among pixels is a square matrix composed of elements obtained by second-order differentiation of the brightness value of the image in the x direction and the y direction, and is expressed by the following equation (1). Ru.

... Formula (1)

Based on the relationship between the eigenvalues λ ₁ and λ ₂ of this Hessian matrix,

... Formula (2)
Pixels that satisfy the relationship are regarded as linear structures and are emphasized.

Here, FIG. 7(a) shows a conceptual diagram of the crack area A estimated by the crack estimating unit 121 and the correct actual crack area B, and FIG. 7(b) , a skeleton C having a line width of one pixel is shown, which is formed by pixels that are considered to be a linear structure as described above based on the estimated crack area A. Skeleton C indicates the stretching direction and length of the cracked portion.

Then, the linear structure is evaluated by changing the scale of the Hessian matrix on the skeleton C, and the crack width is evaluated by examining the scale at which the linearity evaluation value is maximum for pixels determined to be linear structures. (See FIG. 7(c)). By scaling the Hessian matrix and evaluating the crack width in this way, the crack shape analysis unit 125 calculates the correct actual crack area as the shape of the crack location, as shown in FIG. 7(c). A linear structure D close to B can be obtained.

In this way, by using the shape analysis method of the crack location using the Hessian matrix, the skeleton It is possible to obtain the stretching direction and length of the cracked portion based on , and obtain the width of the cracked portion based on the Hessian matrix.

The method using the Hessian matrix described above is disclosed in the paper "Classification of crack width in concrete structures using image processing" (Concrete Engineering Annual Proceedings, Vol. 34, No. 1, 2012). .

Next, the information/image storage unit 131 of the storage unit 130 stores, in addition to the image acquired by the image acquisition unit 115, an image in which the crack area is colored by the crack coloring unit 122, and each of the processing unit 120. Information, data, etc. generated through processing by the functional units 121 to 125 are stored at least temporarily.

The crack estimation learning model 132 is a learning model generated by machine learning using crack images related to various cracks as training data. The crack estimation learning model 132 can be created using, for example, an arbitrary external computer device (not shown) as a learning device, and can be stored in the storage unit 130. The crack estimation learning model 132 may be generated by machine learning using crack images as training data for each different object such as a tile or panel. An estimated learning model is generated and stored in the storage unit 130.

The crack estimation learning model 132 is generated by performing machine learning with a neural network composed of multiple layers each including neurons. As such a neural network, a deep neural network such as a convolutional neural network (CNN) can be used.

In this embodiment, in addition to object detection that estimates what appears where in an image, Mask R-CNN (Region-based Convolutional Neural Network) is used. According to Mask R-CNN, a candidate region for an object is extracted using CNN, and by simultaneously estimating the region position and class probability, the object is multiplied by a bounding box and the class to which the object belongs is determined. In addition to estimating the object (what it is), it is also possible to estimate the shape of the object by performing class classification on a pixel-by-pixel basis within the bounding box. Therefore, by using the crack estimation learning model 132 of this embodiment, it is possible to estimate not only the position of the crack region in the image but also the shape of the crack region.

The object area estimation learning model 133 is a learning model generated by machine learning using images of various objects such as tiles and panels as training data. The object region estimation learning model 133 can also be created using, for example, an arbitrary external computer device (not shown) as a learning device, and can be stored in the storage unit 130. The object area estimation learning model 133 may be generated by machine learning using cracked images for each of various objects such as tiles and panels as training data; in this case, multiple A target object area estimation learning model is generated and stored in the storage unit 130.

In the present embodiment, the object region estimation learning model 133 is also generated by performing machine learning using Mask R-CNN (Region-based Convolutional Neural Network). Therefore, by using the target region estimation learning model 133 of this embodiment, it is possible to estimate not only the region in an image where the target region exists, but also the position and shape of each target region in that region. .

<Example of crack area detection method>
Next, a crack area detection method using the information processing system according to the present embodiment will be described with reference to FIG. 8 and the like. FIG. 8 is a flowchart showing a process for implementing a crack area detection method by the information processing system according to the present embodiment.

First, the image acquisition unit 115 of the server 1 acquires original images captured by the camera mounted on the unmanned flying vehicle 4 or the camera used by the user (S101).

The original image to be acquired is an image of the object whose crack area is to be detected, such as on the wall of a building or civil engineering structure. FIG. 9 shows an example of an original image to be detected.

Next, the crack estimating unit 121 of the server 1 estimates the crack area in the original image in which one or more objects (tiles, panels, etc. on the walls of buildings, civil engineering structures, etc.) are reflected. The process to do so is executed (S102).

This process of estimating the crack area may include a process in which the crack estimation unit 121 divides the original image of which the crack area is to be detected into a plurality of areas.

FIG. 10 is a diagram conceptually showing a state in which the image of the detection target is divided into a plurality of regions in a grid pattern. In the example shown in FIG. 10, the image to be detected is divided into a total of nine regions, three vertically by three horizontally. The crack estimation unit 121 associates the image of each divided area divided by the crack estimation unit 121 with information indicating which part of the entire original image before division corresponds, and stores the image in the information/image storage unit 131. Store. Note that the process of dividing the detection target image in this way is an option, and the subsequent process may be performed on one entire image without dividing the original image of the detection target.

In the process of estimating the crack area, the crack estimation unit 121 next estimates the crack area using the crack estimation learning model 132 for each of the divided areas of the original image to be detected as described above. Execute the process to estimate.

FIG. 11 shows an image in which the estimated crack area is shown on the divided area where it is estimated that the crack area exists as a result of the crack area estimation process by the crack estimation unit 121. The crack estimating unit 121 stores information about the estimated crack area (divided area where the cracked area exists, position of the cracked area in the divided area, size/shape of the cracked area, etc.) as information/image storage. It is stored in the section 131.

Next, the crack coloring unit 122 of the server 1 executes a process of coloring the crack area estimated by the crack estimation unit 121 according to the color of surrounding objects, etc. (S103).

As an example of coloring processing, the crack coloring unit 122 defines an enlarged area in the image that includes the estimated crack area and is wider than the crack area (see FIG. 12(a)). Then, the crack coloring unit 122 statistically determines a color based on the color information of objects included in the enlarged area (in the illustrated example, wall tiles and joints between tiles), and the determined color The cracked area is colored using the steps shown in FIG. 12(b) to generate an image with the cracked area colored (see FIG. 12(b)). As shown in FIG. 12(b), in the image in which the cracked area is colored, the cracked area blends in with the surrounding objects, so that it appears that almost no cracked area exists on the wall surface. The crack coloring unit 122 stores an image of the generated crack area in a colored state in the information/image storage unit 131.

Next, the target object area estimating unit 123 of the server 1 determines whether the target object (in the illustrated example, a tile on the wall) is generated by the crack coloring unit 122 in the image with the crack area colored. A process of estimating the existing target object area using the target object area estimation learning model 133 is executed (S104).

FIG. 13 is a diagram showing the target object area estimated by the target object area estimation unit 123. Through the process of estimating the object area by the object area estimating unit 123, the area in which the object (in the illustrated example, a wall tile) exists is determined even for the image of the divided area in which the colored crack area exists. Furthermore, the position, shape, etc. of each object (tile) in that area are individually detected. In addition, in the process of step S104, the target object area estimating unit 123 similarly estimates the target area where the target object (in the illustrated example, a wall tile) exists for the divided area in which the crack area is not estimated. A process of estimating the area using the object area estimation learning model 133 is executed. The object area estimation unit 123 stores information regarding the estimated object area for each divided area in the information/image storage unit 131.

Next, the superimposing unit 124 of the server 1 adds the crack estimated by the crack estimating unit 121 to the target object area estimated by the target object area estimating unit 123 in the image (divided area) in which the crack area is colored. The crack area before the coloring process by the crack coloring unit 122 is superimposed to execute a process of specifying the crack area in the object area (S105).

The above superimposition process by the superimposition unit 124 is performed so that the original crack area before coloring overlaps the position where the colored crack area exists in the estimated object area. Through the process of estimating the target area (S104) by the target area estimation unit 123, the area where the target object (tile in the illustrated example) exists and the position, shape, etc. of each target object (tile) in the image are detected. Therefore, this superimposition process does not simply detect the location of the cracked area on the wall surface, but also identifies which object (tile) on the wall surface has the cracked area. be able to. The superimposition unit 124 transfers information indicating the relationship between the target object area and the crack area identified by this process (which target object (tile) has the crack area, etc.) to the information/image storage unit 131. Store in. FIG. 14 shows a state in which the original crack region before coloring is superimposed on the estimated object region. In the example shown in FIG. 14, two tiles in which cracked areas exist are identified by superimposition processing, and are shown with a different brightness than other tiles.

Next, the crack shape analysis unit 125 of the server 1 executes a process of analyzing the shape (length, width, etc.) of the crack area based on the crack area estimated by the crack estimation unit 121 ( S106).

For example, the crack shape analysis unit 125 is configured to perform a shape analysis process of a crack location using a method using a Hessian matrix as described above, and by the shape analysis process, the direction of extension of the crack location is determined. , the length, and the width of the cracked area are acquired as information regarding the shape of the cracked area, and the acquired information regarding the shape of the cracked area is stored in the information/image storage unit 131 . FIG. 15 is an enlarged view of the object region shown in FIG. 14, and FIG. 15 shows a skeleton generated based on the estimated crack area and an actual skeleton obtained by evaluating the crack width. A linear structure close to the crack area is shown. FIG. 15 also shows numerical information regarding the length and width of each segment constituting the crack location, which was obtained as a result of the analysis by the crack shape analysis unit 125.

Finally, if the crack estimating unit 121 has executed the process of dividing the original image into a plurality of divided regions in step S102, the crack estimating unit 121 regenerates the image of the divided divided regions into one original detection target image. The construction process is executed (S107).

The crack estimating unit 121 converts the divided areas into original parts based on information stored in the information/image storage unit 131 that indicates which part of the entire image before division the image of each divided area corresponds to. Reconstruct into two detection target images. Among the images of each divided area in which the target object area was estimated, in the divided area in which the crack area was estimated, the shape of the cracked area was analyzed and identified by the above process, and the crack location was identified in the estimated target area. is superimposed on top of.

FIG. 16 shows one original detection target image reconstructed from each divided region in which the target object region was estimated. In FIG. 16, tiles shown in colors with low brightness (close to black) are target object regions estimated by the target object region estimating unit 123. Additionally, in FIG. 16, there are cracks with specified lengths and widths in two objects (tiles) located on the left side of the window frame portion of the wall surface to be detected. It is shown. The crack estimation unit 121 stores the thus reconstructed image and various data related to the image in the information/image storage unit 131 in association with each other. Various data related to the reconstructed image include data obtained through each of the above processes (estimated object area (for example, ID may be assigned and managed), position and shape of each object, The estimated number of object regions, the object region in which the crack exists (for example, the ID may be identified and managed), the shape (length and width) of the crack, and the existence of the crack. (In particular, information regarding the number of object regions where there are cracks whose length or width exceeds a reference value). Part or all of these images and various data related to the images may be transmitted to the terminal 2 in response to a request from the terminal 2. The image and various data related to the image may be viewable by the user on a predetermined user interface via the input/output unit 24 (eg, display) of the terminal 2. In particular, by making it possible to confirm the number of object areas that have cracks (in particular, the number of object areas that have cracks that exceed the standard value in either length or width), repair Preparation of new objects to be prepared at the time of preparation becomes smooth. Note that the number of object regions in which there are cracks whose length or width exceeds a reference value is extracted by processing the processing unit 120 by determining the value of at least one of the length and width of the crack. This may be performed based on the result of comparison with a reference value, or alternatively, after data regarding cracks is received on the terminal 2, a similar comparison may be performed on the terminal 2 to extract it.

As described above, according to the server 1 of the present embodiment, the crack area in the original image is colored according to at least the color of the estimated crack area, and the target object is colored in the image in which the crack area is colored. By performing processing to estimate the area of existing objects, even objects with cracks (tiles, panels, etc.) can be prevented from being detected as being divided by the cracks. It becomes possible to detect the target object more accurately. Furthermore, by superimposing at least the estimated object area and the position of the crack area to identify the crack area in the object area, it is possible to identify not only the position of the crack area in the object area estimated in the original image but also the position of the crack area in the object area estimated in the original image. , it is also possible to estimate which object in the object region has a crack region.

(Modified example)
Next, a modification of the server 1 of this embodiment will be described.

In the embodiment described above, the superimposition unit 124 of the server 1 adds the estimated area by the crack estimating unit 121 to the object area estimated by the object area estimating unit 123 in the image (divided area) in which the crack area is colored. It has been explained that the process of specifying the crack area in the object area by superimposing the crack area before the coloring process by the crack coloring unit 122 is executed (step S105 in FIG. 8). This modification provides processing that can be an alternative to such superimposition processing.

In this modification, instead of the superimposition process by the superimposition unit 124 of the server 1 (step S105 in FIG. 5), the crack estimating unit 121 performs the process estimated by the object area estimation unit 123 in the process in step S104 in FIG. The crack area estimating process described with reference to step S102 in FIG. 8 is re-executed on the original image associated with the target object area to identify the crack area in the target object area. Similar to the superimposition process by the superimposition unit 124, this process also allows not only the position of the crack area in the object area estimated in the original image but also the location of the crack area in which object in the object area. It is also possible to estimate whether there are any animals present.

The embodiments described above are merely illustrative to facilitate understanding of the present invention, and are not intended to be interpreted as limiting the present invention. It goes without saying that the present invention can be modified and improved without departing from its spirit, and that the present invention includes equivalents thereof.

1 Information processing system 2 Unmanned aerial vehicle

Claims (10)

1. a crack estimation unit that estimates a crack area in one or more objects in a structure with respect to an original image in which the objects are reflected;
   a crack coloring unit that colors the crack area in the original image according to at least the estimated color around the crack area;
   an object area estimation unit that estimates an object area in which the object exists in the colored image;
   An information processing system characterized by:
2. further comprising a superimposing unit that specifies a crack region in the target object region by at least superimposing the estimated position of the target object region and the crack region;
   The information processing system according to claim 1, characterized in that:
3. The crack estimating unit re-performs the estimation of the crack area on the original image associated with the estimated object area, and identifies the crack area in the object area.
   The information processing system according to claim 1, characterized in that:
4. further comprising a crack shape analysis unit that performs a crack shape analysis on the crack region in the specified object region;
   The information processing system according to claim 2 or 3, characterized in that:
5. The crack estimation unit divides the original image into two or more divided images and estimates a crack area for each divided image.
   The information processing system according to any one of claims 1 to 3, characterized in that:
6. The crack coloring unit generates an enlarged area by enlarging the estimated crack area, and colors the crack area with a statistically determined color based on color information in the enlarged area.
   The information processing system according to any one of claims 1 to 3, characterized in that:
7. The crack coloring unit acquires color information of the object and colors the crack area with a color that matches or substantially matches the color information.
   The information processing system according to any one of claims 1 to 3, characterized in that:
8. A program that causes a computer having a processing unit to perform information processing,
   The program causes the processing unit to
   Estimating a crack area in one or more objects in a structure with respect to an original image in which the objects are reflected;
   Coloring the crack area in the original image according to at least the estimated color of the surrounding area of the crack area;
   estimating an object area where the object exists in the colored image;
   A program to run.
9. estimating a crack area in one or more objects in a structure with respect to an original image in which one or more objects in the structure are reflected, by a crack estimating unit;
   Coloring the crack area in the original image according to at least the estimated color of the surrounding area of the crack area using a crack coloring unit;
   estimating, by a target object area estimating unit, a target object area in which the target object exists in the colored image;
   An information processing method that executes on a computer.
10. a crack estimation unit that estimates a crack area in one or more objects in a structure with respect to an original image in which the objects are reflected;
    a crack coloring unit that colors the crack area in the original image according to at least the estimated color around the crack area;
    an object area estimation unit that estimates an object area in which the object exists in the colored image;
    A server characterized by:
    
    
    

Images