JP6647171B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to an information processing device, an information processing method, and a program.

  With the aging of roads, bridges, facilities, etc., efficient maintenance of buildings is required. In order to carry out maintenance of the building, it is necessary to find the defect of the building, but it is difficult to perform quantitative evaluation and comprehensive analysis by visual confirmation. Therefore, automatic detection by a machine is spreading. For example, Patent Literature 1 has a plurality of cameras mounted on a vehicle and arranged so that a part of a road width can be used as an imaging range, and the imaging ranges partially overlap each other so that the entire road width can be imaged. For each predetermined traveling distance of the vehicle, a sun photographing means for simultaneously and identifiably photographing a road surface image in the sun, mounted on the vehicle, a part of the road width is taken as a photographing range, and the photographing ranges of each other are partially A plurality of cameras arranged so as to be able to capture the entire width of the road while overlapping each other, and for each predetermined traveling distance of the vehicle, at the same time, shade image capturing means for identifying a road surface image in the shade in a distinguishable manner; A plurality of cameras in the shade imaging means and the overlapping region in the road width direction and the traveling direction of the vehicle on the road surface image corrected by the correction data; Editing means for calculating the degree of coincidence of the overlapping area in the road width direction and the traveling direction of the vehicle of the road surface image after correction, and editing the joint of the road surface image based on the degree of coincidence. Is described.

JP 2011-90367 A

  In some cases, a plurality of objects to be analyzed appear in a captured image. In such a case, if the captured image is analyzed as it is, an object to be analyzed may not be correctly classified. For this reason, the accuracy of analysis may not be improved.

  It is an object of some aspects of the present invention to provide an information processing device, an information processing method, and a program capable of accurately analyzing a state of a surface of a building or the like.

  Another embodiment of the present invention has an object to provide an information processing device, an information processing method, and a program that can achieve the effects described in the embodiments described below.

In order to solve the above-described problem, one embodiment of the present invention is an image in which an image is continuously captured by an imaging device mounted on a moving object that moves along a surface, and an image including a region including the surface is captured. An acquiring unit, a partial image generating unit that divides the image acquired by the image acquiring unit, and outputs a plurality of partial images that are a part of the image, and, for each of the partial images output by the partial image generating unit, An information processing apparatus comprising: a state determination unit that determines a state of a surface in an image.
Further, another aspect of the present invention is a method in which the information processing device is continuously imaged by an imaging device mounted on a moving body moving along a surface, and acquires an image in which a region including the surface is imaged. One step, a second step of dividing the image obtained in the first step, and outputting a plurality of partial images that are part of the image, and for each of the partial images output in the second step, And a third step of determining the state of the surface.
Further, another aspect of the present invention is a computer including: a first step of acquiring an image in which an image is continuously captured by an imaging device mounted on a moving object that moves along a surface, and a region including the surface is captured; And a second step of dividing the image obtained in the first step and outputting a plurality of partial images that are a part of the image; and for each of the partial images output in the second step, a surface in the partial image And a third step of determining the state of the program.

  According to an embodiment of the present invention, the state of a surface of a building or the like can be analyzed with high accuracy.

FIG. 1 is a diagram showing an outline of a failure analysis system 1 according to a first embodiment of the present invention. The figure showing the 1st example of the picture which picturized the road surface picture concerning the embodiment. The figure showing the 2nd example of the picture which picturized the road surface picture concerning the embodiment. The figure showing the 3rd example of the picture which picturized the road surface picture concerning the embodiment. The figure showing the 4th example of the picture which picturized the road surface picture concerning the embodiment. The figure showing the 5th example of the picture which picturized the road surface picture concerning the embodiment. FIG. 2 is a block diagram showing a configuration of a failure analysis system 1 according to the embodiment. The figure which shows the data structure of the road surface image information which concerns on the embodiment. The figure which shows the data structure of positioning information. The figure which shows the data structure of defect information. The figure which shows the data structure of ledger information. FIG. 2 is a block diagram illustrating a configuration of a failure analysis device. 9 is a flowchart showing the flow of the overall processing by the failure analysis device. 9 is a flowchart illustrating a flow of an image extraction process performed by the failure analysis device. The figure which shows an example of an inspection image. The figure which shows an example of an area extraction process. The figure which shows an example of the presentation screen of defect information. FIG. 9 is a diagram showing an outline of a first curve analysis method according to a second embodiment of the present invention. The figure which shows the outline | summary of the 2nd curve analysis technique. 9 is a flowchart illustrating a flow of an image extraction process performed by the failure analysis device. The figure which shows the area | region extraction process which concerns on a modification.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[First Embodiment]
[Overview of the failure analysis system 1]
A first embodiment of the present invention will be described.
FIG. 1 is a diagram showing an outline of the failure analysis system 1.
The failure analysis system 1 according to the present embodiment is a system that analyzes the state of the surface of a building. In the following, as an example, a case where the failure analysis system 1 analyzes a visible failure that appears on the surface of a building will be described.
The buildings are, for example, roads, bridges, facilities, and the like, and include social infrastructure.
A surface is an exterior or interior surface of a building. That is, the surface of the building is, for example, the road surface or bridge, the upper surface, the side surface, or the lower surface of the facility.
Visible defects are physical irregularities, color changes, and the like.

Hereinafter, as an example, a case in which a defect that appears on a road surface (hereinafter, referred to as “road surface defect”) is analyzed will be described.
Road surface defects include, for example, ruts, potholes, cracks, and steps. FIG. 2 to FIG. 6 are diagrams illustrating examples of images of road surface defects. The image P1 shown in FIG. 2 shows an example of the rut D1. The image P2 shown in FIG. 3 shows an example of the pothole D2. An image P3 shown in FIG. 4 shows an example of a crack D3. An image P4 shown in FIG. 5 shows an example of the step D4. In addition, the failure analysis system 1 may analyze the state of the road surface such as patching in addition to the road surface failure. An image P5 shown in FIG. 6 shows an example of patching D5.

Next, an outline of analysis of a road surface defect using the defect analysis system 1 will be described.
First, the user drives the vehicle 10 including the imaging unit 11 to travel on a road for analyzing a road surface problem. The imaging unit 11 is installed in the vehicle 10 so that a road surface can be imaged. Then, the imaging unit 11 images a road surface changing as the vehicle 10 travels as needed. Hereinafter, the road surface image is referred to as a “road surface image”. In addition, the user measures the position of the vehicle 10 as needed during the imaging of the road surface image. Hereinafter, information (data) indicating the positioning result is referred to as “positioning information”.

  Next, the failure analysis system 1 analyzes information indicating the road surface image (hereinafter, referred to as “road surface image information”) using the artificial intelligence for failure analysis. The artificial intelligence for failure analysis includes, for example, a multilayer perceptron (multilayer neural network). In the artificial intelligence for failure analysis, deep learning (deep learning) is performed in advance using a teaching image (for example, images P1 to P5) that captures an analysis target such as a road surface failure. Any technology or framework that has been used in the conventional artificial intelligence may be applied to the artificial intelligence for failure analysis. By the deep learning, the defect analysis system 1 can detect the road surface defect reflected in the road surface image by using the defect analysis artificial intelligence. Further, the failure analysis system 1 can identify the location where the failure exists by specifying the positioning result at the time of capturing the road surface image where the road surface failure is detected.

The failure analysis system 1 associates and outputs the detected road surface defect and the position of the road surface defect as a result of the analysis of the road surface image information. The defect analysis system 1 shows, for example, the location of the road surface defect on a map and shows the type of the road surface defect in association with the location. The analysis result may be editable by a user.
As described above, the defect analysis system 1 detects a defect reflected in an image using the artificial intelligence that has undergone deep learning. The deep learning has a characteristic that high learning accuracy can be obtained if learning is performed with appropriate teaching information. Therefore, the failure analysis system 1 can comprehensively detect the failure of the building in the imaged range.

Further, the failure analysis system 1 does not need to include special measuring devices such as a laser irradiator, an electronic streak imaging device, and an acceleration sensor, unlike a conventional road surface property measuring vehicle. Therefore, the failure analysis system 1 can accurately and inexpensively detect the failure of the building. The failure analysis system 1 may include a laser irradiator, an electronic streak imaging device, an acceleration sensor, and the like.
The above is the description of the outline of the failure analysis system 1.

[Configuration of the failure analysis system 1]
Next, the configuration of the failure analysis system 1 will be described.
FIG. 7 is a block diagram illustrating a configuration of the failure analysis system 1.
The failure analysis system 1 includes a vehicle 10, a failure analysis device 30, a failure information management device 50, a ledger information management device 70, and a map information management device 90. The vehicle 10 includes an imaging unit 11, a positioning unit 12, and an information terminal device 13. The information terminal device 13, the defect analysis device 30, the defect information management device 50, the ledger information management device 70, and the map information management device 90 can connect to the network NW and communicate with each other. .

  The vehicle 10 is a moving body that moves so as to be able to image a road surface. For example, the vehicle 10 is an automobile, a bicycle, a train, or the like, and is a moving body that moves on a building. Note that, instead of a moving body that moves while contacting a building such as the vehicle 10, a moving body that does not contact the building such as a radio control helicopter or a drone may be used.

  The imaging unit 11 captures an image of a road surface and generates road surface image information. The imaging unit 11 may capture a moving image (video) or may continuously capture still images. Here, as an example, a case where the frame rate of a moving image or the imaging interval of a still image is constant will be described. The road surface image information generated by the imaging unit 11 may be output to another device via a storage medium, or may be transmitted to another device via a network. The imaging unit 11 may be installed inside the vehicle 10. In such a case, the imaging unit 11 may image the road surface through a windshield or a rear glass of the vehicle 10. A polarizing filter may be attached to the lens of the imaging unit 11 in order to suppress the reflection by the glass from being reflected in the road surface image.

  The positioning unit 12 performs positioning. The positioning unit 12 is, for example, a GPS (Global Positioning System) sensor. The positioning unit 12 may perform positioning using a geostationary satellite-type satellite navigation reinforcement system, a speed sensor such as ODB2 (On-board diagnostics 2), an acceleration sensor, or the like. The positioning unit 12 may perform positioning at a predetermined time interval, or may perform positioning in synchronization with acquisition of a road surface image by the imaging unit 11. If the acquisition of the road surface image by the imaging unit 11 is not synchronized, the start of positioning and the start of imaging are matched. Then, for the difference between the timing of imaging and the timing of positioning, the position corresponding to each road surface image may be specified by dividing the movement of two positioning points by the number of road surface images therebetween.

  The information terminal device 13 is an electronic device including a computer system such as a vehicle-mounted terminal such as a car navigation terminal, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant) terminal, a personal computer, and a wearable terminal. The information terminal device 13 receives information from another device connected to the network NW, and displays the content of the received information or outputs sound. For example, the information terminal device 13 receives the information indicating the analysis result of the road surface defect and displays the analysis result. The information terminal device 13 may include an input unit such as a touch panel, mechanical buttons, a keyboard, a microphone, and the like. In this case, the input unit may transmit the content of the user input information received from the user to another device connected to the network NW.

  The imaging unit 11, the positioning unit 12, and the information terminal device 13 may be an integrated device or may be separate devices. For example, one smartphone may perform all of imaging of a road surface image, positioning by GPS, and displaying the analysis result of a road surface defect. Further, the imaging unit 11, the positioning unit 12, and the information terminal device 13 may be operated by voice input. Thereby, even while the vehicle 10 is running, the driver of the vehicle 10 can operate various devices by itself. Further, the imaging unit 11, the positioning unit 12, and the information terminal device 13 may be remotely operated via a network NW, or may be operated via short-range wireless communication such as infrared rays.

  The failure analysis device 30 is a device that analyzes a road surface defect from a road surface image. The defect information management device 50 is, for example, an electronic device including a computer system such as a personal computer and a server device. The failure analysis device 30 is equipped with a failure analysis artificial intelligence.

The defect information management device 50 is a device that manages defect information. Here, "information management" refers to writing, editing, deleting, reading, receiving, and transmitting information newly. The defect information management device 50 is an electronic device including a computer system such as a server device, for example.
The defect information is information on the road surface defect such as the content of the road surface defect and the location of the road surface defect. That is, the defect information is an example of information indicating the state of the surface. The defect information may describe that there is no road surface defect.

The defect information management device 50 includes a first type defect information storage unit 51 and a second type defect information storage unit 52. The first type defect information storage unit 51 stores the first type defect information. The second type defect information storage unit 52 stores the second type defect information.
The first type defect information is information on road surface defects confirmed manually. The first type defect information is used for machine learning of the artificial intelligence for defect analysis.
The second type of defect information is information relating to a road surface defect detected by the defect analysis artificial intelligence. That is, the second type defect information is information indicating an analysis result by the defect analysis device 30.

The ledger information management device 70 is a device that manages road ledger information. The ledger information management device 70 is an electronic device including a computer system such as a server device, for example. The ledger information management device 70 includes a ledger information storage unit 71 that stores road ledger information.
The map information management device 90 is a device that manages map information. The ledger information management device 70 is an electronic device including a computer system such as a server device, for example. The map information management device 90 includes a map information storage unit 91 that stores map information. The map information is information including a name, coordinates, an address, a range, and the like of a building, and is information used for generating a map.
The above is the description of the configuration of the failure analysis system 1.

[Data structure of information]
Next, a data configuration of information processed by the failure analysis system 1 will be described.
First, the road surface image information will be described. Here, a case where the road surface image information is a moving image will be described as an example.
FIG. 8 is a diagram illustrating a data configuration of road surface image information.
In the example illustrated in FIG. 8, the road surface image information includes frame number information (“frame number” in FIG. 8), frame time information (“frame time” in FIG. 8), and frame image information (“frame image” in FIG. 8). )), And these are associated with each other.
The frame number information is identification information of each frame forming a moving image of the road surface image. The frame number information is, for example, a number assigned to each frame from the start to the end of the moving image in ascending order.

The frame time information is information indicating the timing of displaying each frame when a moving image is reproduced.
The frame image information is information indicating an image of each frame. The frame image may be encoded.
The above is the description of the data configuration of the road surface image information.

Next, positioning information will be described.
FIG. 9 is a diagram illustrating a data configuration of the positioning information.
In the example illustrated in FIG. 9, the positioning information includes time stamp information (“time stamp” in FIG. 9) and positioning coordinate information (“coordinates” in FIG. 9), and is configured by associating them.
The time stamp information is information indicating the time at which positioning was performed.
The positioning coordinate information is information indicating a position specified as a result of positioning.
The above is the description of the data configuration of the positioning information.

Next, the defect information will be described.
FIG. 10 is a diagram illustrating a data configuration of the defect information.
In the example shown in FIG. 10, the defect information includes a road defect ID (“road defect ID” in FIG. 10), address information (“address” in FIG. 10), and defect coordinate information (“coordinate” in FIG. 10). , Road surface defect type information (“road surface defect type” in FIG. 10) and defect image information (“image” in FIG. 10), and are configured to correspond to each other.

The road surface defect ID (Identifier) is identification information for individually identifying a road surface defect detected by the defect analysis device 30 or manually confirmed.
The address information is information indicating an address of a place where a road surface defect exists.
The defect coordinate information is information indicating coordinates of a place where a road surface defect exists.
The road surface defect type information is information indicating the type of road surface defect.
The defect image information is information indicating an image of a road surface defect.
The above is the description of the data configuration of the defect information.

Next, ledger information will be described.
FIG. 11 is a diagram showing a data structure of ledger information.
In the example illustrated in FIG. 11, the ledger information includes route number information (“route number” in FIG. 11), road type information (“road type” in FIG. 11), and route name information (“route name” in FIG. 11). ), Section number information (“section number” in FIG. 11), improved section information (“improved section” in FIG. 11), road surface type information (“road surface type” in FIG. 11), and full width information (FIG. 11 "wide width") and road width information ("road width" in FIG. 11), and are configured to correspond to each other.

The line number information is identification information for identifying a road line.
The road type information is information indicating the type of the road.
The route name information is information indicating the name of the route of the road.
Section number information is information indicating a section of a road on a route.
The improvement section information is information indicating an improvement section of the road.
The road surface type information is information indicating the type of the road line.
The full width information is information indicating the full width of the road.
The road width information is information indicating the width of the road.
The above is the description of the data structure of ledger information.

[Configuration of the failure analysis device 30]
Next, the configuration of the failure analysis device 30 will be described.
FIG. 12 is a block diagram illustrating the configuration of the failure analysis device 30.
In the example illustrated in FIG. 12, the failure analysis device 30 includes an input unit 31, a display unit 32, a sound reproduction unit 33, a communication unit 34, a storage unit 35, and a control unit 360.
The input unit 31 receives input of various information from the user of the failure analysis device 30. The input unit 31 includes, for example, a pointing device such as a mouse and a touch pad, a keyboard, and the like.
The display unit 32 displays various information such as texts and images. The display unit 32 includes, for example, an organic EL (Electro-Luminescence) display panel, a liquid crystal display panel, and the like. The analysis result of the road surface failure by the failure analysis device 30 is displayed.
The audio reproduction unit 33 reproduces audio. The sound reproducing unit 33 is, for example, a speaker, an amplifier, or the like.
The communication unit 34 transmits and receives information to and from other devices. The communication unit 34 includes, for example, a communication IC (Integrated Circuit), a communication port, and the like.
The storage unit 35 stores various information. The storage unit 35 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. In addition, the storage unit 35 may include an HDD (Hard Disc Drive), an EEPROM (Electrically Erasable Programmable Read Only Memory), a flash memory, and the like. The storage unit 35 stores various programs to be executed by a CPU (Central Processing Unit, not shown) and a GPU (Graphics Processing Unit, not shown) included in the failure analysis device 30 and results of processing executed by the CPU and the GPU. I do. The storage unit 35 includes a learning result storage unit 351.
The learning result storage unit 351 stores the learning result of the failure analysis artificial intelligence provided in the failure analysis device 30. The learning result is, for example, information indicating the state of the neural network. The information indicating the state of the network of the neural network is, for example, information indicating a value of a parameter of an activation function related to each node (unit) configuring the neural network.

  The control unit 360 is realized by, for example, executing a program stored in the storage unit 35 in advance by the CPU or GPU included in the failure analysis device 30. A part or the whole of the control unit 360 may be realized as an integrated circuit of hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit). The control section 360 includes an information acquisition section (image acquisition section) 361, an image conversion section 362, an analysis section 365, a result generation section 368, and a result editing section 369.

  The information acquisition unit 361 acquires various types of information necessary for analyzing a road surface defect via the input unit 31 and the communication unit 34. The information acquisition unit 361 reads, for example, road surface image information and positioning information from a storage medium. The information acquisition unit 361 acquires, for example, road surface image information, positioning information, and user input information from the information terminal device 13. The information acquiring unit 361 acquires, for example, defect information from the defect information management device 50. The information acquiring unit 361 acquires, for example, ledger information from the ledger information management device 70. The information acquiring unit 361 outputs the acquired various information to the image converting unit 362, the analyzing unit 365, and the result editing unit 369.

  The image conversion unit 362 acquires the road surface image information and the positioning information from the information acquisition unit 361. The image conversion unit 362 converts the acquired road surface image information. Further, the image conversion unit 362 associates each frame of the road surface image with the positioning coordinate information based on the frame time information of the road surface image information and the time stamp of the positioning information. Note that, for example, when the road surface image is an encoded moving image, the image conversion unit 362 may convert each frame into a still image. Further, the image conversion unit 362 includes an image extraction unit 363 and an area extraction unit (partial image generation unit) 364.

The image extracting unit 363 performs an image extracting process.
The image extraction process is a process of extracting an image to be analyzed for the presence or absence of a road surface defect (hereinafter, referred to as an “inspection image”) from the road surface image information. In other words, the image extraction process is a process of specifying, extracting, extracting, or narrowing down an inspection image from a series of images captured by the imaging unit 11. The image extraction process is a process of identifying, extracting, extracting, or thinning out an image for which the presence or absence of a road surface defect is not analyzed from a series of images captured by the imaging unit 11.

  The image conversion unit 362 extracts an inspection image based on the positioning information so that temporal overlap of the imaging ranges becomes necessary and sufficient. For example, when the imaging interval of each frame of the road surface image information is short and the moving speed of the vehicle 10 is low, the same portion of the road surface is repeatedly imaged over a plurality of frames. When the presence / absence of a road surface defect is analyzed for each of the temporally overlapping frames as described above, detection omission of the road surface defect can be reduced, but the load required for the analysis increases.

  Therefore, the image extracting unit 363 extracts the inspection image based on the positioning information so that the same portion of the road surface is included at a predetermined frequency in the inspection regions of the plurality of frames. The inspection area is, for example, an area in the image to be analyzed for the presence or absence of a road surface defect. In the extraction of the inspection image, the frequency at which the same portion of the road surface is included may be determined based on the positioning accuracy of the positioning unit 12 or the recognition accuracy of the fault analysis artificial intelligence. For example, when the positioning accuracy or the recognition accuracy is high, the frequency may be low, and when the positioning accuracy or the recognition accuracy is low, the frequency may be high.

By extracting the inspection image in this way, the failure analysis device 30 can reduce the load required for the analysis while maintaining the recognition accuracy of the road surface failure. Further, when the inspection images are displayed continuously, the failure analysis device 30 can display the road surface image as if the road surface image was captured while the vehicle 10 was running at a constant distance, that is, at a constant speed. Therefore, the failure analysis device 30 can easily confirm the road surface defect in the road surface image even after the road surface image is captured.
The image extracting unit 363 outputs information of the inspection image extracted by the image extracting process to the region extracting unit 364.

The region extracting unit 364 acquires information on the detected image from the image extracting unit 363. The region extraction unit 364 performs a region extraction process on the detected image.
The region extraction process is a process of extracting an inspection region from an inspection image. In other words, the region extraction process is a process of specifying, extracting, extracting, narrowing down, generating, or dividing the inspection image from the inspection image, and dividing the inspection image. The region extraction process is a process of specifying, extracting, extracting, or thinning out a region for which the presence or absence of a road surface defect is not analyzed from an inspection image. An object other than the road surface may appear in the inspection image. For example, a house beside a road, another vehicle on a road, or the like may appear in the inspection image. If a road surface defect is analyzed for objects other than the road surface, an object other than the road surface may be erroneously recognized as a road surface defect, and the recognition accuracy may be reduced. In particular, if the machine learning for objects other than the road surface is insufficient, such erroneous recognition is likely to occur. In addition, if a road surface defect is analyzed for objects other than the road surface, the load required for analyzing the road surface defect increases.

Therefore, in the region extraction processing, a road surface region is extracted from a test image as a test region. In the region extraction processing, a region outside the road surface may be excluded. Further, in the region extraction processing, a plurality of overlapping regions may be extracted from one image. When a plurality of regions overlap, a vertex portion or a side portion of a certain region is included in another region.
The region extraction unit 364 outputs information on the inspection region extracted by the region extraction processing to the analysis unit 365. Further, the region extracting unit 364 outputs information indicating the position of each inspection region in the inspection image to the analysis unit 365.

The analysis unit 365 is implemented as a failure analysis artificial intelligence. The analysis unit 365 includes a learning unit 366 and a recognition unit 367.
The learning unit 366 executes machine learning using the first type defect information managed by the defect information management device 50. The learning unit 366 may perform the reinforcement learning using an image of an object other than a road surface defect that may appear in a road surface image, in addition to the first type defect information. For example, reinforcement learning may be performed using an image of a manhole, an image of a gutter, or the like. Accordingly, the fault analysis artificial intelligence can identify an object other than the road surface defect, and thus it is difficult to erroneously recognize an object other than the road surface defect as a road surface defect. Therefore, the recognition accuracy of the road surface defect can be improved.

  The recognition unit (state determination unit) 367 analyzes the presence or absence of a road surface defect in each inspection region based on the information on the inspection region acquired from the region extraction unit 364. That is, the recognition unit 367 determines the state of the road surface in each inspection area. The recognizing unit 367 may determine each of the road surface defects detected in each inspection area as one road surface defect, or collectively determine the road surface defects detected in a plurality of adjacent inspection regions as one road surface defect. May be. The recognizing unit 367 may determine each of the road surface defects detected in each inspection image as one road surface defect, or collectively determine the road surface defects detected in successive inspection images as one road surface defect. May be. As described above, the recognition unit 367 may perform the determination by integrating the determination results of the plurality of inspection regions and the plurality of inspection images.

  In addition, the recognition unit 367 can specify the position of the road surface defect by specifying the positioning information corresponding to the inspection image in which the road surface defect has been detected. In addition, the recognition unit 367 can more accurately specify the position of the road surface defect by acquiring information indicating the position of the inspection region in the inspection image from the region extraction unit 364. In addition, the failure analysis device 30 combines a road surface failure analysis result using other methods such as a road surface irregularity detection result based on a change in acceleration or a road surface failure analysis result by another image analysis program. May be analyzed.

  The recognizing unit 367 transmits the analysis result to the defect information management device 50. As a result, the analysis result is stored in the defect information management device 50 as the second type defect information. The analysis result includes, for example, an inspection image in which a road surface defect is detected, an inspection region, a type of the road surface defect, a determination score (certainty factor), and the like. The analysis unit 365 specifies the degree and range of occurrence of the road surface defect by referring to the value of the determination score, the analysis result of the road surface defect in the front and rear road surface images, the analysis result of the road surface defect in a plurality of inspection areas of the same road surface image, and the like. can do. For example, when there are a plurality of types of road surface defect types, the analysis unit 365 may set the type having the highest determination score, or may set all types.

  In the case where the determination result is different in the overlapping inspection area, the analysis unit 365 may employ any of the determination results. For example, when the analysis unit 365 determines that “there is a road surface defect” in one inspection region and determines that “there is no road surface defect” in the other inspection region, the analysis unit 365 performs OR processing and determines that “there is a road surface defect”. Alternatively, it may be determined that “there is no road surface defect” by performing an AND process. When the OR process is performed and it is determined that “there is a road surface defect”, the defect analysis device 30 can prevent omission of detection of the road surface defect. When it is determined that “there is no road surface defect” by performing the AND process, erroneous detection of the road surface defect can be suppressed.

  The result generation unit 368 generates presentation information such as text, an image, and sound indicating the first type defect information and the second type defect information (that is, the analysis result by the analysis unit 365). The result generation unit 368 generates, for example, a moving image to which defect information is added or a moving image that reproduces an inspection image in frame number order. A specific example of the presentation information screen generated by the result generation unit 368 will be described later. The result generation unit 368 presents the generated presentation information via the display unit 32 and the audio reproduction unit 33. At this time, the result generation unit 368 may generate presentation information with reference to arbitrary information managed by the defect information management device 50, the ledger information management device 70, and the map information management device 90.

The result editing unit 369 edits the defect information according to an instruction from the user. For example, the result editing unit 369 may edit the second type defect information stored in the defect information management device 50 according to an instruction from a user. Further, for example, the result editing unit 369 may store the second type defect information stored in the defect information management device 50 as the first type defect information according to an instruction from the user. Thus, the image of the road surface defect detected by the analysis unit 365 can be used for learning the defect artificial intelligence. Note that the result editing unit 369 may perform the above-described processing according to a result output by another program for detecting a road surface defect or another program used for road surface monitoring, instead of an instruction from the user.
The above is the description of the configuration of the failure analysis device 30.

[Operation of Failure Analysis Device 30]
Next, the operation of the failure analysis device 30 will be described.
FIG. 13 is a flowchart showing the flow of the overall processing performed by the failure analysis device 30.
(Step S100) The information acquisition unit 361 acquires road surface image information and positioning information. Thereafter, control unit 360 proceeds with the process to step S102.
(Step S102) The image conversion unit 362 performs an image extraction process to extract an inspection image. After that, the control section 360 advances the process to step S104.

(Step S104) The image conversion unit 362 performs an area extraction process to extract an inspection area. After that, the control unit 360 advances the process to Step S106.
(Step S106) The analysis unit 365 analyzes a road surface defect included in each inspection region of each inspection image. The analysis unit 365 causes the defect information management device 50 to store the defect information indicating the analysis result as the second type defect information. Thereafter, control unit 360 proceeds with the process to step S108.
(Step S108) The result generation unit 368 generates presentation information for presenting the user with the defect information managed by the defect information management device 50. The result generation unit 368 outputs the generated presentation information to the display unit 32 and the audio reproduction unit 33. After that, the control unit 360 ends the processing illustrated in FIG.
Note that the processing of steps S102 and S104 can be omitted.
The above is the description of the overall processing flow by the failure analysis device 30.

(Image extraction processing)
Next, an image extraction process by the failure analysis device 30 will be described.
Here, as an example, a case where a road surface image corresponding to a movement of a predetermined set distance is extracted from a series of road surface images that change according to the movement of the vehicle 10 will be described.
FIG. 14 is a flowchart illustrating the flow of the image extraction process performed by the failure analysis device 30.
(Step S200) The image extracting unit 363 selects a frame (image) based on the frame ID. For example, when a frame has not been selected, the image extracting unit 363 selects the first frame. If a frame has already been selected, the image extracting unit 363 selects the next frame after the selected frame. After that, the control unit 360 advances the process to step S202.

(Step S202) The image extracting unit 363 refers to the positioning information corresponding to the selected frame and calculates the moving distance from the positioning result corresponding to the immediately preceding frame. Hereinafter, this moving distance may be referred to as “distance between images”. After that, the control section 360 advances the process to step S204.
(Step S204) The image extracting unit 363 adds the calculated moving distance to the total moving distance. Note that the total moving distance is initialized at the start of the image extraction processing. After that, the control section 360 advances the process to step S206.

(Step S206) The image extracting unit 363 compares the total moving distance with a predetermined set distance. If the total moving distance is larger than the set distance (step S206; YES), control unit 360 proceeds to step S208. When the total moving distance is smaller than the set distance (Step S206; NO), the control unit 360 advances the process to Step S214.

  Here, the set distance may be set, for example, according to the recognition accuracy of a road surface defect in one inspection image. For example, when it is possible to detect a road surface defect not only in the vicinity but also in a distance in one inspection image, the set distance may be increased and the overlap of the road surface in each inspection image may be reduced. Further, if a road surface defect that is distant can not be detected in one inspection image, the set distance may be reduced, and the overlap of the road surface in each inspection image may be increased. This makes it possible to maintain the recognition accuracy of the road surface defect while reducing the load required for analyzing the road surface defect.

(Step S208) The image extracting unit 363 sets the total moving distance to 0 and initializes it. Thereafter, control unit 360 proceeds with the process to step S210.
(Step S210) The image extracting unit 363 associates the currently selected image with the positioning information. After that, the control section 360 advances the process to step S212.
(Step S212) The image extracting unit 363 sets the selected image as the inspection image. After that, the control unit 360 advances the process to step S214.

(Step S214) The image extracting unit 363 checks the presence / absence of the next image and the presence / absence of the positioning result corresponding to the next image. When the image and the positioning result are present (Step S214; YES), the control unit 360 advances the process to Step S216. When the image and the positioning result do not exist (Step S214; NO), the control unit 360 advances the process to Step S216.
(Step S216) The image extracting unit 363 stores the inspection image and the positioning information corresponding to the inspection image in the storage unit 35, and makes the reference possible. After that, the control unit 360 ends the processing illustrated in FIG.
Note that the detection image may be extracted regardless of the set distance, for example, at intervals of 10 frames. Although omitted in the processing illustrated in FIG. 14, the positioning information may be corrected based on speed information of the vehicle 10 or the like.
The above is the description of the image extraction processing.

[Area extraction processing]
Next, the region extraction processing will be described.
FIG. 15 is a diagram illustrating an example of the inspection image.
In the example illustrated in FIG. 15, the imaging unit 11 is installed so as to capture an image in front of the vehicle 10. The inspection image P6 captured by the imaging unit 11 includes a road surface of a two-lane road divided by three lines. In addition, the inspection image P6 includes a roadside sectioned by the curb R4.

  Here, in the inspection image P6, the roadway is located below the vehicle 10 and the space above the vehicle 10 is open. Since the object near the imaging surface of the imaging unit 11 is large, the lower side (front side) of the inspection image P6, that is, the road surface close to the vehicle 10 is large, and the center part (depth side) of the inspection image P6, that is, from the vehicle 10 A distant road surface appears small. That is, the lower road surface of the inspection image P6 is imaged with a larger number of pixels even though the road surface is actually the same area. Therefore, even if the road surface defect has the same area, the lower road surface defect can be recognized with higher accuracy between the lower part and the central part of the inspection image P6. Further, in the imaging of the inspection image P6, even if another vehicle is present in the vicinity of the vehicle 10, the image of the inspection image P6 is very close to the vehicle 10 below the inspection image P6. The road surface will be reflected more reliably than before. When the vehicle 10 makes a right or left turn, the amount of change in the imaging range is smaller at the lower side than at the center of the inspection image P6. Therefore, when the vehicle 10 makes a right or left turn, a part that is not analyzed may occur at the center of the inspection image P6. In addition, since the vehicle 10 basically travels near the center of the lane, when the imaging unit 11 is set to capture the traveling direction of the vehicle 10, particularly in the lateral direction of the inspection image P6, Roads other than the road surface, road shoulders, and the like are easily reflected at the ends.

FIG. 16 is a diagram illustrating an example of the region extraction process.
The inspection image P7 shown in FIG. 16 is an image captured similarly to the inspection image P6 shown in FIG. Areas A1 to A10 shown in FIG. 16 each indicate an inspection area set in the inspection image P7. By setting the inspection areas A1 to A10 in this way, each of the inspection areas A1 to A10 is unlikely to include an object other than the road surface. This reduces the possibility of erroneously recognizing an object other than a road surface defect as a road surface defect, so that the defect analysis device 30 can improve the recognition accuracy of a road surface defect. Further, the position of the road surface defect in the image can be specified. Further, even when a plurality of road surface defects are shown in the inspection image P7, each of the inspection areas A1 to A10 is likely to include only individual road surface defects. Thereby, the failure analysis device 30 can accurately recognize the type of each road surface failure.

Further, as described above, between the lower side and the central part of the inspection image P7, a road surface defect that is photographed on the lower side can be recognized with higher accuracy. Therefore, in the vertical direction of the inspection image P7, more inspection areas are set below the center. Further, as described above, in the vertical direction of the inspection image P7, the road surface appears below the center side. As described above, in the vertical direction of the inspection image P7, the change in the imaging range when turning right or left is smaller on the lower side than on the center side. Therefore, by setting more inspection areas on the lower side, road surface defects can be recognized with high accuracy. Further, as described above, along the lateral direction of the inspection image P7, roadsides other than the road surface and the like are likely to be reflected at the end of the image. Therefore, in the lateral direction of the center of the inspection image P7, more inspection areas are set near the center than at the ends. In addition, the inspection areas A1 to A10 are set so that a part thereof overlaps each other. As a result, even when the road surface defect is included in the boundary or the end of the inspection region, the road surface defect can be included in the vicinity of the center in the inspection region set so as to overlap the inspection region. Therefore, the failure analysis device 30 can suppress detection omission of a road surface failure and improve recognition accuracy.
The above is the description of the region extraction processing.

[Presentation of defect information]
Next, the presentation of the defect information will be described.
The defect information may be presented by the defect analysis device 30 or the information terminal device 13.
FIG. 17 is a diagram illustrating an example of a screen for presenting the defect information.
In the example shown in FIG. 17, the screen O1 includes a plurality of regions O2 to O4. The screen O1 is converted into a web page, for example, and can be referred to from a web browser of an arbitrary terminal.
The region O2 is a region where a marker O21 indicating the position of the vehicle 10, a marker O22 indicating the position of the road surface defect, and a position where the road surface image is acquired, that is, an image in which the traveling route O23 of the vehicle 10 is displayed on a map are presented. . The marker O21 of the vehicle 10, the scale of the map, and the range of the map can be changed according to the operation by the user. The range of the map may be determined according to the position of the marker O21 of the vehicle 10, the position of the marker O22 of the road surface defect, and the traveling route O23 of the vehicle 10.

  Further, the road surface defect markers O22 may be put together according to the scale of the map or the density of the road surface defects. For example, the defect analysis device 30 may combine the road surface defects detected from a plurality of inspection regions of one inspection image, the road surface defects detected from a plurality of continuous inspection images, and the like into one marker O22. The assembled markers O22 may be individually confirmed according to the selection from the user. In addition, the road surface defect marker O22 has different display modes (design, color, shape, size, transparency, etc.) depending on the type and degree of the road surface defect, the occurrence range, and whether it is the first type defect information or the second type defect information. ).

The area O3 is an area for displaying a road surface image. The area O3 continuously displays the inspection images extracted by the image extraction processing. When the frequency of overlap between the images is constant in the image extraction processing, the moving image reproduced in the region O3 is expressed as if it were captured at a constant traveling speed. Therefore, the user can efficiently confirm the road surface trouble.
The region O4 is a region for displaying a list of frames of the road surface image including the detected road surface defect. In the area O4, an inspection area O41 in which a road surface defect is detected in each frame is displayed.

  The display of the regions O2 to O4 may be associated with each other. For example, a road surface image captured at the position O21 of the vehicle 10 in the region O2 may be displayed in the region O3. Further, a road surface image including a road surface defect located at a predetermined distance from the position O21 of the vehicle 10 may be displayed in the region O4. For example, in the area O2, the display of the area O3 or the display of the area O4 may be switched according to the change of the position of the marker O21 of the vehicle 10.

  In addition, any information other than the above may be displayed on the presentation screen O1. For example, when the road surface defect marker O22 is selected, the defect analysis device 30 may display various kinds of information such as the position, the road surface defect type, the imaging time, the route number, the road construction history, and the like for the road surface defect. . Further, the failure analysis device 30 may display a numerical value such as the total number of road surface defects or the average of road surface defects for each traveling distance on the presentation screen O1. In addition, the failure analysis device 30 analyzes the road surface failure using the imaging time, the weather, and other methods (the detection result of the road surface unevenness based on the change in acceleration, the analysis result of the road surface failure by another image analysis program), and the like. May be displayed.

In addition, the information terminal device 13 and the failure analysis device 30 may receive correction of the failure information by the user via the presentation screen O1. For example, when the marker O22 of the road surface defect is selected by the user, the deletion of the marker O22 or the change of the type of the road surface defect may be accepted. Further, when the road surface image displayed in the region O3 and the region in the road surface image are selected, the failure analysis device 30 determines the selected region and the road surface image and the positioning information corresponding to the road surface image. May be generated based on the above. Then, the generated defect information may be stored in the defect information management device 50 as the first type defect information.
The above is the description of the presentation of the defect information.

[Summary of First Embodiment]
As described above, the defect analysis device 30 performs the learning for performing the deep learning on the image obtained by capturing the surface (for example, the road surface) based on the teaching data (for example, the first type defect information) indicating the state of the surface. A unit 366 and an information acquisition unit 361 for acquiring an image (for example, a road surface image) in which a region including a surface is captured and associated with position information (for example, positioning information) indicating the captured position. And a recognizing unit 367 that determines the state of the surface in the image acquired by the information acquiring unit 361 based on the learning result of the learning unit 366.
Thus, the failure analysis device 30 can comprehensively analyze the state of each location on the imaged surface.

Further, in the failure analysis device 30, the recognizing unit 367 integrates the determination results of the state of the plane in each of the plurality of partial images output from one image, and determines the state of the plane.
Thus, the failure analysis device 30 determines the state of each location on the surface with reference to the surrounding state. Therefore, the failure analysis device 30 can improve the accuracy of determining the state of the surface.

Further, in the failure analysis device 30, the recognizing unit 367 integrates the determination results of the state of the surface in each of the plurality of continuously captured images acquired by the information acquiring unit 361, and determines the state of the surface. judge.
Thus, the failure analysis device 30 determines the state of each location on the surface with reference to the surrounding state. Therefore, the failure analysis device 30 can improve the accuracy of determining the state of the surface.

Further, in the defect analysis device 30, for example, the teaching data (for example, the first type defect information) is data obtained by determining the determination result by the recognition unit 367 based on an input from a user or a determination result by another device. It is.
Thus, when the user confirms the validity of the determination result of the failure analysis device 30, the image used for the determination can be used for deep learning. For example, even when the determination result is incorrect, the failure analysis device 30 can perform re-learning, so that a more correct determination result can be displayed. Therefore, the failure analysis device 30 can improve the accuracy of determining the state of the surface.

Further, the failure analysis device 30 is continuously imaged by the imaging unit 11 mounted on a moving body (for example, the vehicle 10) moving along a surface (for example, a road surface), and an image in which a region including the surface is imaged. The first image is thinned out from an image (for example, a road surface image) that is continuously captured based on the information acquisition unit 361 that obtains the remaining information and the movement information (for example, positioning information) of the moving body, so that the remaining An image extracting unit 363 for extracting two images (for example, inspection images), and a recognizing unit 367 for determining a state of a surface in the second image for each of the second images.
Thereby, the failure analysis device 30 adjusts the amount of the image for determining the state of the surface according to the movement of the moving body. Therefore, the failure analysis device 30 can reduce the load required for the determination while maintaining the determination accuracy of the surface state.

Further, in the failure analysis device 30, the image extracting unit 363 extracts the second image (for example, an inspection image) based on a moving distance of the moving body (for example, the vehicle 10) with respect to time.
Thereby, the failure analysis device 30 adjusts the amount of the image for determining the state of the surface according to the moving speed of the moving object. Therefore, the failure analysis device 30 can reduce the load required for the determination while maintaining the determination accuracy of the surface state.

Further, the failure analysis device 30 obtains an image in which an image including a region including the surface is continuously captured by the imaging unit 11 mounted on a moving body (vehicle 10) moving along a surface (for example, a road surface). Information extraction unit 361, an area extraction unit 364 that outputs a plurality of partial images that are a part of the image acquired by the information acquisition unit 361, and each of the partial images output by the area extraction unit 364, And a recognizing unit 367 for determining the state of the surface.
Thus, the failure analysis device 30 determines the state of the surface from a part of the image. Therefore, when an element that reduces the determination accuracy appears in the image, the failure analysis device 30 can perform the determination by excluding the element. Therefore, the failure analysis device 30 can improve the accuracy of determining the state of the surface.

Further, in the failure analysis device 30, the area extracting unit 364 converts the partial image (for example, the lower inspection area A1 in FIG. 16) in front of the moving body (for example, the vehicle 10) with respect to the traveling direction. More than the partial image at the back (for example, the inspection area A10 at the center in FIG. 16) is output.
Thereby, the failure analysis device 30 focuses on the image to determine the state of the surface on the near side where the amount of information is abundant. Further, since the vicinity of the moving object is photographed on the near side of the image, objects other than the surface are unlikely to be photographed. On the near side of the image, the amount of change in the imaging range when the moving direction of the moving object changes is small. Therefore, the failure analysis device 30 can improve the determination accuracy of the surface state and reduce the load required for the determination.

Further, in the failure analysis device 30, the region extracting unit 364 divides an image (for example, an inspection image), and divides the divided first partial image (for example, the inspection region A1 in FIG. 16) and the first partial image. A second partial image including a vertex portion or a side portion (for example, an area A5 in FIG. 16) is output.
Thereby, the failure analysis device 30 generates a plurality of overlapping partial images from one image. Therefore, the failure analysis device 30 can improve the accuracy of determining the state of the boundary surface between the partial images.

In addition, the failure analysis device 30 obtains an image obtained by capturing an image captured by the imaging unit 11 mounted on a moving object (for example, the vehicle 10) moving along a surface (for example, a road surface) and capturing an area including the surface. Information acquisition unit 361, state information (for example, defect information) indicating the state of a surface in the image, and position information (for example, positioning) indicating the position of the surface in each of the images acquired by the image acquisition unit. Information), and a display that displays, on a map, a position indicated by the position information, state information indicating a state of a surface at the position, and an image including the surface in association with each other. A unit 32.
Thus, the user can immediately grasp which surface at which position is in what state by referring to the map. That is, the failure analysis device 30 can present the state of the surface in an easily understandable manner.

Further, in the defect analysis device 30, the display unit 32 displays a surface (for example, a road surface) to which the position information (for example, defect coordinate information) belongs within a predetermined range (for example, within a predetermined distance), and For a plurality of surfaces having the same information (for example, defect information) (for example, a road surface defect exists), the state information is displayed in association with a representative position representing the plurality of surfaces.
Thereby, even when a plurality of pieces of state information are concentrated on the map, the failure analysis device 30 indicates the representative position, so that the state of the surface can be presented in an easily understandable manner.
The above is the description of the first embodiment.

[Second embodiment]
A second embodiment will be described. Here, the same components as those in the above-described embodiments are denoted by the same reference numerals, and the description is referred to.
[Overview of the failure analysis system 1A]
The failure analysis system 1A (not shown) according to the present embodiment has the same configuration as the failure analysis system 1 according to the first embodiment. However, the defect analysis system 1A is different in that in the image extraction process, a process for detecting a curve of a road is performed. In a curve, the moving direction of the vehicle changes. In a curve, even if the moving speed of the vehicle 10 is constant, the change in the imaging range of the road surface image, especially on the far side, becomes larger than a straight line. Therefore, the moving distance between the images is corrected according to the degree of the curve. As a result, the analysis accuracy of the road surface defect can be improved, and the load of the processing required for the analysis can be reduced.

[Configuration of failure analysis system 1A]
The failure analysis system 1A includes a failure analysis device 30A (not shown) instead of the failure analysis device 30 included in the failure analysis system 1. The failure analysis device 30A includes an image extraction unit 363A (not shown) instead of the image extraction unit 363 included in the failure analysis device 30.
The image extracting unit 363A performs an image extracting process in the same manner as the image extracting unit 363. However, the difference is that the image extracting unit 363A corrects the moving distance between the images according to the curve of the road. The image extracting unit 363A corrects the moving distance between the images as the degree of the curve is larger (ie, the steeper the curve), and increases the smaller the degree of the curve (ie, the curve is gentler). The correction is made so that the moving distance between the images is reduced. In other words, the smaller the degree of the curve, the thinner the road image, and the smaller the degree of the curve, the thinner the road image. As a result, more images are extracted as the degree of the curve is larger, and fewer images are extracted as the degree of the curve is smaller.

Here, two types of curve analysis methods will be described.
First, a first curve analysis method will be described.
FIG. 18 is a diagram illustrating an outline of the first curve analysis method.
In the example shown in FIG. 18, eight positioning points m1 to m8 exist on the road R10. The road R10 is substantially curved over the positions m4 to m7.
In the first curve analysis method, the degree of inclination is measured from the positioning information of three consecutive points. For example, when analyzing the curve at the positioning point m5, the moving direction v4 from the positioning point m4 to the positioning point m5 and the moving direction v5 from the positioning point m5 to the positioning point m6 are calculated. Then, the angle θ between the two moving directions v4 and v5 is calculated. Thereby, the image extracting unit 363A can calculate the angle θ indicating the degree of the curve.

Next, a second curve analysis method will be described.
FIG. 19 is a diagram showing an outline of the second curve analysis method.
In the example shown in FIG. 19, as in the example shown in FIG. 18, eight positioning points m1 to m8 exist on the road R10.
In the second curve analysis method, the degree of the curve is calculated by comparing the sum of the distances between the positions measured within the predetermined period with the distance between the positions measured at the beginning and end of the predetermined period. For example, when the positioning points specified within a predetermined period from the start of the measurement of the positioning point m2 are the positioning points m3 to m7, the image extracting unit 363 determines the positioning points m2 to m3, m3 to m4, m4 to m5, and m5. M6, and the sum L1 of the distances between m6 and m7 is calculated. Further, the image extracting unit 363A calculates a linear distance L2 between the positioning points m2 to m7. The image extracting unit 363A can specify the degree of the curve by comparing the distance L1 and the distance L2. For example, as the distance L2 is shorter than the distance L1, the degree of the curve is greater. If the distance L1 and the distance L2 are approximately the same, the degree of the curve is small.

In the first curve analysis method described above, a right turn and a left turn can be determined.
In the above-mentioned second curve analysis method, only the calculation of the distance and the comparison of the distance are performed, so that the load required for the processing can be reduced.
The above is the description of the configuration of the failure analysis system 1A.

[Operation of Failure Analysis Device 30A]
Next, the operation of the failure analysis device 30A will be described. Here, the operation in the image extraction processing will be described.
FIG. 20 is a flowchart illustrating the flow of the image extraction process performed by the failure analysis device 30A.
The processes of steps S200, S202, and S204 to S216 shown in FIG. 20 are the same as the processes shown in FIG.

(Step S203A) After the process of step S202, the image extracting unit 363A calculates the degree of the curve. Then, the image extracting unit 363A corrects the moving distance between the images according to the degree of the curve. After that, the control section 360 advances the process to step S204.
Here, the case where the moving distance between the images is corrected has been described, but the set distance may be corrected instead of the moving distance between the images. Further, the correction of the distance may be performed by an arbitrary method. For example, a correction amount for each degree of curve may be set in advance, and the distance may be corrected based on the correction amount, or the correction may be performed using a function including a parameter indicating the degree of curve.
The above is the description of the operation of the failure analysis device 30A.

[Summary of Second Embodiment]
As described above, the failure analysis device 30A extracts the second image (for example, the inspection image) based on the amount of change (for example, the degree of curve) in the moving direction of the moving body (for example, the vehicle 10).
Thus, the failure analysis device 30A changes the amount of extraction of the second image according to, for example, the amount of change in the moving direction of the moving object. That is, the extraction amount of the second image is changed in accordance with the change in the imaging range of the image. Therefore, even when the moving direction of the moving body is not constant, it is possible to extract the second image necessary and sufficient for determining the state of the surface.

In addition, when the amount of change in the moving direction (for example, the degree of curve) of the moving body (for example, the vehicle 10) is large, compared to when the amount of change in the moving direction of the moving body is small, more second images ( For example, an inspection image) is extracted.
Thereby, even if the moving direction of the moving object is not constant, the failure analysis device 30A can extract the second image necessary and sufficient for determining the state of the surface.
The above is the description of the second embodiment.

[Modification]
As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the above-described embodiments, and includes a design and the like within a range not departing from the gist of the present invention. For example, the components described in the first and second embodiments can be arbitrarily combined. In addition, for example, each configuration described in the above-described first and second embodiments can be omitted if it is not necessary to exhibit a specific function. Further, for example, each configuration described in the first and second embodiments can be arbitrarily separated and provided in a separate device.

  The information terminal device 13 may be carried by a user without being mounted on the vehicle 10. The information terminal device 13 may generate defect information on a road surface defect that the user has found inside and outside the vehicle 10. For example, a still image may be captured in response to a voice input of “crack” by a user, and defect information in which “crack” is associated with the captured still image as a road surface defect type may be generated. In this case, the defect information generated by the information terminal device 13 may be stored in the defect information management device 50 as first-type defect information. Alternatively, the flag information may be associated with the road surface image being captured in response to a voice input of “crack” by the user. The flag information is information indicating a result of confirmation of a road surface defect by the user. The failure analysis device 30 allows the road surface image associated with the flag information to be preferentially extracted as a detected image in the image extraction processing, or increases the number of inspection areas to be extracted in the area extraction processing, and is generous. The analysis may be performed. Thereby, the failure analysis systems 1 and 1A can prevent omission of detection of a road surface defect, and can improve recognition accuracy of the road surface defect.

  Note that the imaging unit 11 and the positioning unit 12 may have a function of supporting imaging and positioning. For example, when the installation environment of the imaging unit 11 is adjusted, or when positioning is possible due to reception of GPS radio waves, the fact that imaging and positioning can be performed may be notified.

  The information terminal device 13 may present the defect information to the user in synchronization with the movement of the information terminal device 13. For example, the marker O21 of the vehicle 10 shown in FIG. 17 may be synchronized with the position of the own device. Further, when the own device is located near the marker O22, the user may be notified that there is a road surface defect nearby. Thereby, the user can confirm a road surface defect located near the user. The user can easily confirm a road surface defect detected in the past, for example, by driving the vehicle while carrying the information terminal device 13. Therefore, the defect analysis systems 1 and 1A can easily confirm the position and progress of the road surface defect, and can notify the danger of the road surface defect.

As described above, the information terminal device 13 changes the surface (for example, the road surface) based on the position of the moving object (for example, a vehicle different from the one at the time of capturing the road surface image) based on the output from the failure analysis device 30. Display the included image.
Thereby, since the information terminal device 13 presents, for example, an image of a surface near the moving object, the information terminal device 13 can easily present the state of the surface to the user who is moving with the moving object.

Further, the information terminal device 13 is based on the position of the moving object (for example, a vehicle different from the one at the time of capturing the road surface image) and the position information (for example, positioning information) based on the output from the failure analysis device 30. Then, the status information corresponding to the position information is displayed.
Thereby, since the information terminal device 13 presents, for example, the state of the surface in the vicinity of the moving object, the information terminal device 13 can easily present the state of the surface to the user who is moving with the moving object.

  Note that the failure analyzers 30 and 30A may perform machine learning in advance so as to recognize objects and defects other than those described above. For example, the failure analyzer 30 recognizes manholes, tunnel wall damage, road fallen objects, oil stains on road surfaces, guardrail damage, utility pole damage, graffiti on roads, road signs, blurred lane markings, snowfall, flooding, damage to gutters, and the like. You may make it. The degree of the item to be recognized may be recognized.

  Note that the failure analysis devices 30 and 30A may determine the state of the road surface by integrating the road surface images captured by the plurality of imaging units 11. For example, a plurality of vehicles 10 run in parallel on a road, and the imaging unit 11 mounted on each vehicle captures a road surface image. In addition, the positioning unit 12 mounted on each vehicle 10 performs positioning. The failure analysis devices 30 and 30A rearrange and reconstruct the plurality of road surface images captured in this manner, for example, based on the imaging time of each frame. Then, the failure analysis devices 3030 and 30A may determine the state of the road surface with respect to the reconstructed road surface image.

Note that the failure analysis devices 30 and 30A may perform a process of excluding (thinning out) areas other than the inspection area in the area extraction processing.
FIG. 21 is a diagram illustrating another example of the region extraction processing.
In the example shown in FIG. 21, the area A21 is thinned out from the inspection image P8, and the remaining trapezoidal area is set as the inspection area. The area A21 may be determined in advance according to how the road surface is captured in the captured image. As described above, the failure analysis devices 30 and 30A may determine an area that is not an inspection area.

  In the above-described embodiment, the case where a plurality of inspection regions having the same size are extracted in the region extraction processing has been described, but the present invention is not limited to this. For example, in the road surface image, the inspection area may be set small on the depth side, that is, a distant portion, and the inspection area may be set large on the near side, that is, in the vicinity. For example, in the example shown in FIG. 16, the size of the region A10 may be set smaller than the sizes of the regions A1 to A4. Thereby, on the depth side of the road surface image, it is possible to suppress a plurality of road surface defects from being included in one detection region or an object other than a road surface defect from being included in the detection region.

  Further, in the above-described embodiment, the case where a plurality of inspection areas are extracted in an overlapping manner in the area extraction processing has been described. Each inspection area may be extracted so as not to overlap. Further, the defect analysis devices 30 and 30A may extract the inspection area so as to cover all the parts of the inspection image.

  In the region extraction processing, the inspection region may not be fixed. For example, the failure analysis devices 30 and 30A may change the position and the number of the inspection areas according to the change in the moving direction. For example, the failure analysis devices 30 and 30A set more inspection areas on the right side of the inspection image in the case of a right turn, and set more inspection areas on the left side of the inspection image in the case of a left turn. Is also good. Further, the failure analysis devices 30 and 30A may change the position and the number of the inspection area according to the amount of change in the moving direction, for example. For example, when the degree of the curve is large, more inspection areas may be set in front of the inspection image.

  Further, the failure analysis devices 30 and 30A may output an analysis result indicating that analysis is not possible for the inspection area. For example, the failure analysis devices 30 and 30A cannot analyze when the road surface is not shown in the inspection area, when the inspection area is unclear due to the influence of shadow, when there is snowfall, or when there is flooding. The analysis result may be output together with the reason.

  In the above-described embodiment, a mode in which a road surface defect is detected using the defect analysis artificial intelligence has been described, but the present invention is not limited to this. For example, the defect analysis devices 30 and 30A may extract an image feature amount and analyze a road surface defect. Further, the failure analysis devices 30 and 30A may analyze a road surface failure using a technique such as a support vector machine or clustering.

Note that the presentation mode of the defect information is not limited to the above. The failure analysis devices 30 and 30A may analyze, for example, road surface images captured at different times and present only a portion where the road surface failure has increased.
Note that the analysis range of a series of road surface images captured at one time may be determined by an input from a user. The failure analysis devices 30 and 30A may receive, for example, a range of frames to be analyzed by specifying time or by specifying a place. When receiving the designation of a place, the failure analysis devices 30, 30A may determine the analysis range with reference to the coordinates of the designated place and the positioning information associated with the road surface image.

  The accuracy of the positioning information may be improved by an arbitrary method. For example, the failure analysis devices 30 and 30A use the speed information measured by the Doppler method by GPS or the speed information obtained by ODB2, etc., so that even if there is an error in the positioning result by GPS, the failure analysis device 30 or 30A can use the speed information based on the speed information. It is possible to improve the positioning accuracy by correcting the moving distance. Further, for example, the failure analysis devices 30 and 30A may improve the positioning accuracy by using road information, traveling route information input by a user, and the like.

Note that the imaging unit 11 may be mounted on the vehicle 10 in any mode.
The imaging unit 11 may be installed inside the vehicle, or may be installed outside the vehicle. Further, the imaging unit 11 may be installed so as to image the front of the vehicle, or may be installed so as to image the rear. Further, the imaging unit 11 may be carried by the user, for example, worn by the user.

  Also, a program for realizing the functions of the information terminal device 13, the failure analysis device 30, the failure information management device 50, and the map information management device 90 is recorded on a computer-readable recording medium, and is recorded on the recording medium. The computer program may be read and executed to execute the processing as the information terminal device 13, the failure analysis device 30, the failure information management device 50, and the map information management device 90. Here, "to make the computer system read and execute the program recorded on the recording medium" includes installing the program in the computer system. The “computer system” here includes an OS and hardware such as peripheral devices. Further, the “computer system” may include a plurality of computer devices connected via a network including a communication line such as the Internet, a WAN, a LAN, and a dedicated line. The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built in a computer system. As described above, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. The recording medium also includes an internal or external recording medium accessible from the distribution server for distributing the program. The code of the program stored in the recording medium of the distribution server may be different from the code of the program in a format executable by the terminal device. That is, any format can be stored in the distribution server as long as it can be downloaded from the distribution server and installed in a form executable by the terminal device. The program may be divided into a plurality of programs, downloaded at different timings, and then combined by the terminal device, or the distribution server that distributes each of the divided programs may be different. Further, the “computer-readable recording medium” holds a program for a certain period of time, such as a volatile memory (RAM) in a computer system serving as a server or a client when the program is transmitted via a network. Shall be included. Further, the program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

1, 1A: Failure analysis system, 10: Vehicle, 11: Imaging unit, 12: Positioning unit, 13: Information terminal device, 30, 30A: Failure analysis device, 31: Input unit, 32: Display unit, 33: Voice reproduction Unit, 34 communication unit, 35 storage unit, 351 learning result storage unit, 360 control unit, 361 information acquisition unit, 362 image conversion unit, 363, 363A image extraction unit, 364 area extraction unit 365 analysis part, 366 learning part, 367 recognition part, 368 result generation part, 369 result editing part, 50 defect information management device, 51 type I defect information storage part, 52 type II defect Information storage unit, 70: ledger information management device, 71: ledger information storage unit, 90: map information management device, 91: map information storage unit

Claims (5)

An image acquisition unit that is continuously imaged by an imaging device mounted on a moving body that moves along the surface of the building, and acquires an image in which a region including the surface is imaged,
A partial image generating unit that divides the image obtained by the image obtaining unit and outputs a plurality of partial images that are part of the image;
For each of the partial images output by the partial image generation unit, a state determination unit that determines a defect that appears on the surface in the partial image,
Equipped with a,
The partial image generation unit, the partial image near the position of the moving body, output more than the partial image far away,
Information processing device. An image acquisition unit that is continuously imaged by an imaging device mounted on a moving body that moves along the surface of the building, and acquires an image in which a region including the surface is imaged,
A partial image generating unit that divides the image obtained by the image obtaining unit and outputs a plurality of partial images that are part of the image;
For each of the partial images output by the partial image generation unit, a state determination unit that determines a defect that appears on the surface in the partial image,
Equipped with a,
The partial image generation unit, the partial image closer to the position of the moving body, output more than the partial image far away,
The partial image includes the image in which the surface to be determined is left and at least a part of the area other than the surface is excluded.
The information processing device according to claim 1. An image acquisition unit that is continuously imaged by an imaging device mounted on a moving body that moves along the surface of the building, and acquires an image in which a region including the surface is imaged,
A partial image generating unit that divides the image obtained by the image obtaining unit and outputs a plurality of partial images that are part of the image;
For each of the partial images output by the partial image generation unit, a state determination unit that determines a defect that appears on the surface in the partial image,
Equipped with a,
The partial image generation unit divides an image, and places the divided first partial image and a second partial image including a vertex portion or a side portion of the first partial image near the position of the moving object. Outputting such that the partial image of is output more than the distant partial image,
Information processing device. The information processing device is
A first step of acquiring an image in which an image is continuously taken by an imaging device mounted on a moving body that moves along the surface of the building, and a region including the surface is taken;
A second step of dividing the image acquired in the first step and outputting a plurality of partial images that are a part of the image;
A third step of determining, for each of the partial images output in the second step, a defect that has appeared on the surface in the partial image;
It includes,
In the second step, the partial images closer to the position of the moving object are output more than the partial images farther away,
Information processing method. On the computer,
A first step of acquiring an image in which an image is continuously taken by an imaging device mounted on a moving body that moves along the surface of the building, and a region including the surface is taken;
A second step of dividing the image acquired in the first step and outputting a plurality of partial images that are a part of the image;
A third step of determining, for each of the partial images output in the second step, a defect that has appeared on the surface in the partial image;
Was executed,
Wherein in the second step, the said partial image in the vicinity with respect to the position of the moving object, because the program to output more than far the partial image.