JP6955295B1 - Identification device, identification program, and identification method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an identification device, an identification method and an identification program capable of accurately detecting the presence or absence of damage to a road surface. In an identification device including a terminal device 10 and an information processing server 20, the terminal device is mounted on a vehicle and is a trained model for a photographing unit that captures an image of a road surface and an image captured by the photographing unit. It is provided with an identification unit for identifying the presence or absence of damage to the road surface by performing identification using the above, and a transmission unit for transmitting the detected image in which the identification unit has detected the damage to the information processing server. The information processing server 20 has a position determination unit that determines whether or not the damage to the road surface captured by the plurality of detected images transmitted from the transmission unit is the same location, and a road surface that is determined to be the same location by the damage determination unit. It is provided with an evaluation unit that integrates information on the damage of the road surface in chronological order and evaluates the presence or absence of damage to the road surface. [Selection diagram] Fig. 1

Description

The present disclosure relates to identification devices, identification programs, and identification methods.

In recent years, a system that captures the state of a road with a camera and detects the presence or absence of damage to the road surface has been known.

For example, in Patent Document 1, the road surface is imaged by a camera while the inspection vehicle is running on the road surface, and the shadow of the inspection device, the painted display guide on the road surface, the dirt on the road surface, and the like are detected from the captured image. A system has been disclosed that removes a non-damaged area.

Japanese Unexamined Patent Publication No. 2005-43324

The condition of the road surface may be photographed differently depending on the time zone in which the image is taken. For example, in the early morning or evening, if the direction in which shadows are formed by sunlight is different, different images will be taken. In addition, the condition of the road surface changes with time. For this reason, when determining the presence or absence of damage using a photographed image of a temporary state, the damage detection accuracy may vary.

An object of the present disclosure is to provide an identification device, an identification method, and an identification program capable of accurately detecting the presence or absence of damage to a road surface.

The identification device of the present disclosure is an identification device including a terminal device and an information processing server. , An identification unit that identifies the presence or absence of damage to the road surface by performing identification using a trained model, and a transmission unit that transmits the detected image in which the identification unit detects damage to the information processing server. The information processing server is determined by the position determination unit that determines whether or not the damage to the road surface captured by the plurality of detected images transmitted from the transmission unit is the same location, and the damage determination unit that the damage determination unit determines that the damage is the same location. It is provided with an evaluation unit that integrates information on road surface damage in chronological order and evaluates the presence or absence of road surface damage.

According to the identification device, the identification program, and the identification method of the present disclosure, it is possible to accurately detect the presence or absence of damage to the road surface while greatly suppressing the amount of data communication from the terminal device for photographing.

It is a block diagram which shows the structural example of the identification apparatus which concerns on this embodiment. It is a block diagram which shows the example of the functional structure of the mobile terminal shown in FIG. It is a figure explaining the state which mounted the terminal device in a vehicle. It is a figure which shows the example of the 1st screen displayed on the information processing server. It is a block diagram which shows the example of the functional structure of the server shown in FIG. It is a figure which shows the example of the 2nd screen displayed on the information processing server. It is a figure which shows the 1st example of the type of damage which is identified by the identification apparatus. It is a figure which shows the 2nd example of the type of damage which is identified by the identification apparatus. It is a figure explaining the flow of the process executed by the identification apparatus. It is a figure which shows an example of evaluation data.

Hereinafter, embodiments will be described with reference to the drawings. The present embodiment described below does not unreasonably limit the content of the present disclosure described in the claims. Moreover, not all of the configurations described in the present embodiment are essential constituent requirements of the present disclosure.

(Embodiment)
<Overview>
The outline of the embodiment of the identification device 1 of the present disclosure will be described.
The identification device 1 according to the present embodiment is a device that identifies damage to the road surface on the road on which the vehicle travels and detects the presence or absence of damage.

<1. System configuration>
First, the configuration of the identification device 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration example of the identification device 1 according to the present embodiment. The identification device 1 shown in FIG. 1 includes a terminal device 10 and an information processing server 20.
The terminal device 10 and the information processing server 20 are connected to each other so as to be able to communicate with each other via the network 30. The network 30 is composed of a wired or wireless network.

The terminal device 10 and the information processing server 20 are connected to the network 30 by using the wireless base station 31 or the wireless LAN standard 32. The terminal device 10 and the information processing server 20 may be connected to the network 30 by wire communication.
Here, an aggregate of each device (for example, the terminal device 10 and the information processing server 20) constituting the identification device 1 can be grasped as one "information processing device". That is, the identification device 1 may be realized as an aggregate of a plurality of devices, and the distribution of a plurality of functions for realizing the identification device 1 may be appropriately determined based on the processing capacity of the hardware of each device.

<2. Configuration of terminal device 10>
The terminal device 10 is a terminal mounted on a vehicle. The terminal device 10 is a general-purpose mobile terminal such as a smartphone, a tablet terminal, or a laptop personal computer. As the terminal device 10, a plurality of smartphones, tablet terminals, laptop personal computers, and the like may be used. Such a mobile terminal may be a terminal mounted on the vehicle for the purpose of confirming the traveling route of the vehicle while driving.
Further, the terminal device 10 may use a general-purpose drive recorder mounted on the vehicle for the purpose of recording a situation at the time of an accident, and a device equipped with each function as the terminal device 10 described later.
That is, in the present invention, the terminal device 10 for photographing the state of the road surface is not limited to a dedicated device used only for photographing the road surface, and utilizes the photographing function of the terminal device 10 mounted for other purposes. , It may be a device that also serves as an incidental image of the road surface.

The terminal device 10 includes a processor 11, a memory 12, a storage 13, a communication IF 14, and an input / output IF 15.
The processor 11 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The memory 12 is for temporarily storing a program, data processed by the program or the like, and is a volatile memory such as a DRAM (Dynamic Random Access Memory).

The storage 13 is a storage device for storing data, and is, for example, a flash memory, an HDD (Hard Disc Drive), or an SSD (Solid State Drive).

The communication IF 14 is an interface for inputting / outputting signals because the identification device 1 communicates with an external device.

The input / output IF15 is an interface with an input device (for example, a pointing device such as a mouse, a keyboard) for receiving an input operation from a user, and an output device (display, a speaker, etc.) for presenting information to the user. Functions as.

The information processing server 20 includes a processor 21, a memory 22, a storage 23, a communication IF 24, and an input / output IF 25. The processor 21, the memory 22, the storage 23, the communication IF 24, and the input / output IF 25 are the same as the processor 11, the memory 12, the storage 13, the communication IF 14, and the input / output IF 15, respectively. Is omitted.

The information processing server 20 is realized by a computer, a mainframe, or the like. The information processing server 20 may be realized by one computer or may be realized by combining a plurality of computers.

<1. Configuration of terminal device 10>
FIG. 2 is a block diagram showing a functional configuration of the terminal device 10. As shown in FIG. 2, the terminal device 10 operates with a plurality of antennas (antenna 111, antenna 112) and wireless communication units (first wireless communication unit 121, second wireless communication unit 122) corresponding to each antenna. It includes a reception unit 130, a voice processing unit 140 (including a microphone 141 and a speaker 142), a photographing unit 150, a storage unit 160, a control unit 170, and a GPS antenna 180. The terminal device 10 may include a keyboard as the operation reception unit 130.

The terminal device 10 also has functions and configurations (for example, a battery for holding electric power, a power supply circuit for controlling the supply of electric power from the battery to each circuit, and the like) which are not particularly shown in FIG. As shown in FIG. 2, each block included in the terminal device 10 is electrically connected by a bus or the like.

The antenna 111 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 111 receives radio waves from the space and gives a received signal to the first wireless communication unit 121.

The antenna 112 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 112 receives radio waves from the space and gives a received signal to the second radio communication unit 122.

Since the terminal device 10 communicates with other wireless devices, the first wireless communication unit 121 performs modulation / demodulation processing for transmitting / receiving signals via the antenna 111. Since the terminal device 10 communicates with other wireless devices, the second wireless communication unit 122 performs modulation / demodulation processing for transmitting / receiving signals via the antenna 112. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation / demodulation and frequency conversion of the wireless signal transmitted / received by the terminal device 10 and give the received signal to the control unit 170.

The operation receiving unit 130 has a mechanism for receiving a user's input operation. Specifically, the operation reception unit 130 includes the display 131. The operation reception unit 130 is configured as a touch screen that detects the contact position of the user with respect to the touch panel by using a capacitance type touch panel.

The display 132 displays data such as an image, a moving image, and a text under the control of the control unit 170. The display 132 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

The voice processing unit 140 performs modulation / demodulation of the voice signal. The voice processing unit 140 modulates the signal given from the microphone 141 and gives the modulated signal to the control unit 170. Further, the voice processing unit 140 gives a voice signal to the speaker 142. The voice processing unit 140 is realized by, for example, a processor for voice processing.
The microphone 141 receives the voice input and gives the voice signal corresponding to the voice input to the voice processing unit 140. The speaker 142 converts the voice signal given from the voice processing unit 140 into voice and outputs the voice to the outside of the terminal device 10.

The photographing unit 150 is a device mounted on a vehicle and taking an image of a road surface. Specifically, the photographing unit 150 is a camera that receives light by a light receiving element and outputs it as a photographed image 161.
The photographing unit 150 is, for example, a depth camera capable of detecting the distance from the photographing unit 150 to the photographing target. The shooting unit 150 can shoot not only still images but also moving images. When the photographing unit 150 shoots a moving image, the image for each frame constituting the moving image is treated as the captured image 161. FIG. 3 is a diagram illustrating a state in which the terminal device 10 is mounted on the vehicle.

FIG. 3A is a diagram showing the appearance of the terminal device 10 mounted on the vehicle. FIG. 3B is a diagram showing an example of a captured image 161 displayed on the terminal device 10 mounted on the vehicle.
As shown in FIG. 3A, the terminal device 10 is mounted on the vehicle in a posture in which the photographing unit 150 faces the front of the vehicle. The photographing unit 150 intermittently photographs the state of the road surface in front of the vehicle at predetermined time intervals.

As shown in FIG. 3B, a bounding box B surrounding the damage to the road surface is output to the captured image 161. By outputting the bounding box B surrounding the damage in this way, not only the presence or absence of damage on the road surface but also the position of the damage can be identified, and more detailed information on the state of damage can be obtained. In addition, the bounding box B emphasizes the position of the damage, making it easy to confirm the identification result.

The GPS (Global Positioning System) antenna 180 detects the position of the vehicle and acquires information on the route traveled by the vehicle. The GPS antenna 180 transmits the acquired information on the vehicle position and the traveling route to the transmission / reception unit 172 of the control unit 170. The transmission / reception unit 172 transmits the received information on the vehicle position and the traveling route to the information processing server 20.

FIG. 4 is a diagram showing an example of the first screen DP1 displayed on the information processing server 20 according to the present embodiment. The first screen DP1 may be displayed when the identification unit 173 receives the detection image in which the damage is detected. Here, the identification device 1 transmits the information regarding the traveling route of the vehicle acquired from the GPS antenna 180 to the information processing server 20 as the information regarding the position where the image is taken.

In this example, the traveling route indicated by the reference numeral R is shown on the map DP11, and the position where the detection image 2021 (see FIG. 5) in which the damage to the road surface is detected is captured by the marker indicated by the reference numeral D is shown. There is. As a result, it becomes clear where and what kind of damage is occurring, and it is possible to notify an appropriate manager according to the location of the damage.
The first screen DP1 is a map DP11 showing the position where the image was taken, an information DP12 of the road surface where the image was taken, a data ID DP13 for uniquely identifying the image, and a time stamp DP14 showing the date and time when the image was taken. Etc. are included.

In the illustrated example, the information DP12 of the road surface on which the image was taken is the traffic volume (Traffic Volume, number of lanes), width (Road Width), legal maximum speed (Max Speed) and type (Road Type). However, information other than these may be included, or any one of them may not be included.
By displaying the data ID DP13 and the time stamp DP14 in this way, it is possible to uniquely identify the data of the captured image 161 and clarify when and what kind of damage has occurred. , It is possible to provide a judgment material such as the timing of repair.

The storage unit 160 shown in FIG. 2 is composed of, for example, a flash memory or the like, and stores data and programs used by the terminal device 10. In a certain aspect, the storage unit 160 stores the captured image 161, the first trained model 162, and the second trained model 163. The captured image 161 is an image captured by the photographing unit 150.

The control unit 170 controls the operation of the terminal device 10 by reading the program stored in the storage unit 160 and executing the instructions included in the program. The control unit 170 is, for example, an application installed in the terminal device 10 in advance. By operating according to the program, the control unit 170 exhibits functions as an input operation reception unit 171, a transmission / reception unit 172, an identification unit 173, and a display processing unit 174.

The input operation reception unit 171 performs a process of accepting a user's input operation to the input device.

The transmission / reception unit 172 performs a process for the terminal device 10 to transmit / receive data to / from an external device such as an information processing server 20 according to a communication protocol. The transmission / reception unit 172 executes a process of transmitting the detection image 2021, which is an image in which the identification unit 173 has detected damage, out of the captured images 161 captured by the photographing unit 150 to the information processing server 20.

The identification unit 173 identifies the presence or absence of damage to the road surface with respect to the photographed image 161 captured by the photographing unit 150 of the terminal device 10.
In the present embodiment, the photographing unit 150 captures images at predetermined time intervals, and the identification unit 173 identifies damage to the road surface within a predetermined time interval. Here, the predetermined time interval is, for example, 1.2 seconds. When taking images at 1.2 second intervals, if the vehicle is driven at 40 km / h (about 11.1 m / s), the road surface will be taken every 13 m, and the road surface will be taken without interruption. be able to.
In this way, the identification unit 173 recognizes the damage to the road surface within the photographing interval of the photographing unit 150. Therefore, by appropriately selecting a predetermined time interval according to the speed of the vehicle, the road surface is photographed without interruption. And the damage can be identified.

The identification unit 173 uses the first trained model 162 to identify the presence or absence of damage to the road surface.
The first trained model 162 is stored in the storage unit 160 in advance, for example. The first trained model 162 may be generated and updated at any time based on the captured image 161 captured by the photographing unit 150.
The first trained model 162 is a model for identifying whether or not there is a damaged portion of the road surface in the image.

The first trained model 162 is obtained by causing a machine learning model to perform machine learning according to a model learning program based on training data. For example, in the present embodiment, the first trained model 162 is trained to output the damage location and the damage type with respect to the input image information.
At this time, for the learning data, for example, the image information of the road surface in which the state of the road surface was photographed in the past is used as the input data, and the information regarding the damaged portion and the information regarding the type of damage are provided with respect to the input image information. Correct output data.

Specifically, the first trained model 162 is similar by comparing the feature amount (reference feature amount) of the image of the road surface having various types of damage with the feature amount of the captured image 161 to be evaluated. The degree is assessed to determine if there is any type of damage. In the evaluation of the degree of similarity, if the feature amount of the captured image 161 to be evaluated is within the range of the preset threshold value with respect to the reference feature amount, it is determined that the damage is present. Reference features are set for various types of damage. The type of damage will be described later.

The first trained model 162 according to the present embodiment is, for example, a composite function with parameters in which a plurality of functions are composited. A parameterized composition function is defined by a plurality of adjustable functions and a combination of parameters. The first trained model 162 according to the present embodiment may be a composite function with any parameter that satisfies the above requirements, but is assumed to be a multi-layer neural network model (hereinafter referred to as “multi-layer network”). The first trained model 162 using the multi-layer network has an input layer, an output layer, and at least one intermediate layer or a hidden layer provided between the input layer and the output layer. The first trained model 162 is expected to be used as a program module that is a part of artificial intelligence software.

As the multi-layer network according to the present embodiment, for example, a deep neural network (DNN), which is a multi-layer neural network targeted for deep learning, can be used. As the DNN, for example, a convolutional neural network (CNN) for an image may be used.

Further, the identification unit 173 determines whether or not the damage detected in the detection image 2021 is located in the road region by using the second trained model 163.

The second trained model 163 is stored in the storage unit 160 in advance, for example. The second trained model 163 may be generated and updated at any time based on the captured image 161 captured by the photographing unit 150.
The second trained model 163 is a model for identifying whether or not the damaged portion detected in the image is in the road area.

The second trained model 163 is obtained by causing the machine learning model to perform machine learning according to the model learning program based on the training data. For example, in the present embodiment, the second trained model 163 is trained to output whether or not the damaged portion is in the road area with respect to the input image information.
At this time, for the learning data, for example, the image information of the road surface in which the state of the damaged road surface was photographed in the past is used as the input data, and whether or not the damaged part is located in the road area with respect to the input image information. The information about is used as the correct answer output data.

Specifically, the second trained model 163 compares and resembles the feature amount (reference feature amount) of the image of the road surface with damage in the road area and the feature amount of the detected image 2021 to be evaluated. The degree is assessed to determine if there is any type of damage. In the evaluation of the similarity, if the feature amount of the detection image 2021 to be evaluated is within the range of the preset threshold value with respect to the reference feature amount, it is determined that the damage is in the road area. NS.

The second trained model 163 according to the present embodiment is, for example, a composite function with parameters in which a plurality of functions are composited. A parameterized composition function is defined by a plurality of adjustable functions and a combination of parameters. The second trained model 163 according to the present embodiment may be a composite function with any parameter that satisfies the above requirements, but is assumed to be a multi-layer network. The second trained model 163 using the multi-layer network has an input layer, an output layer, and at least one intermediate layer or a hidden layer provided between the input layer and the output layer. The second trained model 163 is expected to be used as a program module that is a part of artificial intelligence software.

The identification unit 173 associates information about the photographed position and the traveling route with the photographed image 161. The identification unit 173 associates the captured image 161 with information regarding the time when the captured image 161 was captured, the position of the vehicle, and the traveling direction of the vehicle.

The display processing unit 174 performs a process of presenting information to the user. The display processing unit 174 performs a process of displaying the display image on the display 132, a process of outputting the sound to the speaker 142, and the like.

<2. Functional configuration of information processing server 20>
FIG. 5 is a diagram showing a functional configuration of the information processing server 20. As shown in FIG. 3, the information processing server 20 exerts functions as a communication unit 201, a storage unit 202, and a control unit 203.

The communication unit 201 performs processing for the information processing server 20 to communicate with an external device.

The storage unit 202 stores data and programs used by the information processing server 20. The storage unit 202 stores the detected image 2021, the third trained model 2022, the fourth trained model 2023, and the evaluation data 2024.

The detection image 2021 is image information transmitted from the terminal device 10. That is, the detection image 2021 is a captured image 161 in which the identification unit 173 of the terminal device 10 has detected damage to the road surface.
The third trained model 2022 and the evaluation data 2024 will be described later.

The control unit 203 functions as a transmission / reception unit 2031, a re-identification unit 2032, a position determination unit 2033, and an evaluation unit 2034 when the processor 21 of the information processing server 20 performs processing according to a program.

The transmission / reception unit 2031 controls a process in which the information processing server 20 transmits a signal to an external device according to the communication protocol, and a process in which the information processing server 20 receives a signal from the external device according to the communication protocol.

The transmission / reception unit 2031 acquires information on the external environment in which the vehicle travels. Information about the external environment includes information about the weather, time of day, and brightness.

The re-identification unit 2032 reidentifies the presence or absence of damage to the road surface by identifying the plurality of detected images 2021 transmitted from the terminal device 10 using the third trained model 2022.

The third trained model 2022 is stored in the storage unit 202 in advance, for example. The third trained model 2022 may be generated and updated at any time based on the captured image 161 captured by the photographing unit 150.
The third trained model 2022 is a model for discriminating whether or not there is a damaged portion of the road surface in the image, and has the same configuration as the first trained model 162 except for the determination criteria in the discrimination.

The criteria for identification by the third trained model 2022 used by the re-identification unit 2032 of the information processing server 20 are different from the criteria for identification by the first trained model 162 used by the identification unit 173 of the terminal device 10. For example, the discrimination by the first trained model 162 is a looser judgment criterion than the discrimination by the third trained model 2022.

Specifically, the threshold value for the reference feature amount set in the third trained model 2022 has a narrower allowable numerical range than the threshold value for the reference feature amount set in the first trained model 162. There is. That is, in the third trained model 2022, when the similarity of the feature amounts is higher than that of the first trained model 162, it is determined that the captured image 161 is damaged.

In addition, the re-identification unit 2032 redetermines whether or not the damage to the road surface is located in the road area. The re-identification unit 2032 uses the fourth trained model 2023 to determine whether the damage detected in the detection image 2021 is located within the road area.

The fourth trained model 2023 is stored in the storage unit 202 in advance, for example. The fourth trained model 2023 may be generated and updated at any time based on the captured image 161 captured by the photographing unit 150.
The fourth trained model 2023 is a model for discriminating whether or not the damaged part detected in the image is in the road area, and has the same configuration as the second trained model 163 except for the judgment criteria in the discrimination. I have.

The criteria for identification by the fourth trained model 2023 used by the re-identification unit 2032 of the information processing server 20 are different from the criteria for identification by the second trained model 163 used by the identification unit 173 of the terminal device 10. For example, the discrimination by the second trained model 163 is a looser judgment criterion than the discrimination by the fourth trained model 2023.

Specifically, the threshold value for the reference feature amount set in the fourth trained model 2023 has a narrower allowable numerical range than the threshold value for the reference feature amount set in the second trained model 163. There is. That is, in the fourth trained model 2023, when the similarity of the feature amount is higher than that of the second trained model 163, it is determined that the detected road surface damage is in the road region.

The position determination unit 2033 determines whether or not the damage to the road surface captured by the plurality of detection images 2021 transmitted from the transmission / reception unit 172 of the terminal device 10 is the same location. Specifically, the position determination unit 2033 uses the detection image 2021 and the position information in which the detection image 2021 is captured to determine whether or not the damage to the road surface captured in the plurality of detection images 2021 is the same location. do.
Further, the position determination unit 2033 may determine whether or not the damage to the road surface captured by the plurality of detection images 2021 is the same location by using the information regarding the traveling route of the vehicle.

The evaluation unit 2034 integrates information on road surface damage determined to be the same location by the position determination unit 2033 in chronological order to evaluate the presence or absence of road surface damage. Specifically, the evaluation unit 2034 calculates the existence probability of road surface damage by integrating information on road surface damage in chronological order.
Here, the information regarding the damage to the road surface includes at least one of the information regarding the presence or absence of damage at the position where the detection image 2021 was taken and the detection image 2021.

The evaluation unit 2034 may weight the existence probability by using the information about the external environment. For example, the detection result in fine weather with good visibility may be weighted higher than the detection result in rainy weather with poor visibility. In addition, the detection result in the daytime when the outside is bright may be given higher weight than in the evening when the outside is dark.
The evaluation unit 2034 integrates information regarding the presence or absence of road surface damage in an arbitrary period for each coordinate position in which the coordinate damage is photographed.

FIG. 6 is a diagram showing an example of the second screen DP2 displayed on the information processing server 20 according to the present embodiment.
The second screen DP2 may be displayed when the re-identification unit 2032 of the information processing server 20 receives the detection image 2021 in which damage is detected and the identification result, and may be displayed side by side with the first screen DP1. The information processing server 20 may display the second screen DP2 and ask the administrator to confirm whether the identification result is correct, and if it is incorrect, accept the correction of the identification result from the administrator.

The second screen DP2 includes a detection image 2021 captured by the photographing unit 150, a checked phase DP22, a time stamp DP23, a correspondence status DP24, and a damage type DP25. On the second screen DP2, a bounding box B surrounding the damage identified by the identification unit 173 is displayed.

In the checked phase DP22, the progress of the damage checking work is displayed. In the illustrated example, the icon of the mobile terminal indicated by the reference numeral DP221 indicates that the identification of the damage in the terminal device 10 has been completed. The display icon indicated by the reference numeral DP222 indicates that the damage identification on the information processing server 20 has been completed. The eye icon indicated by the reference numeral DP223 indicates that the subsequent check process by crowdsourcing has been completed.

The time stamp DP23 on the second screen DP2 shows information regarding the date and time given when the detection image 2021 was taken. That is, it is information indicating the date and time when the detected image 2021 was taken.
The response status DP24 on the second screen DP2 describes information on the response to be taken for the damage. Information on the response includes, for example, follow-up, a report schedule to the manager, a report to the manager, a repair schedule, and the like.

The type of damage is described in the damage type DP25 on the second screen DP2. Here, the types of damage will be described with reference to FIGS. 7 and 8.
FIG. 7 is a diagram showing a first example of the type of damage identified by the identification device 1. FIG. 8 is a diagram showing a second example of the type of damage identified by the identification device 1.

The type of damage identified by D00 shown in FIG. 7A is a linear crack extending in the vertical direction (traveling direction) and refers to damage occurring in the wheel traveling portion.
The type of damage identified by D01 shown in FIG. 7B is a linear crack extending in the vertical direction and refers to damage occurring in the construction joint portion. The construction joint portion means a joint of asphalt pavement.

The type of damage identified in D10 shown in FIG. 7C is a linear crack extending along the lateral direction (width direction of the road) and refers to damage occurring in the wheel running portion.
The type of damage identified in D11 shown in FIG. 7D is a linear crack extending along the lateral direction and refers to damage occurring in the construction joint portion.

The type of damage identified by D20 shown in FIG. 8A refers to a hexagonal crack.
The types of damage identified by D40 shown in FIG. 8B refer to steps, potholes, and peels.
The type of damage identified by D43, shown in FIG. 8C, refers to pedestrian crossing fading.
The type of damage identified by D44 shown in FIG. 8D refers to the faint white line.

The above types of damage are merely examples, and other types of damage may be included. In this way, not only the presence or absence of damage to the road surface, but also the type of damage can be identified, and detailed information about the state of damage can be obtained.

<3. Control process of identification device 1>
Next, the control process of the identification device 1 will be described. FIG. 9 is a diagram illustrating a flow of processing executed by the identification device 1.
As shown in FIG. 9, first, the photographing unit 150 captures images at predetermined time intervals (step S110). The photographing unit 150 stores the photographed image 161 in the storage unit 160.
After step S110, the identification unit 173 identifies the presence or absence of damage on the road surface captured in the captured image 161 (step S111). Specifically, the identification unit 173 obtains an output regarding the presence or absence of damage by inputting the captured image 161 into the first trained model 162.

If no damage is confirmed in step S111 (No in step S112), the identification process for the captured image 161 by the identification device 1 is terminated, and the captured image 161 corresponding to the next frame is identified.

If damage is confirmed in step S111 (Yes in step S112), the identification unit 173 determines whether the damage is within the road area (step S113). Specifically, the identification unit 173 inputs the captured image 161 to the second trained model 163 to obtain an output of whether or not the damage is within the road area.

In step S113, when it is output that the damage is not within the road area (No in step S114), specifically, when the damage is located on a sidewalk, a surrounding building, or the like, the captured image 161 by the identification device 1 The identification process for the image 161 is completed, and the captured image 161 corresponding to the next frame is identified.

When it is output in step S113 that the damage is within the road area (Yes in step S114), the identification unit 173 associates information about the shooting position (step S115).
After step S115, the transmission / reception unit 172 of the terminal device 10 transmits the detection image 2021 to the information processing server 20 (step S116). As a result, the transmission / reception unit 2031 of the information processing server 20 receives the detected image 2021 and stores it in the storage unit 202.

After step S116, the re-identification unit 2032 of the information processing server 20 re-identifies the presence or absence of damage to the detected image 2021 (step S117). Specifically, the re-identification unit 2032 obtains an output regarding the presence or absence of damage by inputting the captured image 161 into the third trained model 2022.

If no damage is confirmed in step S117 (No in step S118), the identification process for the captured image 161 by the identification device 1 is terminated, and the captured image 161 corresponding to the next frame is identified.

If damage is confirmed in step S117 (Yes in step S118), the re-identification unit 2032 determines whether the damage is within the road area (step S119).
Specifically, the re-identification unit 2032 inputs the captured image 161 into the fourth trained model 2023 to obtain an output of whether or not the damage is within the road area.

In step S119, when it is output that the damage is not within the road area (No in step S120), specifically, when the damage is located on a sidewalk, a surrounding building, or the like, the captured image 161 by the identification device 1 The identification process for the image 161 is completed, and the captured image 161 corresponding to the next frame is identified.

When it is output in step S119 that the damage is within the road area (Yes in step S120), a visual check by crowdsourcing is executed (step S121). Specifically, the detection image 2021 is transmitted to an unspecified number of checkers and checked. In this result, if the number of answers that it is damaged exceeds a certain number, it is treated as damaged.

After step S121, the position determination unit 2033 determines the position of the detection image 2021. That is, it is determined whether or not the detected image 2021 in which damage is confirmed is the same location. Specifically, the storage unit 202 of the information processing server 20 stores a large number of detected images 2021 having different shooting dates and times. The position determination unit 2033 identifies these large numbers of detected images 2021 as the same location by using coordinates such as latitude and longitude.

After step S122, the evaluation unit 2034 calculates the damage probability distribution (step S123). Specifically, information on the presence or absence of road surface damage during an arbitrary period is integrated for each coordinate position where the road surface damage is photographed. The evaluation result at this time will be described with reference to FIG. FIG. 10 is a diagram showing an example of evaluation data 2024.

As shown in FIG. 10, the evaluation unit 2034 integrates information on road surface damage in chronological order for each coordinate position from the start point (start) to the destination (Goal) in the traveling path of the vehicle. Specifically, the evaluation unit 2034 calculates the probability that time is a variable. In the example of the evaluation data 2024 shown in FIG. 10, the evaluation index of the detection image 2021 determined to be damaged in the visual check by crowdsourcing is set to 1, and there is no damage in any of the steps up to the visual check by crowdsourcing. The evaluation index of the determined detection image 2021 is counted as 0. Then, the existence probability is calculated as 1 for the total number of samples.

The evaluation unit 2034 sets a threshold value of the existence probability as a final judgment. For example, when the threshold value is set to 80%, it is determined that there is damage to the coordinates having an existence probability of 80% or more. At this time, the evaluation unit 2034 may perform weighting according to the external environment (weather, time, brightness) at the shooting point. For example, in bad weather, misidentification of scratches is assumed. Therefore, if the coordinates (x1, y1) on December 1 are bad weather, the evaluation index is set to x0.9. Similarly, the evaluation index of the detection image 2021 taken in the evening or in a dark environment may be multiplied by a preset weighting coefficient. The value of such a weighting coefficient can be arbitrarily set according to the degree of influence of the conditions of the external environment.

As described above, according to the identification device 1 according to the present embodiment, the identification unit 173 of the terminal device 10 identifies the presence or absence of damage to the photographed image 161 captured by the photographing unit 150 of the terminal device 10. Then, of the captured images 161 and only the detected image 2021 in which damage is detected in the terminal device 10 is transmitted to the information processing server 20.
Therefore, it is not necessary to transmit all the captured images 161 out of the enormous amount of captured images 161 taken from the terminal device 10 to the information processing server 20, and damage corresponding to some of the captured images 161 is detected. It suffices to transmit only the detected detection image 2021 from the terminal device 10 to the information processing server 20. As a result, the amount of data communication from the terminal device 10 for photographing can be greatly suppressed.

Further, the position determination unit 2033 of the information processing server 20 determines whether or not the photographed damage positions are the same, and the evaluation unit 2034 integrates the information on the road surface damage in chronological order. , Evaluate the presence or absence of road surface damage.
Therefore, it is possible to comprehensively determine the presence or absence of damage from a large number of captured images 161 instead of the determination by one captured image 161 and to accurately detect the presence or absence of damage to the road surface.

Further, the evaluation unit 2034 calculates the existence probability of the road surface damage by integrating the information on the road surface damage in an arbitrary period for each coordinate position where the road surface damage is photographed in chronological order. As a result, it is possible to statistically determine the presence or absence of damage while absorbing the erroneous determination in the identification of damage.

Further, since the evaluation unit 2034 weights the existence probability by using the information about the external environment, it is possible to perform highly accurate identification in consideration of the difference in the conditions at the time of shooting.

Further, the position determination unit 2033 determines whether or not the detection image 2021 and the damage on the road surface photographed by the plurality of detection images 2021 are the same location by using the position information in which the detection image 2021 is captured. This makes it possible to confirm how the damage at the same location changes by using a large number of captured images 161.

In addition, the position determination unit 2033 determines whether or not the damage to the road surface captured in the plurality of detection images 2021 is the same location by using the information regarding the traveling route of the vehicle. As a result, it is possible to accurately grasp the position of the damage with respect to the time when the image was taken in consideration of the traveling direction of the vehicle.

In addition, the identification unit 173 determines whether or not the damage detected in the detection image 2021 is located in the road area. Therefore, damage to the sidewalk and surrounding equipment can be excluded as noise.

Further, the information processing server 20 identifies the detected image 2021 using the trained model to identify whether or not the road surface is damaged, and whether or not the road surface damage is located in the road area. Further, a re-identification unit 2032 for performing the determination again is provided. Therefore, damage can be identified with higher accuracy.

Further, the re-identification unit 2032 re-identifies the presence or absence of damage to the road surface according to an identification standard different from that of the identification unit 173. Therefore, for example, by setting the identification standard by the re-identification unit 2032 higher than the identification standard by the identification unit 173, the captured image 161 that is likely to be damaged in the primary identification by the terminal device 10 is picked up, and the information processing server 20 In the secondary discrimination in, various discriminations can be made, such as excluding noise.

(Modification example)
In the above-described embodiment, an example is shown in which the information on the road surface damage integrated by the evaluation unit 2034 in chronological order is the information on the presence or absence of damage at the position where the detection image 2021 is taken. No. For example, the evaluation unit 2034 may integrate the detected image 2021 as information on road surface damage in chronological order. In this case, it is possible to obtain a set (moving image) of images depicting the state of damage growing in chronological order.

Further, in the above-described embodiment, an example of evaluating the captured image 161 captured by the terminal device 10 mounted on one vehicle has been shown, but the present invention is not limited to such an embodiment. For example, the photographed image 161 photographed by a large number of vehicles may be evaluated.
Specifically, the terminal device 10 identifies the presence or absence of damage to the captured images 161 captured by a large number of pre-registered users. Then, among the plurality of captured images 161 the detected image 2021 in which damage is detected is transmitted from each terminal device 10 to the information processing server 20, and the information processing server 20 performs each of the above-mentioned evaluation processes. May be good. If such a configuration is adopted, by adopting a dual-purpose device such as a general-purpose mobile terminal or a drive recorder as the terminal device 10, a large number of images are taken at a huge number of shooting points on various traveling routes. The detected image 2021 of the above can be easily acquired. As a result, the accuracy of the calculated probability of existence of road surface damage can be ensured.

Although some embodiments of the present disclosure have been described above, these embodiments can be implemented in various other embodiments, and various omissions, replacements, and modifications are made without departing from the gist of the invention. It can be performed. These embodiments and modifications thereof shall be included in the scope of the invention described in the claims and the equivalent scope thereof, as well as in the scope and gist of the invention.

<Additional notes>
The matters described in the embodiment are added below.

(Appendix 1)
An identification device 1 including a terminal device 10 and an information processing server 20.
The terminal device 10
The shooting unit 150, which is mounted on the vehicle and takes an image of the road surface,
The identification unit 173, which identifies the presence or absence of damage to the road surface by identifying the image captured by the photographing unit 150 using the trained model,
Among the images, a transmission unit 172 that transmits the detection image 2021 in which the identification unit 173 has detected damage to the information processing server 20 is provided.
The information processing server 20
The position determination unit 2033 that determines whether or not the damage to the road surface captured by the plurality of detection images 2021 transmitted from the transmission unit 172 is the same location,
An identification device 1 comprising an evaluation unit 2034 that integrates information on road surface damage determined to be the same location by the position determination unit 2033 in chronological order and evaluates the presence or absence of road surface damage.

(Appendix 2)
The identification device 1 according to (Appendix 1), wherein the evaluation unit 2034 calculates the existence probability of road surface damage by integrating information on road surface damage in chronological order.

(Appendix 3)
The information processing server 20
Get information about the external environment in which the vehicle travels
Evaluation unit 2034
The identification device 1 according to (Appendix 2), which weights the existence probability using information on the external environment.

(Appendix 4)
The identification unit 173
Associate the information about the captured position with the detection image 2021 and
The identification according to any one of (Appendix 1) to (Appendix 3), wherein the information on the road surface damage includes information on the presence or absence of damage at the position where the detected image 2021 was taken and at least one of the detected images 2021. Device 1.

(Appendix 5)
The identification unit 173
Associate the information about the captured position with the detection image 2021 and
The position determination unit 2033
Using the position information in which the detected images 2021 and the detected images 2021 are captured, it is determined whether or not the damage to the road surface captured in the plurality of detected images 2021 is the same location, from (Appendix 1) to (Appendix 4). The identification device 1 according to any one.

(Appendix 6)
The information processing server 20
Get information about the vehicle's travel route,
The position determination unit 2033
The identification device 1 according to any one of (Appendix 1) to (Appendix 5), which determines whether or not the damage to the road surface captured in the plurality of detected images 2021 is the same location by using the information on the traveling route of the vehicle. ..

(Appendix 7)
The identification unit 173
The identification device 1 according to any one of (Appendix 1) to (Appendix 6), which determines whether or not the damage detected in the detection image 2021 is located in the road area.

(Appendix 8)
The information processing server 20
By identifying the plurality of detected images 2021 transmitted from the transmission unit 172 using the trained model, it is possible to identify whether or not the road surface is damaged and whether or not the road surface damage is located in the road area. The identification device 1 according to any one of (Appendix 1) to (Appendix 7), further comprising a re-identification unit 2032 for re-determining.

(Appendix 9)
The identification device 1 according to (Appendix 8), wherein the re-identification unit 2032 reidentifies the presence or absence of damage to the road surface according to an identification standard different from that of the identification unit 173.

(Appendix 10)
The information processing server 20
Get information about the vehicle's travel route,
The identification device 1 according to any one of (Appendix 1) to (Appendix 9), wherein the evaluation unit 2034 integrates information regarding the presence or absence of road surface damage at an arbitrary period for each coordinate position where the road surface damage is photographed. ..

(Appendix 11)
An identification program that is executed by an identification system equipped with a processor.
To the processor
A step (step S110) in which the photographing unit 150 mounted on the vehicle captures an image of the road surface,
A step (step S111) of identifying the presence or absence of damage to the road surface by identifying the image captured by the photographing unit 150 using the trained model.
A step (step S116) of transmitting the detection image 2021 in which the identification unit 173 has detected damage to the information processing server 20 among the images.
A step (step S122) of determining whether or not the road surface damages captured by the plurality of detection images 2021 transmitted from the transmission unit 172 are the same location, and
A program for executing a step (step S123) of integrating information on road surface damage determined to be the same location in chronological order and evaluating the presence or absence of road surface damage.

(Appendix 12)
It is a method executed by the identification device 1 provided with a processor.
The processor,
A step (step S110) in which the photographing unit 150 mounted on the vehicle captures an image of the road surface,
A step (step S111) of identifying the presence or absence of damage to the road surface by identifying the image captured by the photographing unit 150 using the trained model.
A step (step S116) of transmitting the detection image 2021 in which the identification unit 173 has detected damage to the information processing server 20 among the images.
A step (step S122) of determining whether or not the road surface damages captured by the plurality of detection images 2021 transmitted from the transmission unit 172 are the same location, and
A method of executing a step (step S123) of integrating information on road surface damage determined to be the same location by the position determination unit 2033 in chronological order and evaluating the presence or absence of road surface damage.

1 ... Identification device 10 ... Terminal device 150 ... Imaging unit 170 ... Control unit 171 ... Input operation reception unit 172 ... Transmission / reception unit 173 ... Identification unit 174 ... Display processing unit 20 ... Information processing terminal 203 ... Control unit 2031 ... Transmission / reception unit 2032 ... Re-identification unit 2033 ... Position determination unit 2034 ... Evaluation unit

Claims (15)

An identification device including a terminal device and an information processing server.
The terminal device is
A shooting unit that is mounted on the vehicle and captures images of the road surface,
An identification unit that identifies the presence or absence of damage to the road surface by performing identification using a trained model on the image captured by the photographing unit.
A transmission unit that transmits the detected image in which the identification unit detects damage among the images to the information processing server without transmitting all the images captured by the photographing unit from the terminal device to the information processing server. And with
The information processing server
A position determination unit that determines whether or not the damage to the road surface captured by the plurality of detection images transmitted from the transmission unit is the same location, and
An identification device including an evaluation unit that integrates information on damage to the road surface determined to be the same location by the position determination unit in chronological order and evaluates the presence or absence of damage to the road surface. An identification device including a terminal device and an information processing server.
The terminal device is
A shooting unit that is mounted on the vehicle and captures images of the road surface,
An identification unit that identifies the presence or absence of damage to the road surface by performing identification using a trained model on the image captured by the photographing unit.
Among the images, a transmission unit that transmits a detection image in which the identification unit has detected damage to the information processing server is provided.
The information processing server
A position determination unit that determines whether or not the damage to the road surface captured by the plurality of detection images transmitted from the transmission unit is the same location, and
Information on the presence or absence of damage to the road surface determined to be the same location by the position determination unit is integrated in chronological order for an arbitrary period to calculate the existence probability indicating whether or not the damage exists as a probability. An identification device including an evaluation unit. The identification device according to claim 1 or 2 , wherein the evaluation unit calculates the existence probability of the road surface damage by integrating information on the road surface damage in chronological order. The information processing server
Get information about the external environment in which the vehicle travels
The evaluation unit
The identification device according to claim 2 or 3 , wherein the existence probability is weighted by using the information about the external environment. The identification unit
The information about the captured position is associated with the detected image,
The identification device according to any one of claims 1 to 4, wherein the information regarding the damage to the road surface includes information regarding the presence or absence of damage at the position where the detected image was taken and at least one of the detected images. .. The identification unit
The information about the captured position is associated with the detected image,
The position determination unit
Any one of claims 1 to 5 , which determines whether or not the damage to the road surface captured in the plurality of detected images is the same location by using the detected image and the position information in which the detected image was captured. The identification device described in the section. The information processing server
Obtain information about the traveling route of the vehicle and
The position determination unit
The identification device according to any one of claims 1 to 6 , wherein it is determined whether or not the damage to the road surface captured in the plurality of detection images is the same location by using the information on the traveling route of the vehicle. The identification unit
The identification device according to any one of claims 1 to 7 , which determines whether or not the damage detected in the detection image is located in the road area. The information processing server
By identifying the plurality of detected images transmitted from the transmission unit using the trained model, the presence or absence of damage to the road surface and the damage to the road surface are located in the road region. The identification device according to any one of claims 1 to 8 , further comprising a re-identification unit for re-determining whether or not the image is present. The identification device according to claim 9 , wherein the re-identification unit reidentifies the presence or absence of damage to the road surface according to an identification standard different from that of the identification unit. The information processing server
Obtain information about the traveling route of the vehicle and
The identification device according to any one of claims 1 to 10 , wherein the evaluation unit integrates information regarding the presence or absence of road surface damage in an arbitrary period for each coordinate position where the road surface damage is photographed. An identification program that is executed by an identification system equipped with a processor.
To the processor
The step that the shooting unit mounted on the vehicle takes an image of the road surface,
A step of identifying the presence or absence of damage to the road surface by identifying the image captured by the photographing unit using a trained model.
A step of transmitting the detected image in which damage is detected in the identification step among the images to the information processing server without transmitting all the images captured by the photographing unit.
A step of determining whether or not the damage to the road surface captured in the plurality of detected images transmitted in the transmission step is the same location, and
A program for executing a step of integrating information on damage to the road surface determined to be the same location in the determination step in chronological order and evaluating the presence or absence of damage to the road surface. It is a method performed by an identification device equipped with a processor.
The processor,
The step that the shooting unit mounted on the vehicle takes an image of the road surface,
A step of identifying the presence or absence of damage to the road surface by identifying the image captured by the photographing unit using a trained model.
A step of transmitting the detected image in which damage is detected in the identification step among the images to the information processing server without transmitting all the images captured by the photographing unit.
A step of determining whether or not the damage to the road surface captured in the plurality of detected images transmitted in the transmission step is the same location, and
A method of executing a step of integrating information on damage to the road surface determined to be the same location in the determination step in chronological order and evaluating the presence or absence of damage to the road surface. An identification program that is executed by an identification system equipped with a processor.
To the processor
The step that the shooting unit mounted on the vehicle takes an image of the road surface,
A step of identifying the presence or absence of damage to the road surface by identifying the image captured by the photographing unit using a trained model.
Among the images, a step of transmitting a detection image in which damage is detected in the identification step to an information processing server, and a step of transmitting the damage to the information processing server.
A step of determining whether or not the damage to the road surface captured in the plurality of detected images transmitted in the transmission step is the same location, and
Information on the presence or absence of damage to the road surface determined to be the same location in the determination step is integrated in chronological order for an arbitrary period to calculate the existence probability indicating whether or not the damage exists as a probability. A program that executes steps and. It is a method performed by an identification device equipped with a processor.
The processor,
The step that the shooting unit mounted on the vehicle takes an image of the road surface,
A step of identifying the presence or absence of damage to the road surface by identifying the image captured by the photographing unit using a trained model.
Among the images, a step of transmitting a detection image in which damage is detected in the identification step to an information processing server, and a step of transmitting the damage to the information processing server.
A step of determining whether or not the damage to the road surface captured in the plurality of detected images transmitted in the transmission step is the same location, and
Information on the presence or absence of damage to the road surface determined to be the same location in the determination step is integrated in chronological order for an arbitrary period to calculate the existence probability indicating whether or not the damage exists as a probability. Steps and how to perform.

Images