JP7372616B2 - Stone and gravel detection system, stone and gravel detection method and program - Google Patents

Description

The present invention relates to a gravel detection system, a gravel detection method, and a program.

BACKGROUND ART For the purpose of disaster prevention, for example, objects on the ground such as stones and gravel on a riverbed are detected and a report is created in which the size and number of objects are recorded. In particular, it is extremely important to detect the quality (size) of sediment that can cause damage downstream if washed away by floods.

Generally, to detect stones and gravels, first, a photograph of a mountain stream or riverbed is taken through aerial photography or ground survey, and then a worker visually extracts and identifies the stones and stones photographed in the photograph. For stones and gravels that were deemed necessary to be measured, the size was determined visually, the number was counted, and a report was created. Furthermore, when it was necessary to identify the location of stones and gravels, workers would go to mountain streams and riverbeds to survey the locations of stones and gravels.

Patent Document 1 discloses a technology that allows an unmanned flying vehicle to accurately and efficiently collect aerial survey images that can be used to generate highly accurate three-dimensional images. Furthermore, Patent Document 2 discloses that even if the subject is far away and the distance to the subject is large, the distance to the subject is accurately measured, and distance measurement data and scale data based on the distance measurement data are accurately displayed on the photo screen. A technique for imprinting has been disclosed.

Japanese Patent Application Publication No. 2006-27331 Japanese Patent Application Publication No. 10-281765

However, the above-mentioned method of creating a report that describes the size and number of stones and gravels is manual, which requires time and effort, and the cost of creating the report is also high.

Furthermore, in the case of manually creating a report, there is a possibility that the criteria for determining the size of stones and gravel may differ depending on the worker.

In addition, it takes time to train the workers who create the reports.

The present invention has been made in view of the above-mentioned problems, and it is possible to easily and quickly create a report that describes the size and number of objects on the ground, and as a result, the report can be created at a low cost. The object of the present invention is to provide a stone and gravel detection system, a stone and gravel detection method, and a program.

In order to solve the above problems, a stone and gravel detection system according to one aspect of the present invention combines a plurality of captured images of the ground containing stone and gravel to generate an overhead image viewed from above. a machine learning section that detects the position of the stones photographed in the bird's-eye view image; and a display control for displaying a screen that superimposes and displays the position of the stones detected by the machine learning section on the bird's-eye view image. A map/image display section that generates a signal, a major axis/minor axis calculation section that calculates the major axis and minor axis of the gravel detected by the machine learning section, and a major axis/minor axis calculation section that calculates the major axis and minor axis of the stone gravel detected by the machine learning section. The present invention includes a histogram generation section that generates a histogram of the size of gravel based on the major axis and the minor axis, and a report generation section that generates a report using the histogram generated by the histogram generation section.

According to the present invention, it is possible to easily and quickly create a report describing the size and number of objects on the ground, and as a result, the report can be created at low cost.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the whole structure of the stone and gravel detection system which is one Example of this invention. FIG. 2 is a diagram showing a basic communication flow between a data processing server and a user in the stone and gravel detection system. FIG. 3 is a diagram illustrating an example of an orthoimage generated by a three-dimensional data generation unit of a data processing server. It is a figure which shows an example of the screen display of the detection result of stone and gravel displayed on a user terminal. FIG. 3 is a diagram illustrating a procedure in which the machine learning unit of the data processing server generates a machine learning model. FIG. 6 is a diagram illustrating an example of setting input for a region of interest. FIG. 7 is a diagram illustrating another example of setting input for a region of interest. FIG. 7 is a diagram showing still another example of setting input of a region of interest. FIG. 7 is a diagram showing still another example of setting input of a region of interest. FIG. 7 is a diagram illustrating an example of setting input for an erroneous recognition area. FIG. 3 is a diagram illustrating an example of a histogram generated by a histogram generation unit of a data processing server. FIG. 3 is a diagram showing an example of an investigation report created by a report generation unit of a data processing server.

Embodiments of the present invention will be described below with reference to the drawings. The embodiments described below do not limit the claimed invention, and all of the elements and combinations thereof described in the embodiments are essential to the solution of the invention. is not limited.

The stone and gravel detection system of this embodiment is a system that uses images to detect ground objects such as stone and gravel in mountain streams and riverbeds, measures the size and number of these ground objects, and creates a report. The stone and gravel detection system of this embodiment is applied to a system that detects stone and gravel on a riverbed, but ground objects detected by the stone and gravel detection system are not limited to stones, but include sand, trees, and fallen trees that are smaller than stone and gravel. (driftwood) or other ground objects. Furthermore, the input data for extracting ground objects is not limited to images, but may also be point cloud data or other data that can grasp spatial conditions.

<Overall system configuration>
FIG. 1 is a diagram showing the overall configuration of a stone and gravel detection system that is an embodiment of the present invention.

As shown in FIG. 1, the stone and gravel detection system 1 includes a data processing server 3, and the data processing server 3 may be configured by one or more server computer machines. The data processing server 3 provides a service to each of one or more users to help them inspect objects specified by each user using images and compile the inspection results into a report. provide.

The data processing server 3 can perform data communication via the web server 5 with a web browser of an information processing terminal (hereinafter referred to as user terminal) 7 such as a personal computer or a smartphone used by each user. In FIG. 1, only one user terminal 7 is illustrated for convenience, and the details of this system will be described below by taking as an example a case where one user uses the data processing server 3.

The data processing server 3 includes processing components such as a processing request reception section 11, a data storage/search section 13, and an analysis/calculation section 15.

The processing request receiving unit 11 receives various processing requests from users (for example, various processing requests for performing an inspection, various processing requests for creating a report, etc.), and performs the requested processing. Pass the request to a processing component that does the following:

The data storage/retrieval unit 13 stores data such as maps, images, test results, and reports of various inspection targets, and stores data such as maps, images, test results, and reports of various inspection targets, and stores data such as maps, images, test results, and other data in response to data requests from other processing components. Read data and pass it to its processing component. Further, the data storage/retrieval unit 13 stores programs such as firmware executed when the data processing server 3 is started, and an operating system that controls the basic operations of the data processing server 3.

The analysis/calculation unit 15 controls the entire data processing server 3. Furthermore, when the data processing server 3 is started, the firmware stored in the data storage/retrieval unit 13 is executed by the analysis/calculation unit 15, so that the analysis/calculation unit 15 functions as a processing component described below. Execute.

The analysis/calculation section 15 includes an image registration section 17, an image combination section 19, a three-dimensional data generation section 21, a machine learning section 23, a spatial filter section 25, a major axis/minor axis calculation section 27, a histogram generation section 29, and a map/image generation section 29. Display section 31, report generation section 33, report display section 35, report output section 37, land cover classification section 39, land cover analysis section 41, spatial filter inference section 43, report format registration section 45, report format It has processing components such as an analysis unit 47.

The image registration unit 17 captures an image taken from the user terminal 7 of the ground containing stones and gravels as a detection target (this includes not only a photographic image using general visible light but also a photographic image using non-visible light such as infrared light). (which may include photographic images and other types of images) and register those images in the data storage/retrieval unit 13 in association with the geographical information (latitude, longitude, etc.) of the images and the user. do.

Usually, a large number of stones and gravels, which are objects to be detected, exist on the ground. Therefore, a large number of captured images are registered for a detection target space such as a riverbed. For example, when a camera mounted on an unmanned flying vehicle takes a picture of a vast detection target space such as a riverbed, the camera takes a large number of high-quality images, each of which captures a portion of the riverbed. The camera stores geographic information when photographing a part of the riverbed, linking it to the photographed image.

Therefore, the image registration unit 17 registers each photographed image in association with the geographical information at the time of photographing. At this time, the image registration unit 17 associates and registers one of the geographic information linked to each captured image as geographic information representative of the riverbed that is the detection target space. It's okay.

The image combining unit 19 generates a mosaic image of the entire detection target space by combining a large number of registered captured images.

If the input data (image) used by the image combination unit 19 to generate a mosaic image satisfies a predetermined condition, the three-dimensional data generation unit 21 performs SfM (Structure from motion) processing on this input data. Then, three-dimensional data of the captured detection target space is generated, and an orthoimage is generated from this three-dimensional data. This ortho image is a bird's-eye view of the detection target space (river bed) from above. An ortho image is an image that is orthographically projected by correcting distortions caused by camera tilt and relative height based on photogrammetry technology. Since the method of generating three-dimensional data using SfM and SfM and generating an orthoimage based on this three-dimensional data is well known, a description thereof will be omitted here.

The orthogonal image is generated to ensure ground object detection accuracy (especially detection accuracy of size and position), which will be described later. Thereafter, based on this orthogonal image (overhead image), the stone and gravel, which is the detection target object existing in the detection target space, is inspected. The three-dimensional data and ortho image are stored in the data storage/retrieval section 13.

The machine learning unit 23 detects the position of the stone gravel photographed in the orthoimage. More specifically, the machine learning unit 23 uses a machine learning model to detect the position where the stone gravel is photographed from the orthoimage. Since the method for generating a machine learning model is well known, a description thereof will be omitted here.

The spatial filter section 25 performs spatial filtering processing on the orthoimage generated by the three-dimensional data generation section 21 based on the region of interest set and input by the user via the user terminal 7. The region of interest is an area in which the long axis/breadth axis calculation unit 27 and the histogram generation unit 29, which will be described later, calculate the long axis and short axis of stones and the number of stones. Set. Spatial filtering processing means setting a region of interest in the orthoimage to limit the region to be subjected to calculation processing by the major axis/minor axis calculation unit 27 or the like.

The major axis/minor axis calculating unit 27 calculates the detection by the machine learning unit 23 based on the output result of the machine learning unit 23, that is, the stone and gravel detection result from the ortho image within the region of interest set by the spatial filter unit 25. Calculate the major axis and minor axis of the gravel.

The histogram generation unit 29 generates a histogram of the size of stones and a cumulative histogram of the sizes of stones based on the calculation results of the long axis and short axis of the stones calculated by the long axis/breadth axis calculation unit 27. .

The map/image display unit 31 sends an ortho image displayed on the screen of the user terminal 7 to the Web server 5 in response to a processing request from the user. The web server 5 sends a work screen incorporating the orthoimage to the user terminal 7, and the user terminal 7 displays this work screen on the screen of a web browser.

In particular, the map/image display section 31 can generate a display control signal for displaying a screen that displays the position of the stone gravel detected by the machine learning section 23 superimposed on the orthoimage.

In response to a request from a user, the report generation unit 33 uses data such as an ortho image, a stone and gravel position detection result, a histogram of stone and gravel sizes, and a cumulative histogram of stone and gravel sizes to determine the size of stones in the region of interest. Create a gravel investigation report. At this time, the report generating section 33 accepts a comment input by the user via the user terminal 7 via the processing request receiving section 11, and creates an investigation report including this comment.

The report display section 35 sends the investigation report created by the report generation section 33 to the Web server 5 in response to a request from a user. The web server 5 sends the investigation report to the user terminal 7, and the user terminal 7 displays this work screen on the screen of the web browser.

The report output unit 37 sends the investigation report created by the report generation unit 33 to the Web server 5 in response to a request from a user. The web server 5 allows the user to download the investigation report to the user terminal 7.

The land cover classification unit 39 estimates the land cover of each region or each pixel in the orthogonal image from the spectral radiation characteristics and sub-vixel texture information of each pixel in the orthogonal image. The processing itself of the land cover classification unit 39 is known, and for example, the technique disclosed in Japanese Patent Application Laid-Open No. 8-320930 can be applied.

The land cover analysis unit 41 examines the correlation between the spatial filter, which is the region of interest set and input by the user, and the land cover classification result classified by the land cover classification unit 39. As an example, the land cover analysis unit 41 extracts the land cover estimation result of each region or each pixel in the orthoimage within the spatial filter.

When creating a survey report for a detection target space that has not yet been analyzed in the stone and gravel detection system 1, the spatial filter inference unit 43 selects the optimal one based on the land cover analysis results by the land cover analysis unit 41. Set up a spatial filter that seems to work.

The report format registration unit 45 registers a procedure for deriving the format (sample) and analysis of the investigation report that the user desires to create, input by the user via the user terminal 7 . The procedure for deriving the format and analysis of the registered investigation report is stored in the data storage/retrieval unit 13.

The report format analysis section 47 analyzes the format of the investigation report registered by the report format registration section 45 and stored in the data storage/retrieval section 13, and converts the format into a text section, a graph section, a photograph section, etc. The information is classified and an investigation report template having substantially the same ratio as the classification result is generated.

<System operation>
FIG. 2 shows the basic communication flow between the data processing server 3 and the user in the stone and gravel detection system 1.

In FIG. 2, in step 51, the user 9 uploads a captured image, which will be the basis for creating an investigation report, to the data processing server 3 via the user terminal 7. In step 53, the image registration unit 17 of the data processing server 3 registers the uploaded photographed image and stores it in the data storage/search unit 13.

In step 55, the image combining unit 19 combines the captured images registered in step 53 to generate a mosaic image of the detection target space. Thereafter, the three-dimensional data generation section 21 performs SfM processing on the mosaic image generated by the image combination section 19 to generate three-dimensional data and an orthoimage. FIG. 3 shows an example of an orthoimage P generated by the three-dimensional data generation unit 21.

In step 57, based on the orthoimage generated by the three-dimensional data generation unit 21, the machine learning unit 23 detects the position of the stone gravel photographed in the orthoimage. Then, in step 59, the map/image display unit 31 causes the user terminal 7 to display a screen in which the position of the detected gravel is superimposed on the orthoimage.

FIG. 4 shows an example of a screen display of the stone and gravel detection results displayed on the user terminal 7. The position of the gravel T detected by the machine learning unit 23 is displayed as a bar B superimposed on the orthoimage P. The length of the bar B is proportional to the approximate size of the gravel T detected by the machine learning unit 23.

FIG. 5 shows a procedure in which the machine learning unit 23 generates a machine learning model. The model generation instructor (teacher) specifies the position where the stone gravel is photographed in the orthoimage P using the rectangular area R. The machine learning unit 23 generates a machine learning model based on the feature amount of the rectangular region R.

In step 61, the user 9, while looking at the screen displayed on the user terminal 7, inputs settings for the region of interest in which the position and size of the gravel should be calculated. The region of interest setting input in step 61 is performed using, for example, an input device such as a mouse provided on the user terminal 7.

FIG. 6 shows an example of the region of interest setting input performed in step 61. First, the user 9 specifies a length 81 of n meters (200 m as an example) upstream and downstream from the planned survey area construction site 80 in a mountain stream or riverbed that is a detection target space. Next, the user 9 sets the width of the region of interest. At this time, areas such as roads along rivers and other areas where stone and gravel are not to be counted are excluded and specified. Furthermore, the user 9 specifies an area (exclusion area) 82, such as a group of concrete blocks, in which stones and gravels are not counted. As a result, the setting input for the region of interest 83 is performed. In addition, the user 9 additionally notes a region 84 in the region of interest 83 where stones and gravels that were not detected by the machine learning unit 23 are photographed.

In step 63, the spatial filter unit 25 performs spatial filtering processing on the orthoimage based on the region of interest 83 set and input by the user. In step 65, the major axis and minor axis calculation unit 27 counts the major axis and minor axis of the stones within the region of interest 83, and further counts the number of stones.

The user 9 can modify the region of interest 83 for which the settings were input in step 61 in step S66. Hereinafter, a method for correcting the region of interest 83 will be described with reference to FIGS. 7 to 10.

As shown in FIG. 7, the user 9 can draw a line segment on the aerial image, which is a planned site 80 for planning, surveying, and designing, so as to cross a mountain stream. (This line segment can simulate the reference line for mountain stream surveys when constructing structures such as erosion control dams and planning and designing various countermeasures.) Then, as the region of interest 83, the upstream side of the line segment and Two rectangular areas having the line segment as one side on the downstream side and having a specified size of 81 (for example, each having a length of 200 m and a width equal to the length of the line segment) are automatically set. These two rectangular areas are set as the region of interest 83.

Then, in step S65, the number and number ratio of stones and gravels existing in the region of interest 83 for each of the plurality of size levels are automatically calculated, and the tabulated results are displayed on the screen (see the table at the bottom right of FIG. 7). ). This tally result displays a color mark specific to the size level of the gravel.

Furthermore, each stone and gravel existing within the region of interest 83 is displayed on the aerial image with a unique color corresponding to the size level to which it corresponds. The user 9 can visually grasp how the level and size of stones are distributed within the region of interest.

The user 9 understands the characteristics of the mountain stream in terms of the quality of the gravel by looking at the tally results, the mountain stream investigation range on the aerial image, the distribution by size of gravel in the region of interest 83, etc. It can be used for planning and designing of erosion control dams, etc., and for various studies.

The user can change the region of interest 83 as follows.

For example, as shown in FIG. 8, while fixing the reference line (line segment 80) for the mountain stream survey, the directions of the regions of interest 83 on the upstream and downstream sides can be changed individually (the rectangular region is a parallelogram). area). It can accommodate bends in the river.

Further, as shown in FIG. 9, while the reference line (line segment 80) for mountain stream investigation is fixed, the length 81 of the region of interest can be changed individually in the directions of the region of interest on the upstream and downstream sides thereof. This is easy to deal with when there is a structure near a mountain stream or when the width of the mountain stream changes.

Furthermore, as shown in FIGS. 8 and 9, by specifying the exclusion area 82 on the aerial photographed image, the exclusion area 82 can be excluded from the region of interest 83.

Furthermore, although not shown, the length and position of the reference line (line segment 80) for the mountain stream survey can be changed.

Furthermore, as shown in FIG. 10, when an area that does not correspond to gravel on the aerial photographed image is automatically recognized and displayed as gravel, the user 9 can check the display of the erroneously recognized area (for example, the surrounding frame 84). You can specify this and make a modification to exclude it from the gravel. Conversely, although not shown, if an area corresponding to gravel is not recognized automatically, specify that area (for example, by surrounding it with a frame) and make corrections to add it to the gravel. be able to.

When the user 9 changes the above-mentioned region of interest 83 or corrects misrecognition, the aggregation is performed again in steps 63 and 65 each time, and the aggregation results are displayed, the region of interest is displayed, and the stones and gravel within the region of interest are displayed. The display is updated. Then, the stones are displayed in different colors according to their sizes, and the number and number ratio of the stones existing in the region of interest 83 for each size level are calculated and displayed on the screen.

After the region of interest 83 is updated, the histogram generation unit 29 generates a histogram of the size of the gravel based on the calculation results of the major axis and minor axis of the gravel calculated by the major axis/minor axis calculation unit 27. , generate a cumulative histogram of gravel sizes.

FIG. 7 shows an example of a histogram generated by the histogram generation unit 29. FIG. 7 is an example of a screen displayed on the user terminal 7 based on an instruction from the user 9 after the histogram generation unit 29 generates a histogram.

This screen displays a histogram 90 of stone and gravel sizes and a cumulative histogram 91 of stone and gravel sizes. The screen also includes an area 92 for displaying the number of stones and gravels detected by the machine learning unit 23, an area 93 for displaying the average diameter of the stones calculated by the major axis/short axis calculation unit 27, and an area 93 for displaying the average diameter of the stones and gravels detected by the machine learning unit 23. An area 94 that displays the diameter (D50) of stones and gravels corresponding to 50% of the total number of stones and gravels, and a region 94 that displays the diameter (D95) of stones and gravels that corresponds to 95% of the total number of stones and gravels detected. A region 95 is provided.

In step 67, the report generation unit 33 creates an investigation report. FIG. 8 shows an example of an investigation report created by the report generation unit 33. The investigation report includes a text section 100 in which bibliographic information and detection results are described, a photograph section 101 in which an image in which the stone and gravel detection results are superimposed on an ortho image, and two types of histograms are displayed. It has a graph section 102.

In step 69, the report display unit 35 displays the investigation report on the screen of the user terminal 7. In step 71, the user 9 edits the investigation report by inputting comments on the investigation report via the user terminal 7. The comment input by the user 9 is added to the text section 100.

At step 73, the report generation section 33 stores the investigation report in the data storage/retrieval section 13. In step 75, the report output unit 37 sends the investigation report in a downloadable state to the Web server 5, and the user 9 downloads the investigation report.

<System effect>
According to this embodiment configured in this way, the position of the stones is detected based on the photographed image, the size and number of stones photographed in the region of interest that has been inputted is counted, and further, A histogram of gravel sizes can be generated and a survey report can be created based on these.

Therefore, according to this embodiment, it is possible to easily and quickly create a report that describes the size, number, and position coordinates of stones and stones, and as a result, it is possible to create the report at low cost.

Variant

Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations.

Moreover, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing, for example, an integrated circuit. Further, each of the above-mentioned configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk, an SSD, or a recording medium such as an IC card, an SD card, or a DVD.

Further, the control lines and information lines are shown to be necessary for explanation purposes, and not all control lines and information lines are necessarily shown in the product. In reality, almost all components may be considered to be interconnected.

1... Stone and gravel detection system 3... Data processing server 5... Web server 7... User terminal 9... User 11... Processing request reception section 13... Data storage/search section 15... Analysis/calculation section 17... Image registration section 19... Image combination Part 21...Three-dimensional data generation unit 23...Machine learning unit 25...Spatial filter unit 27...Long axis/breadth axis calculation unit 29...Histogram generation unit 31...Image display unit 33...Report generation unit 83...Region of interest 90...Histogram 91 ...Cumulative histogram P...Ortho image T...Stone gravel

Claims (5)

an image combining unit that combines a plurality of captured images of the ground including stones and gravel to generate an overhead image viewed from above;
a machine learning unit that detects the position of the stone gravel photographed in the bird's-eye view image;
a map/image display unit that generates a display control signal for displaying a screen that superimposes the position of the gravel detected by the machine learning unit on the bird's-eye view image;
a major axis/minor axis calculation unit that calculates the major axis and minor axis of the gravel detected by the machine learning unit;
a histogram generation unit that generates a histogram of the size of the gravel based on the major axis and minor axis of the gravel calculated by the major axis/minor axis calculation unit;
A stone and gravel detection system comprising: a report generation section that generates a report using the histogram generated by the histogram generation section. a 3D data generation unit that generates 3D data of a detection target space including the ground based on the captured image, and generates an orthoimage from this 3D data;
The stone and gravel detection system according to claim 1, wherein the machine learning unit detects the position of the stone and gravel based on the orthoimage. a processing request reception unit that accepts an input for setting a region of interest for the bird's-eye view image on which the position of the gravel is superimposed;
a spatial filter unit that performs spatial filtering processing on the bird's-eye view image on which the position of the gravel is superimposed based on the setting input of the region of interest;
The stone according to claim 1, wherein the major axis and minor axis calculation unit calculates the major axis and minor axis of the stone gravel detected by the machine learning unit from the bird's-eye view image in the region of interest. Gravel detection system. A gravel detection method performed by a gravel detection system, the method comprising:
A bird's-eye view image is created by combining multiple images taken of the ground containing stones and gravel, and
detecting the position of the stone gravel photographed in the bird's-eye view image;
generating a display control signal for displaying a screen that superimposes and displays the position of the detected stone gravel on the bird's-eye view image;
Calculate the major axis and minor axis of the detected gravel,
Generate a histogram of the size of the gravel based on the calculated major axis and minor axis of the gravel,
A stone and gravel object detection method that generates a report using the generated histogram. A computer program executed by a computer,
When the computer program is executed, the computer program includes: an image combining unit that combines a plurality of captured images of the ground including stones and gravel to generate a bird's-eye view of the ground from above;
a machine learning unit that detects the position of the stone gravel photographed in the bird's-eye view image;
a map/image display unit that generates a display control signal for displaying a screen that superimposes the position of the gravel detected by the machine learning unit on the bird's-eye view image;
a major axis/minor axis calculation unit that calculates the major axis and minor axis of the gravel detected by the machine learning unit;
a histogram generation unit that generates a histogram of the size of the gravel based on the major axis and minor axis of the gravel calculated by the major axis/minor axis calculation unit;
A computer program that causes the computer program to function as a report generation unit that generates a report using the histogram generated by the histogram generation unit.

Images