CA2959229C - Ground surface information analysis system and ground surface information analysis method - Google Patents

Abstract

A ground surface information analysis system for analyzing a ground surface using images includes the units of: acquiring two-dimensional image data including information about the regions of materials on a ground surface at plural times; acquiring three-dimensional image data including the configuration information of the ground surface at the plural times; specifying the respective regions of plural types of materials using the acquired two-dimensional image data; generating time-series multi-spectrum three-dimensional image data from the respective specified regions of the materials and the configurations specified by the acquired three-dimensional image data as combinations of the plural times and the plural types of materials, where the regions and the configurations correspond to the respective times included in the plural times; and calculating the variation amounts of the respective materials on the basis of plural time data pieces included in the generated time-series multi-spectrum three-dimensional image data.

Description

GROUND SURFACE INFORMATION ANALYSIS SYSTEM AND GROUND
SURFACE INFORMATION ANALYSIS METHOD
BACKGROUND OF THE INVENTION
The present invention relates to a ground surface information analysis system and a ground surface information analysis method.
Mining companies have needs for storing excavated minerals and fuels in stacked states and grasping the volumes of these minerals and fuels and the variations of the volumes in detail. Conventionally, in order to meet the above needs, these deposits are manually specified and the volumes of these deposits are manually measured. In addition, mining companies have needs for minutely grasping the road surface conditions of carrier roads in order to effectively observing and maintaining the carrier roads. Conventionally, also in this case, the road surfaces are manually evaluated. In such manual work, it is difficult to cover a wide area, and there is a demerit in that the accuracy of each work fluctuates depending on workers.
As a related technology, Japanese Patent Application Publication No. 2014-126321 discloses a technology which includes: an imaging step of stereoscopically imaging the surface part of wastes T that are disposed and deposited

2 in a hopper 5 prepared in a wastes disposal furnace from obliquely above; a three-dimensional configuration detecting step of acquiring the three-dimensional configuration of the surface part of wastes T on the basis of the stereoscopic image obtained by the step of imaging;
a column model converting step of converting the three-dimensional configuration of the surface part of wastes T
obtained by the step of three-dimensional configuration detecting into a column model composed of plural columns with various heights; and a volume variation amount calculating step of calculating the volume variation of wastes T from the variations of the heights in the column model acquired by the step of column model converting.
SUMMARY OF THE INVENTION
The volume variation amount of wastes can be calculated using the technology disclosed by Japanese Patent Application Publication No. 2014-126321. However, only the volume variation amount of the entirety of wastes can be calculated, and a technology with the use of which the volume variation amount of each material and the like can be calculated is not disclosed by Japanese Patent Application Publication No. 2014-126321.
One of the objects of the present invention is to calculate the variation amount of each material.

3 A representative ground surface information analysis system according to the present invention is a ground surface information analysis system that analyzes a ground surface using an image and includes the units of:
acquiring two-dimensional image data including information about the regions of materials on a ground surface at plural times; acquiring three-dimensional image data including the configuration information of the ground surface at the plural times; specifying the respective regions of plural types of materials using the acquired two-dimensional image data; generating time-series multi-spectrum three-dimensional image data from the respective specified regions of the materials and the configurations specified by the acquired three-dimensional image data as combinations of the plurality of times and the plurality of types of materials, where the regions and the configurations correspond to the respective times included in the plural times; and calculating the variation amounts of the respective materials on the basis of plural time data pieces included in the generated time-series multi-spectrum three-dimensional image data.
According to the present invention, the variation amounts of the respective materials can be calculated.
BRIEF DESCRIPTION OF THE DRAWINGS

4 Fig. 1 is a block diagram showing an example of the configuration of a ground surface information analysis system;
Fig. 2 is a block diagram showing an example of the hardware of the ground surface information analysis system;
Fig. 3 is a flowchart showing an example of the processing of the ground surface information analysis system;
Fig. 4 is a diagram showing an example of a ground coefficient calculation;
Fig. 5 is a diagram showing an example of ground estimation;
Fig. 6 is a diagram showing an example of time-series multi-spectrum three-dimensional information data;
Fig. 7 is a diagram showing examples of deposit models;
Fig. 8 is a flowchart showing an example of a deposit volume calculation;
Fig. 9 is a diagram showing an example of variation amount image; and Fig. 10 is a flowchart showing an example of maintenance judgment.
DESCRIPTION OF THE PREFERRED EMBODIMENT

Hereinafter, a preferred embodiment will be explained with reference to the accompanying drawings.
A ground surface information analysis system, which will be explained hereinafter, is a system that calculates

5 the volumes of stockpiled deposits at a site, at which minerals and fuels are stacked, in a mine that is a target region for high resolution remote sensing image data acquisition, and outputs deposit volume amount images and deposit variation amount images. Furthermore, the road surface conditions within the target region are also output as images. In such a way, although a target, a high resolution remote sensing image data of which is to be acquired and analyzed, is, for example, a deposit station in a mine, another kind of site may be a target.
Fig. 1 is a block diagram showing an example of the configuration of a ground surface information analysis system 100. The ground surface information analysis system 100 includes a remote sensing image data acquisition unit 102 for acquiring remote sensing image data 101a, and a three-dimensional information image acquisition unit 103 for acquiring three-dimensional information image data 101b. These image data pieces may be composed of plural types of data respectively. In addition, these image data pieces are acquired as time-series data respectively.

6 The ground surface information analysis system 100 includes a database generation unit 104 for generating a database 115 using the acquired image data. In the database 115, time-series image data is managed and, at the same time, data obtained by processing image data is also managed, where the data obtained by processing the image data will be explained later. Here, the formula of the management of the data of the database 115 is not limited to a specific formula, and it may be a formula used for a common database product.
The ground surface information analysis system 100 includes a material quality classification unit 105 for classifying the material qualities (types of material) of deposits using the acquired image data; a ground coefficient calculation unit 106 for calculating coefficients representing ground conditions; and a ground estimation unit 107 for estimating the elevations of grounds. Because, if the standards for plural deposits and grounds are considered to be spectra respectively, and if the spectra of the conditions of the deposits and the like are represented by time-series three-dimensional images by these units, time-series multi-spectrum three-dimensional images are generated, therefore it can be said that these units form a time-series multi-spectrum three-dimensional image generation unit 108.

7 Because how a deposit is piled up depends on the material quality of a deposit, the ground surface information analysis system 100 includes a deposit model generation unit 109 for generating deposit models according to the material qualities of deposits and a deposit volume calculation unit 110 for calculating the volumes of modeled deposits. Furthermore, the ground surface information analysis system 100 includes a variation amount image generation unit 111 for generating the calculated volume variations of the respective deposits and the variations of ground conditions as images.
In addition, the ground surface information analysis system 100 includes: a maintenance judgment unit 112 for judging whether maintenance should be done or not from the condition of a ground; a control unit 113 for supporting mines and engineering work by displaying the volume variations of deposits and controlling devices in the case where the maintenance should be done or on the basis of the volume variations of deposits; and an image update unit 114 for making the remote sensing image data acquisition unit 102 and the three-dimensional information image data acquisition unit 103 acquire new image data.
Fig. 2 is a block diagram showing an example of the hardware of the ground surface information analysis system 100. The ground surface information analysis system 100

8 may be a common computer, and includes a CPU (central processing unit) 201 for executing pieces of processing; a RAM (random access memory) 202 for storing programs and data used by the CPU to execute the pieces of processing;
and an I/F (interface) 203 that is connected with a remote sensing observation device 211 and a surface database 212 and used to acquire the remote sensing image data 101a and the three-dimensional information image data 101b.
Furthermore, the ground surface information analysis system 100 includes a memory unit 204 including an HDD
(hard disk drive), an SSD (solid state drive), and a flash memory that have larger capacities than the RAM 202 and are nonvolatile; and a display unit 205 that displays a prompt for inputs and processing results such as an LCD
display; and an input unit 206 such as a key board and a mouse. Data read from the RAM 202 may be written in the memory unit 204 or data read from the memory unit 204 may be written in the RAM 202 through the operation of the CPU
201. In addition, it is also conceivable that programs stored in the memory unit 204 are loaded into the RAM 202.
It is also conceivable that the display unit 205 and the input unit 206 are not implemented in the ground surface information analysis system 100 and these units are remotely controlled by the ground surface information analysis system 100 via a network (not shown).

9 Furthermore, it is conceivable that the ground surface information analysis system 100 includes not only one CPU
201 but also plural CPUs 201. In addition, it is conceivable that the ground surface information analysis system 100 is comprised of plural computers.
The respective units from the remote sensing image data acquisition unit 102 to the image update unit 114, which have been explained with reference to Fig. 1, may be realized in such a way that the CPU 201 executes programs which are stored in the RAM 202 or the memory unit 204 and correspond to the above units respectively. Furthermore, the above units may be respectively realized by combining plural CPUs 201 and plural RAMs 202 one by one. Therefore, the following descriptions, in which the operations of the above units are mainly explained respectively, may be replaced with descriptions in which the operations of the plural CPUs 201 are mainly explained.
Programs to be stored in the memory unit 204 may be loaded into the memory unit 204 from a readout device including storage media or via a network (neither the readout device nor the network is shown). The data of the database 115 may be stored in the RAM 202 or the storage unit 204. The I/F 203 may be a part of the remote sensing image data acquisition unit 102 and a part of the three-dimensional information image data acquisition unit 103.

In addition, the I/F 203 may be an I/F that is connected with a network or may be an I/F that is connected with a unique external bus or the like.
The remote sensing observation device 211 is a 5 remote sensing sensor such as an observation satellite, a manned aerial vehicle, or an unmanned aerial vehicle, and any type of sensor may be used as the remote sensing observation device 211 as long as it is a sensor that senses a remote object. The remote sensing image data

10 101a provided by the remote sensing observation device 211 is, for example, data of satellite photographs taken by RapidEye, Terra, or Aqua (on which MODIS (moderate resolution imaging spectroradiometer) is mounted) and LandSat, or data of air photographs.
Here, the remote sensing image data 101a is time-series image data, and the remote sensing image data 101a may be time-series image data taken by one remote sensing observation device 211 or may be time-series image data taken by plural remote sensing observation devices 211.
The type of remote sensing image data 101a is not limited to a special type, but it is preferable that the remote sensing image data 101a is data including information from which colors may be distinguished. If plural types of data are included in the remote sensing image data 101a and there are differences among the resolutions of the

11 plural types of data, the resolutions of the images included in the remote sensing image data 101a may be converted into the smallest resolution among the resolutions. RGB images taken by an unmanned aerial vehicle (UAV) or a drone are adopted as examples of targets of the following descriptions.
The surface database 212 is a database including the three-dimensional information image data 101b, and the information of the surface database 212 may be information acquired by remote sensing sensors, or may be information acquired by other devices. Although the three-dimensional information image data 101b is, for example, digital surface model (DSM) data or digital elevation model (DEN) data, and the three-dimensional information image data 101b is data representing the elevations of grounds and the elevations of the surfaces of deposits three-dimensionally, the three-dimensional information image data 101b is not limited to these types of data. Pieces of the elevation information of the digital surface model (DSM) data are adopted as examples of targets of the following descriptions.
Fig. 3 is a flowchart showing an example of the processing of the ground surface information analysis system 100. First, the ground surface information analysis system 100 prompts a user to specify a region

12 within which an analysis is executed and a time period during which the analysis is executed using the display unit 205, and stores information specified by the input unit 206 in the RAM 202 or the memory unit 204.
Afterward, the remote sensing image data acquisition unit 102 acquires the remote sensing image data 101a (at Step 301), and the three-dimensional information image data acquisition unit 103 acquires the three-dimensional information image data 101b (at Step 302). Step 301 and Step 302 may be executed in the reverse order or may be executed at the same time.
The remote sensing image data 101a and the three-dimensional information image data 101b that are to be acquired include time-series image data during the same time period and about the same region, and include image data during a time period and about a region specified by the input unit 206. Because the image update unit 114 makes the remote sensing image data acquisition unit 102 and the three-dimensional information image data acquisition unit 103 acquire new image data, it is not necessary that the image data about the region and during the time period specified by the input unit 206 should be acquired at once.
The database generation unit 104 generates the database 115 from the remote sensing image data 101a and

13 the three-dimensional information image data 101b (at Step 303). Image data included in the database 115 may be managed on the basis of times in the time series at which the respective data pieces are photographed or measured and on the basis of plane positions such as GPS (global positioning system) positions at which the respective data pieces are photographed or measured. Subsequently, the database 115 is referred to and updated by the following processing.
If a considerably wide area is a target of photographing or measurement, a time needed for the photographing or measurement is, for example, a time period from 10:10 to 10:30 because the considerable wide area cannot be covered by one photographing or one measurement. Furthermore, a time needed for photographing or measuring another target area may be, for example, from 10:40 to 10:50. Therefore, a time, which will be cited hereinafter, does not represent ten minutes or the like, but includes times of day. In addition, a time period may be replaced with one time point during the time period as a representative time point during the time period.
The material quality classification unit 105 classifies material qualities (types of material) about the region and during a time period specified by the input unit 106 using the remote sensing image data 101a stored

14 in the database 115, attaches numbers to the results of the classification respectively, and stores the numbered results in the database 115 (at Step 304). Because the remote sensing image data 101a are RGB images, the classification of material qualities may be executed by an RGB spectrum analysis or a texture analysis, and furthermore machine learning technologies may be applied to the classification.
For example, the remote sensing image data 101a may be classified on the basis of the color information for each material quality using the RGB images, or alternatively other classification processing methods may be applied to the classification of the remote sensing image data 101a. Although the above descriptions have been made about deposits as the targets of classification so far, the classification of grounds may be made on the basis of the soil properties of grounds, or it may be made on the basis of whether or not the grounds are roads where vehicles run. In addition, it is also conceivable that the material quality classification unit 105 generates material quality image data that is made by attaching material quality classification information to three-dimensional images with elevations using the three-dimensional information image data 101b, and stores the material quality image data in the database 115. The information attached to the three-dimensional images may be information including colors, concentrations, and the like that can be visually recognized by a user.
Here, in the classification of material qualities, 5 the boundaries among the material qualities classified into different groups may be decided by binary extraction, energy minimization extraction, edge extraction, or the like in image processing, and furthermore machine learning technologies may be applied to the classification.
10 Alternatively, the extraction may be transferred stepwise from rough extraction to fine extraction, and for example, the binary extraction may be used as rough extraction and the energy minimization extraction may be used as fine extraction. Because the energy minimization extraction

15 needs a large amount of calculation, it is conceivable that a graph cut technology is used for the energy minimization extraction.
The ground coefficient calculation unit 106 calculates ground coefficients about the region and during the time period specified by the input unit 206 using the three-dimensional information image data 101b stored in the database 115, and stores the calculated ground coefficients in the database 115 (at Step 305). The ground coefficients may be the gradient coefficients of ground surfaces, the rugged coefficients of ground

16 surfaces, or other coefficients. Ground coefficients will be explained later with reference to Fig. 4.
The ground estimation unit 107 estimates the elevations of grounds about the region and during the time period specified by the input unit 206 using the elevations of surfaces included in the three-dimensional information image data 101b stored in the database 115, generates ground image data, and stores the ground image data in the database 115 (at Step 306). The elevations of grounds will be explained later with reference to Fig. 5.
Here, Step 304 to Step 306 form time-series multi-spectrum three-dimensional image generation processing 307, and the data stored in the database 115 by Step 304 to Step 306 form time-series multi-spectrum three-dimensional information image data. Therefore, the time-series multi-spectrum three-dimensional image generation processing 307 may be considered to be processing executed by the time-series multi-spectrum three-dimensional image generation unit 108. The time-series multi-spectrum three-dimensional information image data will be explained later with reference to Fig. 6.
The deposit model generation unit 109 generates deposit models for the material qualities respectively using data in the material quality classification unit 105 especially among the time-series multi-spectrum three-

17 dimensional information image data stored in the database 115, and stores the generated deposit models in the database 115 (at Step 308). The deposit models will be explained later with reference to Fig. 7 in the case where coal, wood dust, and wood are cited as material qualities.
The deposit volume calculation unit 110 calculates the volumes of the material qualities respectively in time-series order using the three-dimensional information image data 101b stored in the database 115, the ground image data generated by the ground estimation unit 107, and deposit models generated by the deposit model generation unit 109 to generate volume image data, and stores the calculated volume image data in the database 115 (at Step 309). How to calculate a volume will be explained later with reference to Fig. 8.
The variation amount image generation unit 111 generates volume variation amount image data and ground coefficient variation amount image data for the respective material qualities in time-series using the volume image data and the ground coefficient image data stored in the database 115, and stores these data pieces in the database 115 (at Step 310). The volume variation amount image data and ground coefficient variation amount image data will be explained later with reference to Fig. 9.
The maintenance judgment unit 112 calculates an

18 index showing a road surface condition or an index showing the variation of the road surface condition using the ground coefficient image data or the ground coefficient variation amount image data stored in the database 115, compares these indexes with predefined thresholds, judges whether the maintenance of the road is necessary or not, and provides the judgment result for the control unit 113 (at Step 311). The judgment processing whether the maintenance is necessary or not will be explained later with reference to Fig. 10.
The control unit 113 controls devices regarding deposits using the time-series multi-spectrum three-dimensional information image data stored in the database 115, the volume variation amount image data, the ground coefficient variation amount image data, and the judgment result provided by the maintenance judgment unit 112 (at Step 312). This control may be, for example, the running control of a transport vehicle for deposits, the direction control of road surface maintenance, or other controls.
The running control of a transport vehicle may be executed in such a way that a road whose road surface condition is bad is replaced with a road whose road surface condition is good. Alternatively, it is conceivable that the respective image data are displayed by the display unit 205.

19 In addition, on receiving the judgment result from the maintenance judgment unit 112, the control unit 113 may inform the image update unit 114 that the processing of the image data acquired by the remote sensing image data acquisition unit 102 and the three-dimensional information image data acquisition unit 103 has been finished. For example, because the running control of a vehicle may be executed independently of processing executed by Step 301 to Step 311, there is no adverse effect if the flow shown by this flowchart proceeds to Step 313 and afterward.
The image update unit 114 judges whether the processing shown by this flowchart should be finished or not when the image update unit 114 is informed that the processing of the image data has been finished (at Step 313), and if it is judged that the processing should not be finished, the image update unit 114 executes an image update in which new image data is acquired and processed (at Step 314). Whether the processing shown by this flowchart should be finished or not may be judged on the basis of whether the processing of the image data about the specified region and during the specified time period has been finished, or it is conceivable that it is always judged that the processing shown by this flowchart should not be finished as long as the system continues the processing. Alternatively, whether the processing shown by this flowchart should be finished or not may be judged on the basis a certain kind of input information given by the input unit 206.
5 After executing Step 314, the image update unit 114 directs the remote sensing image data acquisition unit 102 and the three-dimensional information image acquisition unit 103 to acquire new image data, and the flow goes back to Step 301.
10 Fig. 4 is a diagram showing an example of a ground coefficient calculation. Here, the ground coefficient means a slope coefficient, a rugged coefficient, a gradient coefficient, or a gradient angle coefficient.
The ground coefficient calculation may be considered to be 15 the calculation of a TRI (terrain ruggedness index). In order to calculate the TRI, the ground coefficient calculation unit 106 acquires information regarding the elevations of spots included in the three-dimensional information image data 101b stored in the database 115,

20 and calculates the TRI using the elevations of nine spots adjacent to each other. To put it concretely, the ground coefficient calculation unit 106 subtracts the elevation Z(0, 0) of the central point of the grid cell 401 from the elevations Z(i, j) of eight points other than the central point respectively, squares the differences respectively,

21 add eight results obtained by squaring the differences, and calculates the square root of the obtained sum.
Instead of calculating the square root, it is conceivable that the average value of the sum is calculated by dividing the sum by the value 8. The ground coefficient calculation unit 106 calculates the TRIs of spots respectively in such a way, compares the TRIs with a predefined threshold, and classifies a region into regions formed by spots whose TRIs are equal to or larger than the threshold and regions formed by spots whose TRIs are smaller than the threshold. For example, the ground coefficients may be represented by image data in such a way that hatched regions like a region 402 are regions formed by spots whose TRIs are equal to or larger than the threshold, that is, regions having large gradients, and non-hatched regions like a region 403 are regions formed by spots whose TRIs are smaller than the threshold, that is, nearly planar regions. It is conceivable that the ground coefficient calculation unit 106 stores such image data in the database 115.
Fig. 5 is a diagram showing an example of ground estimation. Because elevation information included in the three-dimensional information image data 101b, which are digital surface model (DSM) data, includes the elevations of deposits, too, the elevations of grounds are not

22 necessarily reflected in the elevation information.
Therefore, the ground estimation unit 107 estimates the elevations of grounds using the three-dimensional information image data 101b that is time-series data. An image 501 is an image showing elevations viewed almost vertically from the upper side at a first spot at a first time. A region surrounded by a solid line like a region 502 is a region whose elevation is higher than the elevations of adjacent regions, and a region surrounded by a dashed line like a region 503 is a region whose elevation is almost equal to the elevations of adjacent regions.
An image 504 is an image showing elevations viewed from the same view point as is the case with the image 501 at a first spot at a second time. A region surrounded by a dashed line like a region 505 is a region whose elevation is almost equal to the elevations of adjacent regions, and a region surrounded by a solid line like a region 506 is a region whose elevation is higher than the elevations of adjacent regions. In such image data of elevations, after the elevation of the region 502 and the elevation of the region 505 are compared with each other, a region 508 whose elevation is the same as the elevation of the region 505 is selected, and after the elevation of the region 503 and the elevation of the region 504 are

23 compared with each other, a region 509 whose elevation is the same as the elevation of the region 503 is selected, with the result that the elevation of the ground is estimated and the data of a ground image 507 is acquired.
Here, although, in order to make the above explanation easily understood, the image 501, the image 504, and the ground image 507 are assumed to be planes, data of these images may be set to three-dimensional image data including elevations.
Fig. 6 is a diagram showing an example of time-series multi-spectrum three-dimensional information data.
"Time 604" is a generic term for plural times forming a time series, and the plural times may be times at which the remote sensing image data and the three-dimensional information image data 101b are photographed or measured, or may be predefined times. "Material Quality 601" is a generic term including "Material Quality (1)", "Material Quality (2)", or "Material Quality (3)" into which deposits are classified, and numbers in the parentheses are numbers given to the results of the classification.
For example, "Material Quality (1)" indicates coal, "Material Quality (2)" indicates wood dust, and "Material Quality (3)" indicates wood.
"Three-dimensional image 605" is a generic term including a three-dimensional image (1, 1) to a three-

24 dimensional image (3, n), and a three-dimensional image (3, 1) of "Three-Dimensional Image 605" in "Material Quality (3)" of "Material Quality 601" or the like may be image data for representing an image showing the color of a wood deposit region different from the colors of other wood deposit regions among three-dimensional images including elevations. "Elevation 602" may include image data for representing three-dimensional images of elevations among the three-dimensional information image data 101b.
"Ground Condition Coefficient 603" includes ground coefficient image data generated by the ground coefficient calculation unit 106, and may include data representing images of the region 402, the region 403, and the like shown in Fig. 4. Furthermore, "Ground Condition Coefficient 603" may include image data of plural coefficients, and it is conceivable that the plural coefficients are a rugged coefficient, a gradient coefficient, a soil property coefficient, and the like.
Fig. 7 is a diagram showing examples of deposit models. A deposit model is a model used for specifying the area of a ground that is covered by a deposit, and the model is generated depending on the material quality ofk the deposit intermittently per certain time period. In other words, a three-dimensional image belonging to "Three-Dimensional Image 605" is generated for each of "Material Quality (1)" to "Material Quality (3)" belonging to "Material Quality 601" and for each of "Time (1) to "Time (3)" belonging to "Time 604". In three-dimensional images belonging to "Three-Dimensional Image 605" in 5 "Material Quality (1)" that is coal, regions 701 that are classified into coal form an orderly shape as shown in Fig.
7. Therefore, a region 702 derived from the outline of the cluster of the plural regions 701 becomes the deposit model of the coal.
10 In three-dimensional images belonging to "Three-Dimensional Image 605" in "Material Quality (2)" that is dust wood, regions 703 that are classified into wood dust are dispersedly disposed, and the regions 703 form a cluster of small particles with various shapes. Therefore, 15 one large-scale deposit obtained by merging adjacent regions 703 together becomes the deposit model of the wood dust. In three-dimensional images belonging to "Three-Dimensional Image 605" in "Material Quality (3)" that is wood, regions 705 that are classified into wood are large 20 rectangles respectively. Therefore, a region 706 obtained by merging these large rectangles 705 becomes the deposit model of the wood.
In addition, as for other material qualities, deposit models depending on the respective material

25 qualities may be built. Here, dotted lines shown in Fig.

26 7 show regions that coincide with the regions of the above deposit models respectively, and these dotted lines are depicted in order to make the regions of the above deposit models easily understood. It is conceivable that deposit models generated for material qualities at certain time intervals respectively are stored in the database 115.
Fig. 8 is a flowchart showing an example of a deposit volume calculation. The deposit volume calculation unit 110 acquires deposit models for each material quality at certain time intervals generated by the deposit model generation unit 109 (at Step 801). As shown in Fig. 7, because the deposit models occupy plane areas such as the regions 702, 704, and 706, the areas of the respective regions are calculated (at Step 802). Here, for example, as for the area of the region 702, it is conceivable that the region 702 is divided into plural regions, the areas of the plural regions are calculated, and the area of the region 702 is an aggregate of the calculated areas of the plural regions. Because the region 702 is an image, the region 702 may be divided into regions in units of the pixels of the region 702.
Next, the deposit volume calculation unit 110 acquires the elevations of the respective deposit models (at Step 803). Here, if a region is divided into plural regions at Step 802, the elevations to be acquired are the

27 elevations of the divided regions. If any one of the divided regions is wide, the elevation of the region may be set to an elevation that represents the region. If the region is divided into regions in units of the pixels of the region 702, the elevations of the pixels are adopted.
Data about elevations may be acquired from the three-dimensional image data 101b.
The deposit volume calculation unit 110 acquires the elevations of grounds under the respective deposit models, that is, the elevations of grounds covered by the respective deposits (at Step 804). If a region is divided into plural regions at Step 802, the deposit volume calculation unit 110 acquires the elevations of the plural regions instead of the elevation of the ground of the original region. The elevation of the ground itself is the elevation of the ground estimated by the ground estimation unit 107.
The deposit volume calculation unit 110 subtracts the elevations of the grounds acquired at Step 804 at Step 804 from the elevations of the deposit models acquired at Step 803 respectively and multiplies the results of this subtraction with the areas calculated at Step 802 respectively to calculate the volumes of the deposits. If a region is divided into plural regions at Step 802, volumes for the respective regions obtained by the

28 division are calculated, and the calculated volumes are summed up. In such a way, the volumes of the plural deposit models are calculated respectively.
Fig. 9 is a diagram showing an example of variation amount image. "Material Quality 901", "Ground Condition Coefficient 903", and "Time 904" shown in Fig. 9 correspond to "Material Quality 601", "Ground Condition Coefficient 603", and "Time 604" shown in Fig. 6 respectively. "Three-Dimensional Volume Amount Image 905"
is a generic term including a three-dimensional image that stereoscopically depict volume calculated by the deposit volume calculation unit 110, and the three-dimensional image is calculated and made three-dimensional for every combination of "Material Quality (1)" to "Material Quality (3)" belonging to "Material Quality 901" and "Time (1)" to "Time (n)" belonging to "Time 904".
A material quality (1) volume quality variation amount image 906 is an image that represents the variation of three-dimensional volume images in the case where a deposit belongs to "Material Quality (1)" (coal) belonging to "Material Quality 901" and time changes from "Time (1)"
to "Time (n) in "Time 904". For example, the material quality (1) volume quality variation amount image 906 may be an image that is obtained by changing the colors and densities of a three-dimensional volume image (1, 1) to a

29 three-dimensional volume image (1, n) and merging these images so that the variation may be visually recognized.
Three-dimensional condition images (b, 1) to (b, n) belonging to "Three-Dimensional Condition Image 907" are obtained in such a way that the ground coefficient image data generated by the ground coefficient calculation unit 106 is converted into stereoscopic three-dimensional images at "Time (1)" to "Time (n)" belonging to "Time 904".
A ground condition variation amount image 908 is an image that represents the variation of the three-dimensional condition image 907 while time changes from "Time (1)" to "Time (n)" belonging to "Time 904". In order to represent this variation, it is conceivable that the colors and densities of the images at "Time (1)" to "Time (n)"
belonging to Time 904 are changed and the images are merged.
Fig. 10 is a flowchart showing an example of maintenance judgment. The maintenance judgment unit 112 acquires the ground condition variation amount images 908 generated by the variation amount generation unit 111 (at Step 1001). Here, it is conceivable that the ground condition variation amount images 908 may be acquired for a rugged coefficient, a gradient coefficient, and a soil property coefficient respectively.
In the case where plural ground condition variation amount images 908 are acquired, the maintenance judgment unit 112 merges the plural ground condition variation amount images 908 into one image (at Step 1002). For example, it is conceivable that a ground condition 5 variation amount image for a gradient coefficient with a rough image resolution and a ground condition variation amount image for a rugged coefficient with a fine image resolution are merged, and image data that is used for generating an image capable of representing even 10 ruggedness on a gradient is generated. Furthermore, if this generated image data is observed on a time-series basis, it represents the variation amount of the angle of a gradient during a predefined time period for example, therefore this generated image data may be considered to 15 provide information indicating that maintenance should be immediately done if the variation amount of the angle of the gradient during the predefined time period is large.
Therefore, the maintenance judgment unit 112 digitizes the variation amount included in the image data 20 used for generating the merged image (at Step 1003). This digitized variation amount may include the variation amount of the angle of a gradient as mentioned above.
Subsequently, the digitized variation amount is compared with a predefined threshold, and if it is judged that the 25 digitized variation amount is equal to or larger than the threshold, the flow proceeds to Step 1005, and if it is judged that the digitized variation amount is smaller than the threshold, the flow proceeds to Step 1006.
The maintenance judgment unit 112 sets information that maintenance is necessary at Step 1005, and sets information that maintenance is not necessary at Step 1006.
Subsequently, these pieces of information are sent to the control unit 113.
In addition, the judgment is not made about only one coefficient. For example, it is conceivable that, after a value obtained by multiplying the variation amount of a digitized rugged coefficient with a first constant and a value obtained by multiplying the variation amount of a digitized soil property coefficient with a second constant are added, and the sum of these values is compared with a predefined threshold, the judgment is made on the basis of the result of this comparison. Furthermore, it is conceivable that the judgment is not made on the basis of any variation amount but the judgment is made on the basis of a ground coefficient included in a three-dimensional condition image 907 at "Time (n)" belonging to "Time 904", that is, a ground coefficient included in a three-dimensional condition image 907 stored in the database 115 for the last time.
In addition, it is conceivable that, after the variation amount image generation unit 111 specifies the variation amount of a volume for each material quality or each deposit model among the variation amounts of volumes calculated by the deposit volume calculation unit 110, and compares the specified variation amount with a predefined threshold, the judgment whether or not maintenance owing to a large accumulation of deposits should be made is made on the basis of the comparison result. Therefore, it is conceivable that three-dimensional image data representing variation amounts is converted into numerical values representing the variation amounts.
As described above, a variation amount can be calculated for each material. Especially, a deposit and a ground can be targets for which variation amounts are calculated, and the variation amount of a volume for each deposit and the variation amount of a ground coefficient can be calculated. Because two-dimensional RGB image data and digital surface model (DSM) data that are photographed by an unmanned aerial vehicle are used for this calculation, a high amount of cost is not needed.
Furthermore, because the variation amount of a volume for each deposit and the variation amount of a ground coefficient can be calculated from these same data pieces, these data pieces are useful for both management and transport of deposits at a mining company or the like.
=

In the calculation of the volumes of deposits, the volumes can be calculated from time-series image data without a special research for the calculation of the volumes. In this case, the calculation of the shapes of grounds under deposits can be included in this calculation, and the volume calculation suitable for each deposit can be executed using deposit models corresponding to the material qualities of the deposits. In addition, the photographing from the above makes it possible to cover a wide area as a target, and also makes it possible that the fluctuation of the accuracy of the calculation does not occur depending on a worker.
Because a variation amount is converted into an image as the result of calculation or estimation, the display of the image makes the condition of the variation easily understood by a user. Furthermore, because the necessity of maintenance is judged on the basis of the result of merging plural ground coefficients, it also becomes possible to make the judgment from the plural viewpoints. In addition, when the maintenance is needed, the mining company can continue to execute its original mining operation and the like by controlling the maintenance work.
The present invention is not limited by the above-described embodiments, and includes various variations.

For example, the above-described embodiments have been described in detail in order to make the present invention easy to understand, and therefore all the components described so far are not always indispensable for the present invention. For example, other components may be added to the components included in the embodiments, a part of the components included in the embodiments may be deleted or replaced with other components. In addition, parts or the entirety of the above-described components, functions, processing units, or processing means may be realized by hardware with the use of integrated circuits for example.
Furthermore, the above-described components, functions, and the like may be realized in such a way that a processor interprets and executes programs that realize the work of the components and the like respectively.
Information regarding the programs, tables, and files that are used for realizing the respective functions may be stored in a recording device such as a memory, an HDD, or an SSD, or in a recording medium such as an IC (integrated circuit) card, an SD card, or a DVD (digital versatile disc).
In addition, in the above-described drawings, control lines and information lines are shown in the case where they are indispensable for explaining the above embodiments, therefore all control lines and information lines necessary in the case of manufacturing the above embodiments are not shown. It is conceivable that in reality almost all components in almost every embodiment 5 are interconnected.

Claims (15)

What is claimed is:

1. A ground surface information analysis system for analyzing a ground surface using images, the ground surface information analysis system comprising:
a processor;
a computer-readable medium having stored thereon statements and instructions which when executed by the processor cause the processor to:
acquire two-dimensional image data including information about regions of materials on the ground surface at a plurality of times;
acquire three-dimensional image data including configuration information of the ground surface at the plurality of times;
specify respective regions of a plurality of types of materials using the acquired two-dimensional image data;
generate time-series multi-spectrum three-dimensional image data from the respective specified regions of the materials and the configurations specified by the acquired three-dimensional image data as combinations of the plurality of times and the plurality of types of materials, where the regions and the configurations correspond to the respective times included in the plurality of times; and calculate variation amounts of the respective materials on the basis of a plurality of time data pieces included in the generated time-series multi-spectrum three-dimensional image data.

2. The ground surface information analysis system according to claim 1, wherein acquire the two-dimensional image data includes:
acquiring remote-sensing image data which an unmanned aerial vehicle obtains by photographing the ground surface as the two-dimensional image data.

3. The ground surface information analysis system according to claim 1, wherein acquire the two-dimensional image data includes:
acquiring two-dimensional image data including color information as information about the regions of the materials, and wherein specify the respective regions of the plurality of types of materials includes specifying the respective regions of the plurality of types of materials based on the color information in the acquired two-dimensional image data.

4. The ground surface information analysis system according to claim 3, wherein the materials on the ground surface are deposits on the ground surface, wherein acquire the two-dimensional data includes acquiring two-dimensional image data including information about the regions of the deposits at a plurality of times, and wherein specify the respective regions of the plurality of types of materials includes:
specifying the respective regions and types of a plurality of types of deposits based on the color information included in the acquired two-dimensional image data.

5. The ground surface information analysis system according to claim 1, wherein the configuration information of the ground surface includes elevation information of the ground;
wherein acquire the three-dimensional image data includes acquiring three-dimensional image data including the elevation information of the ground at a plurality of times;
wherein the statements and instructions when executed by the processor further cause the processor to:

calculate a ground coefficient representing a rugged condition of the ground based on the elevation information included in the acquired three-dimensional image data.

6. The ground surface information analysis system according to claim 1, wherein the configuration information of the ground surface includes elevation information of deposits on the ground;
wherein acquire the three-dimensional image data includes acquiring three-dimensional image data including the elevation information of the deposits at a plurality of times;
wherein the statements and instructions when executed by the processor further cause the processor to:
calculate on elevation of the ground based on a plurality of pieces of the elevation information at the plurality of times included in the acquired three-dimensional image data.

7. The ground surface information analysis system according to claim 4, wherein the statements and instructions when executed by the processor cause the processor to:

generate one region by collectively merging some regions that are adjacent to each other among the specified regions of each of the specified types of deposits;
generate a deposit model corresponding to the one region.

8. The ground surface information analysis system according to claim 7, wherein the configuration information of the ground surface includes elevation information of deposits on the ground;
wherein acquire the three-dimensional image data includes:
acquiring three-dimensional image data including the elevation information of the deposits at a plurality of times;
wherein the statements and instructions when executed by the processor further cause the processor to:
calculate an elevation of the ground based on a plurality of pieces of the elevation information at the plurality of times included in the acquired three-dimensional image data; and calculate volumes of the deposits based on areas of the generated deposit models, the acquired elevation information of the deposits, and the calculated elevation of the ground.

9. The ground surface information analysis system according to claim 8, wherein the statements and instructions when executed by the processor further cause the processor to:
calculate volumes of deposits corresponding to the combinations of a plurality of times and a plurality of types of deposits using the generated time-series multi-spectrum three-dimensional image data; and generate volume variation amount image data of the respective deposits based on the variation amounts of the calculated volumes at the plurality of times.

10. The ground surface information analysis system according to claim 9, wherein the generated volume variation amount image data of the respective deposits is displayed as images.

11. The ground surface information analysis system according to claim 5, wherein the statements and instructions when executed by the processor cause the processor to:
calculate ground coefficients at the plurality of times using the generated time-series multi-spectrum three-dimensional image data; and generate ground condition variation amount image data based on the variation amount of the calculated ground coefficients at the plurality of times.

12. The ground surface information analysis system according to claim 11, wherein the statements and instructions when executed by the processor cause the processor to:
digitize the variation amount of the ground coefficients using the generated ground condition variation amount image data;
compare the digitized variation amount with a predefined threshold respectively;
if the variation amount is judged to be equal to the relevant threshold or larger, determine that maintenance is necessary; and if the variation amount is judged to be smaller than the relevant threshold, determine that maintenance is not necessary.

13. The ground surface information analysis system according to claim 12, wherein the statements and instructions when executed by a processor cause the processor to:
wherein if it is decided that maintenance is necessary, control the ground surface information analysis system so that the ground surface is maintained.

14. A ground surface information analysis method for analyzing a ground surface using images, the ground surface information analysis system comprising:
acquiring two-dimensional image data including information about regions of materials on the ground surface at a plurality of times;
acquiring three-dimensional image data including configuration information of the ground surface at the plurality of times;
specifying respective regions of a plurality of types of materials using the acquired two-dimensional image data;
generating time-series multi-spectrum three-dimensional image data from the respective specified regions of the materials and the configurations specified by the acquired three-dimensional image data as combinations of the plurality of times and the plurality of types of materials, where the regions and the configurations correspond to the respective times included in the plurality of times; and calculating variation amounts of the respective materials on the basis of a plurality of time data pieces included in the generated time-series multi-spectrum three-dimensional image data.

15. The ground surface information analysis method according to claim 14, wherein acquiring the two-dimensional image data includes:
acquiring two-dimensional image data including color information as information about the regions of the materials, and wherein specifying the respective regions of the plurality of types of materials includes:
specifying the respective regions of the plurality of types of materials based on the color information included in the acquired two-dimensional image data.