JP7841863B2 - Face stability evaluation device, control method, and program - Google Patents

Description

This disclosure relates to a technology for evaluating the stability of drilling sites.

In excavation work such as tunnel construction, it is necessary to ensure the safety of the workers and implement appropriate measures as needed by monitoring the stability of the tunnel face. However, manually evaluating the stability of the tunnel face is highly subjective and prone to variability, making it largely dependent on the judgment of experienced and skilled technicians. However, it is often difficult to have such skilled technicians present at the excavation site for extended periods. Therefore, technologies are being developed that utilize information processing to evaluate the stability of the tunnel face.

Patent Document 1 discloses a technique for determining the stability of a tunnel face using data related to blast holes in the tunnel face. Patent Document 2 discloses a technique for determining the stability of a tunnel face by analyzing images of the tunnel face to determine the state of weathering, the frequency of tunnel face collapse, the state of cracks in the tunnel face, the type of rock constituting the tunnel face, and the state of groundwater seepage in the tunnel face.

Japanese Patent Publication No. 2018-071165 Japanese Patent Publication No. 2019-190062

Vijay Badrinarayanan, Alex Kendall, R. Cipolla, "SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation," [online], October 10, 2016, arXiv, Internet <URL:https://arxiv.org/pdf/1511.00561.pdf>

Patent Document 1 assumes that blasting excavation is performed. Therefore, the technology described in Patent Document 1 cannot be used in cases where the tunnel face is excavated using hydraulic breakers or the like. Patent Document 2 evaluates the stability of the tunnel face using only images of the tunnel face. Therefore, information other than visual information is not considered in the evaluation of the stability of the tunnel face. The present invention has been made in view of these problems, and one of its objectives is to provide a new technology for evaluating the stability of a tunnel face.

The face stability evaluation apparatus 2000 of this disclosure includes an acquisition unit that acquires audio data obtained by recording sounds around the face to be evaluated, and video data obtained by imaging the face to be evaluated, and a sound type identification step that identifies the type of sound represented by the audio data,
The system includes a collapse area calculation unit that uses the video data to calculate the collapse area, which is the area of the collapsed region at the tunnel face to be evaluated, and a stability determination unit that determines the stability of the tunnel face to be evaluated based on the identified sound type and the calculated collapse area.

The control method disclosed herein is performed by a computer. The control method includes: an acquisition step of acquiring audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated; a sound type identification step of identifying the type of sound represented by the audio data; a collapse area calculation step of calculating the collapse area, which is the area of the collapse region at the tunnel face to be evaluated, using the video data; and a stability identification step of identifying the stability of the tunnel face to be evaluated based on the identified sound type and the calculated collapse area.

The program of this disclosure causes a computer to execute the control method of this disclosure.

This disclosure provides a new technique for evaluating the stability of a tunnel face.

This figure illustrates an overview of the operation of the face stability evaluation device of Embodiment 1. This is a block diagram illustrating the functional configuration of the face stability evaluation device of Embodiment 1. This is a block diagram illustrating the hardware configuration of a computer that implements a tunnel face stability evaluation device. This is a flowchart illustrating the processing flow performed by the face stability evaluation device of Embodiment 1. This diagram illustrates a method for classifying different types of sounds. This is a diagram illustrating a sedimentary region. This flowchart illustrates the process for determining the stability of the tunnel face according to the judgment rules. This flowchart illustrates the process for determining the stability of the tunnel face according to the judgment rules. This is a block diagram illustrating the functional configuration of a face stability evaluation device having an output unit. This is a diagram illustrating stability information.

The embodiments of this disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations are omitted where necessary for clarity. Unless otherwise specified, predetermined values such as specified values or thresholds are stored in a memory unit accessible from the device using those values. Furthermore, unless otherwise specified, the memory unit consists of one or any number of storage devices.

Figure 1 illustrates an overview of the operation of the face stability evaluation device 2000 according to Embodiment 1. Here, Figure 1 is intended to facilitate understanding of the overview of the face stability evaluation device 2000, and the operation of the face stability evaluation device 2000 is not limited to that shown in Figure 1.

The tunnel face stability evaluation device 2000 evaluates the stability of the tunnel face 10, which is the tunnel face to be evaluated. Here, "tunnel face" refers to the excavation surface at an excavation site using excavation machinery (such as a tunnel construction site).

The stability of the tunnel face 10 is evaluated using audio data 22 and video data 32. Audio data 22 is generated by recording sounds around the tunnel face 10 using a microphone 20. Video data 32 is obtained by photographing the tunnel face 10 using a camera 30. Although the microphone 20 and camera 30 are shown separately in Figure 1, they may be housed in the same enclosure.

The face stability evaluation device 2000 identifies the type of sound represented by the audio data 22 from among several specific types. For example, three types of sound can be used: 1) no excavation sound, 2) light excavation sound, and 3) heavy excavation sound. Sounds classified as "no excavation sound" contain little to no excavation sound. Such sounds represent situations where excavation by the excavating machinery is not taking place.

Sounds classified as "light excavation noise" are primarily composed of the sound of the excavating machinery and have a large proportion of high-frequency components. Such sounds are generated when excavating relatively soft ground. Sounds classified as "heavy excavation noise" are generated by the combined action of the excavating machinery and the bedrock, and compared to light excavation noise, they have a larger proportion of low-frequency components. Such sounds are generated when excavating relatively hard ground. Thus, the difference between light and heavy excavation noise reflects the difference in the hardness of the excavation face 10.

Furthermore, the tunnel face stability evaluation device 2000 uses video data 32 to calculate the area of the collapsed region, which is the area that has collapsed from the tunnel face 10. Then, the tunnel face stability evaluation device 2000 determines the stability of the tunnel face 10 based on the type of sound identified using audio data 22 and the collapsed area calculated using video data 32.

<Example of effects and benefits>
According to the face stability evaluation device 2000 of this embodiment, the stability of the tunnel face 10 is determined by utilizing audio data 22 obtained by recording sounds around the tunnel face 10 and video data 32 obtained by imaging the tunnel face 10. Thus, the face stability evaluation device 2000 provides a new technology for evaluating the stability of the tunnel face. Furthermore, the face stability evaluation device 2000 determines the stability of the tunnel face 10 using not only images but also audio. Therefore, it is possible to determine the stability of the tunnel face 10 more accurately compared to cases where only images are used to determine the stability of the tunnel face 10. Moreover, the face stability evaluation device 2000 can determine the stability of the tunnel face 10 during excavation work using an excavation machine.

The face stability evaluation device 2000 of this embodiment will be described in more detail below.

<Example of functional configuration>
Figure 2 is a block diagram illustrating the functional configuration of the face stability evaluation device 2000 of Embodiment 1. The face stability evaluation device 2000 includes an acquisition unit 2020, a sound type identification unit 2040, a collapse area calculation unit 2060, and a stability determination unit 2080. The acquisition unit 2020 acquires audio data 22 and video data 32. The sound type identification unit 2040 identifies the type of sound represented by the audio data 22. The collapse area calculation unit 2060 calculates the collapse area using the video data 32. The stability determination unit 2080 determines the stability of the face 10 based on the type of sound and the collapse area.

<Example of hardware configuration>
Each functional component of the tunnel face stability evaluation device 2000 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program to control it). The following will further describe the case in which each functional component of the tunnel face stability evaluation device 2000 is implemented by a combination of hardware and software.

Figure 3 is a block diagram illustrating the hardware configuration of the computer 500 that implements the tunnel face stability evaluation device 2000. The computer 500 is any computer. For example, the computer 500 could be a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 500 could be a portable computer such as a smartphone or tablet. The computer 500 may be a dedicated computer designed specifically for implementing the tunnel face stability evaluation device 2000, or it may be a general-purpose computer.

For example, by installing a predetermined application on computer 500, the various functions of the face stability evaluation device 2000 are realized on computer 500. The application consists of programs for realizing each functional component of the face stability evaluation device 2000. The method of obtaining the program is arbitrary. For example, the program can be obtained from a storage medium (such as a DVD disc or USB memory) on which it is stored. Alternatively, the program can be obtained by downloading it from a server device managing the storage device on which it is stored.

The computer 500 includes a bus 502, a processor 504, memory 506, a storage device 508, an input/output interface 510, and a network interface 512. The bus 502 is a data transmission path for the processor 504, memory 506, storage device 508, input/output interface 510, and network interface 512 to send and receive data to and from each other. However, the method of connecting the processor 504 and other components is not limited to bus connection.

The processor 504 is a various type of processor, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array). The memory 506 is the main memory, implemented using RAM (Random Access Memory), etc. The storage device 508 is the auxiliary storage device, implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

The input/output interface 510 is an interface for connecting the computer 500 with input/output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input/output interface 510.

The network interface 512 is an interface for connecting the computer 500 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

The storage device 508 stores programs that implement each functional component of the tunnel face stability evaluation device 2000 (programs that implement the aforementioned applications). The processor 504 reads these programs into memory 506 and executes them to implement each functional component of the tunnel face stability evaluation device 2000.

The face stability evaluation device 2000 may be implemented using one computer 500 or multiple computers 500. In the latter case, the configuration of each computer 500 does not need to be identical; they can be different.

<<Regarding Microphone 20 and Camera 30>>
The microphone 20 is any microphone capable of recording sounds around the tunnel face 10 and generating audio data. The audio data 22 generated by the microphone 20 is stored in any memory unit. For example, the microphone 20 is connected to the computer 500 (tunnel face stability evaluation device 2000) via an input/output interface 510 or a network interface 512. However, it is sufficient that the audio data 22 generated by the microphone 20 can be acquired by the tunnel face stability evaluation device 2000, and the microphone 20 does not need to be connected to the tunnel face stability evaluation device 2000 for communication.

Camera 30 is any camera capable of imaging the tunnel face 10 and generating video data. The video data 32 generated by camera 30 is stored in any storage unit. For example, camera 30 is connected to computer 500 (tunnel face stability evaluation device 2000) via input/output interface 510 or network interface 512. However, if the video data 32 generated by camera 30 can be acquired by the tunnel face stability evaluation device 2000, camera 30 does not need to be connected to the tunnel face stability evaluation device 2000 for communication.

<Processing Flow>
Figure 4 is a flowchart illustrating the processing flow performed by the tunnel face stability evaluation device 2000 of Embodiment 1. The acquisition unit 2020 acquires audio data 22 and video data 32 (S102). The sound type identification unit 2040 identifies the type of sound represented by the audio data 22 (S104). The collapse area calculation unit 2060 calculates the collapse area using the video data 32 (S106). The stability identification unit 2080 identifies the stability of the tunnel face 10 based on the identified sound type and the calculated collapse area (S108).

The processing flow performed by the face stability evaluation device 2000 is not limited to the flow shown in Figure 4. For example, the identification of the sound type (S104) and the acquisition of video data 32 (S106) may be performed in the reverse order of the order shown in Figure 4, or they may be performed in parallel.

The face stability evaluation device 2000 is preferably capable of repeatedly determining the stability of the tunnel face 10 by repeatedly executing the process shown in Figure 4. For example, the face stability evaluation device 2000 determines the stability of the tunnel face 10 at specific time intervals by performing the process shown in Figure 4 at specific time intervals.

<Acquisition of audio data 22 and video data 32: S102>
The acquisition unit 2020 acquires audio data 22 and video data 32 (S102). The method by which the acquisition unit 2020 acquires the audio data 22 is arbitrary. For example, the acquisition unit 2020 acquires the audio data 22 by accessing the storage unit where the audio data 22 is stored and reading the audio data 22. Alternatively, for example, the acquisition unit 2020 acquires the audio data 22 by receiving the audio data 22 transmitted from another device (e.g., a microphone 20).

The same applies to how the acquisition unit 2020 acquires the video data 32. That is, for example, the acquisition unit 2020 acquires the video data 32 by reading it from a storage unit where the video data 32 is stored, or by receiving the video data 32 transmitted from another device (e.g., a camera 30).

Furthermore, the acquisition unit 2020 may acquire data containing both audio data 22 and video data 32, and extract the audio data 22 and video data 32 from that data. For example, suppose the camera 30 is a video camera with a built-in microphone 20. In this case, the camera 30 can be used to generate a video file in which both audio and video are recorded. The acquisition unit 2020 acquires this video file and extracts the data representing the video from the data contained in the video file as video data 32. The acquisition unit 2020 also extracts the data representing the audio from the data contained in the video file as audio data 22.

<Identification of sound type: S104>
The sound type identification unit 2040 identifies the type of sound represented by the audio data 22 (S104). For example, as mentioned above, three types of sounds are used: 1) no excavation sound, 2) light excavation sound, and 3) heavy excavation sound.

There are various methods for classifying the type of sound represented by the audio data 22. For example, the sound type identification unit 2040 classifies the type of sound by processing according to the flow shown in Figure 5. Figure 5 is a diagram illustrating a method for classifying the type of sound. The sound type identification unit 2040 obtains multiple partial audio data by dividing the audio data 22 into predetermined time lengths (for example, every second) (S202). The sound type identification unit 2040 binarizes each partial audio data by comparing it with a threshold value (S204). Hereafter, the value obtained by this binarization will be called the audio flag. An audio flag of 1 means that there is a high probability that the audio represented by the partial audio data includes excavation sounds.

For example, the sound type identification unit 2040 calculates statistical values of sound pressure (such as the mean, maximum, or mode) from the time-series sound pressure data represented by the partial sound data. If the calculated statistical value is above a threshold, the sound type identification unit 2040 converts the partial sound data into a sound flag with a value of 1. Conversely, if the calculated statistical value is below the threshold, the sound type identification unit 2040 converts the partial sound data into a sound flag with a value of 0.

The sound type identification unit 2040 calculates statistical values (such as the mean or mode) of the sound flags obtained from each partial sound data (S206). For example, suppose the length of the sound data 22 is N seconds, the length of the partial sound data is 1 second, and the mean value is used as the type of statistical value. In this case, the sound type identification unit 2040 calculates the mean value of the N sound flags obtained from the N partial sound data.

The sound type identification unit 2040 identifies the type of sound represented by the audio data 22 based on the statistical value of the audio flag (S208). Here, a larger statistical value of the audio flag indicates a higher frequency of excavation noise in the audio data 22. Furthermore, the harder the tunnel face 10, the more intense the excavation, resulting in a higher frequency of excavation noise. Therefore, when the audio data 22 represents heavy excavation noise, the statistical value of the audio flag is larger than when the audio data 22 represents light excavation noise. Also, when the audio data 22 represents light excavation noise, the statistical value of the audio flag is larger than when the audio data 22 does not contain any excavation noise.

Therefore, for example, the numerical range of the statistical value of the audio flag is divided in advance into three ranges: a range representing heavy excavation sounds, a range representing light excavation sounds, and a range representing no excavation sound. To this end, a threshold (hereinafter referred to as the first flag threshold) is defined to separate the range of heavy excavation sounds from the range of light excavation sounds, and a threshold (hereinafter referred to as the second flag threshold) is defined to separate the range of light excavation sounds from the range of no excavation sound. The sound type identification unit 2040 determines which of the three numerical ranges the statistical value of the audio flag calculated for the audio data 22 falls into.

If the statistical value of the audio flag is greater than or equal to the first flag threshold, the sound type identification unit 2040 determines that the type of sound represented by the audio data 22 is a heavy excavation sound. If the statistical value of the audio flag is greater than or equal to the second flag threshold but less than the first flag threshold, the sound type identification unit 2040 determines that the type of sound represented by the audio data 22 is a light excavation sound. If the statistical value of the audio flag is less than the second flag threshold, the sound type identification unit 2040 determines that the type of sound represented by the audio data 22 is no excavation sound.

Here, the sound type identification unit 2040 may perform any preprocessing on the audio data 22 before performing the processing shown in Figure 5. For example, preprocessing could involve removing noise in the high-frequency or low-frequency range. Existing techniques, such as using high-pass or low-pass filters, can be employed to remove noise contained in specific frequency ranges.

The method for classifying sound types is not limited to the method shown in Figure 5. For example, the sound type identification unit 2040 may use a pre-trained machine learning model to classify the sound types represented by the audio data 22. Hereinafter, this model will be referred to as the sound type classification model. The sound type classification model is pre-trained to output a label indicating the sound type represented by the audio data 22 in response to the input audio data 22. Such a model can utilize any machine learning model capable of handling time-series data (e.g., an RNN (recurrent neural network)).

Training a sound classification model can be done using multiple training datasets, each consisting of a pair of training input data and training ground truth data. The training input data is audio data. The training ground truth data is a label indicating the type of sound represented by the corresponding audio data. Various existing techniques can be used to train machine learning models using such training data.

Here, the temporal scale used for identifying the type of sound is arbitrary. For example, the sound type identification unit 2040 identifies one type of sound for the entire sound represented by the audio data 22. Alternatively, the sound type identification unit 2040 may divide the audio data 22 into audio data at predetermined time intervals (e.g., every second) and identify the type of sound represented by each audio data.

<Calculation of collapse area: S106>
The collapse area calculation unit 2060 calculates the collapse area using the video data 32 (S106). The calculation of the collapse area can be achieved by detecting the collapsed area, which is the area that has collapsed, from the image area of the tunnel face 10, and calculating the area of the detected collapsed area.

Here, we will explain how to detect the collapse area from the tunnel face 10. For example, the collapse area calculation unit 2060 detects the motion area, which represents a moving object, from the image area representing the tunnel face 10 by comparing multiple video frames that constitute the video data 32. Various techniques can be used to detect motion, such as background subtraction, the Lucas-Kanade method, or the Gunnar-Farnback method.

Furthermore, the collapse area calculation unit 2060 identifies collapse areas by classifying the moving regions according to the type of object contained within them. For example, a collapse area can be defined as a moving region whose contained object is soil and sediment. Therefore, the collapse area calculation unit 2060 identifies moving regions whose contained object is soil and sediment as collapse areas.

For detecting collapse areas, semantic segmentation can be used, for example. Specifically, the collapse area calculation unit 2060 detects moving areas using video data 32, and then marks the moving areas for each video frame of the video data 32. Marking of moving areas is performed, for example, by superimposing a specific color on the moving areas or replacing the moving areas with a specific color. Then, the collapse area calculation unit 2060 detects collapse areas by performing semantic segmentation on the video frames with marked moving areas.

Semantic segmentation identifies the type of object represented by each pixel in a video frame. This allows for the detection of pixels classified as part of a collapse region, thus enabling the detection of the collapse area.

The specific method for performing semantic segmentation is arbitrary. For example, a classifier trained to perform semantic segmentation can be used. Existing techniques can be used to train a classifier that performs semantic segmentation. For example, the techniques disclosed in Non-Patent Document 1 can be used.

The collapse area calculation unit 2060 calculates the collapse area by calculating the area of the detected collapse region. If multiple collapse regions are detected, for example, the collapse area calculation unit 2060 calculates the total collapse area by summing the areas of each region.

Here, the tunnel face 10 may be divided into multiple sub-regions beforehand, and the collapse area may be calculated for each sub-region. In this case, the collapse area is calculated for each sub-region. Specifically, the collapse area calculation unit 2060 detects each sub-region from each video frame constituting the video data 32, and detects the collapse area for each sub-region. Then, the collapse area calculation unit 2060 calculates the collapse area for each sub-region by summing the areas of the collapse areas detected within each sub-region.

The area of the collapsed region is represented, for example, by the area of that region on the image. That is, the collapsed area calculation unit 2060 calculates the area of the collapsed region as the number of pixels constituting the collapsed region (i.e., the area on the image). Alternatively, for example, the collapsed area calculation unit 2060 may calculate the area of the real-world region corresponding to the collapsed region as the area of the collapsed region. In this case, the ratio of the unit area in the real world to the unit area on the image in the video data 32 is predetermined. Then, the collapsed area calculation unit 2060 calculates the area of the collapsed region on the image and multiplies the calculated value by the above ratio to calculate the area of the collapsed region represented by the area of the real-world region. For example, if one pixel corresponds to N [cm^2] in the real world, the area of the collapsed region can be calculated as the area in the real world (unit: cm^2) by multiplying the number of pixels constituting the collapsed region by N.

In addition, for example, the area of the collapsed region may be expressed as the ratio of the area of the collapsed region to the total area. If a single collapsed area is calculated for the entire tunnel face 10, the total area is the area of the tunnel face 10. On the other hand, if the collapsed area is calculated for each sub-region, the total area is the area of the sub-region.

When the tunnel face is excavated with an excavating machine, the rock mass crushed by the machine accumulates as soil and sediment. This soil and sediment are not what we want to detect as part of a collapse. Therefore, it is preferable not to include the area of soil and sediment crushed by the excavating machine in the collapse area. However, when detecting the area where soil and sediment are moving as the collapse area, the area representing the soil and sediment crushed by the excavating machine (hereinafter referred to as the accumulation area) will also be detected as a collapse area.

Therefore, for example, the collapse area calculation unit 2060 subtracts the area of the deposition area from the collapse area to prevent the deposition area from being treated as part of the collapse area. Figure 6 is an example of a deposition area. In the figure, reference numeral 100 represents the coordinates of the point where the excavation machine contacts the face 10 (hereinafter, excavation position 100). Reference numeral 110 represents the deposition area. In Figure 6, the deposition area 110 is defined as a rectangle with the center of its upper edge being the excavation position 100, the length of its upper edge being a predetermined length L, and its lower edge being the lower edge of the face 10 (i.e., the ground). Note that the shape of the deposition area 110 is not limited to a rectangle and can be any shape.

The identification of the sedimentation area 110 is performed, for example, as follows. First, the collapse area calculation unit 2060 detects the part of the excavation machine that is excavating the face 10 (hereinafter referred to as the excavation section) from each video frame. The excavation section is, for example, the tip of a hydraulic breaker. Here, various existing object recognition technologies can be used to detect the excavation section. For example, a method using a machine learning model trained to detect the excavation section can be employed.

The collapse area calculation unit 2060 detects the coordinates of a specific part of the excavated area (e.g., the center point or centroid) as the excavation position 100. Furthermore, the collapse area calculation unit 2060 determines the deposition area 110 based on the excavation position 100.

The collapse area calculation unit 2060 subtracts the area of the collapse region that is included in the sedimentation region 110 from the total collapse area. Alternatively, instead of subtracting, the collapse area calculation unit 2060 may exclude the collapse region included in the sedimentation region 110 from the calculation when totaling the areas of the collapse regions.

Furthermore, the collapse area calculation unit 2060 may detect the sedimentation area 110 before detecting the collapse area, and exclude the sedimentation area 110 from the area targeted for collapse area detection. In this way, since the collapse area will not be detected from the sedimentation area 110, the area of the sedimentation area 110 where sediment is moving will not be included in the collapse area.

The time scale for calculating the collapse area is arbitrary. For example, the collapse area calculation unit 2060 calculates the collapse area for each video frame included in the video data 32, based on the state represented by that video frame. Alternatively, the collapse area calculation unit 2060 may calculate the collapse area for one video frame at predetermined time intervals (for example, one video frame every second or every minute) from among the video frames included in the video data 32.

In addition, the collapse area calculation unit 2060 may also calculate statistical values (such as average or maximum values) of the collapse area at predetermined time intervals. For example, suppose that for 30 fps (frames per second) video data 32, the collapse area is calculated for each video frame, and statistical values of the collapse area are calculated every second. In this case, the collapse area calculation unit 2060 calculates statistical values of the collapse area for each of the 30 video frames.

<Determination of the stability of the working face 10: S108>
The stability determination unit 2080 determines the stability of the tunnel face 10 based on the type of sound represented by the audio data 22 and the area of the collapsed region (S108). Here, if the audio data 22 does not contain excavation sounds and the collapsed area is large, it means that a large collapse is occurring at the tunnel face 10 even if no vibrations are being applied due to excavation. Therefore, in such a situation, the stability of the tunnel face 10 is considered to be low. On the other hand, if the audio data 22 contains excavation sounds and the collapsed area is small, it means that even if vibrations are being applied to the tunnel face 10 due to excavation, the collapse is small. Therefore, in such a situation, the stability of the tunnel face 10 is considered to be high.

Therefore, for example, the stability determination unit 2080 determines the stability of the tunnel face 10 based on predetermined determination rules. Figures 7 and 8 are flowcharts illustrating the process of determining the stability of the tunnel face 10 according to the determination rules.

In the processes shown in Figures 7 and 8, the stability of the tunnel face 10 is represented by a risk level. A higher risk level for the tunnel face 10 indicates lower stability (higher risk). More specifically, five risk levels are defined: Risk Level 1, Risk Level 2, Risk Level 3A, Risk Level 3B, and Risk Level 4. Risk Level 1 represents the highest stability, and Risk Level 4 represents the lowest stability. Both Risk Level 3A and Risk Level 3B are less stable than Risk Level 2 and more stable than Risk Level 4. However, for Risk Level 3A and Risk Level 3B, it is not specified which is more stable; for example, they may be of similar stability.

The stability determination unit 2080 determines whether or not the audio data 22 contains excavation noise (S302). Here, if the type of sound represented by the audio data 22 is "no excavation noise," it is determined that the audio data 22 does not contain excavation noise; otherwise, it is determined that the audio data 22 contains excavation noise.

If the audio data 22 contains excavation sounds (S302: YES), the stability determination unit 2080 determines whether the collapsed area is greater than or equal to the first area threshold (S304). If the collapsed area is less than the first area threshold (S304: NO), the stability determination unit 2080 determines that the stability of the tunnel face 10 is at danger level 1 (S306). A situation where the stability of the tunnel face 10 is at danger level 1 is one in which excavation is taking place (excavation sounds are included in the audio data 22), but the collapse is small (the collapsed area is less than the first area threshold).

If the collapse area is greater than or equal to the first area threshold (S304: YES), the stability determination unit 2080 determines whether the type of sound represented by the audio data 22 is a heavy excavation sound (S308). If the type of sound represented by the audio data 22 is a heavy excavation sound (S308: YES), the stability determination unit 2080 determines that the stability of the tunnel face 10 is at danger level 2 (S310). If the type of sound represented by the audio data 22 is not a heavy excavation sound (S308: NO), that is, if the type of sound is a light excavation sound, the stability determination unit 2080 determines that the stability of the tunnel face 10 is at danger level 3A (S312).

Here, the situations in which the stability of the tunnel face 10 is classified as danger level 2 or danger level 3A share the following characteristics: excavation is taking place (excavation sounds are included in the audio data 22), and the collapse is relatively large (the collapsed area is greater than or equal to the first area threshold). Therefore, the stability is distinguished by the intensity of the excavation. More specifically, as mentioned above, cases with intense excavation sounds indicate harder bedrock being excavated than cases with lighter excavation sounds. Therefore, cases with intense excavation sounds are considered to have higher stability at the tunnel face 10 than cases with lighter excavation sounds. Thus, when the type of sound represented by the audio data 22 is intense excavation sound, the stability of the tunnel face 10 is judged to be higher than when the type of sound represented by the audio data 22 is light excavation sound (the former is danger level 2, and the latter is danger level 3A).

If the audio data 22 does not contain excavation sounds (S302: NO), the stability determination unit 2080 determines whether the collapsed area is greater than or equal to the second area threshold (S314). If the collapsed area is greater than or equal to the second area threshold (S314: YES), the stability determination unit 2080 determines that the stability of the tunnel face 10 is at danger level 4 (S316). A situation where the stability of the tunnel face 10 is at danger level 4 is one in which a large collapse occurs (collapsed area is greater than or equal to the second area threshold) even though no excavation is taking place (the audio data 22 does not contain excavation sounds).

If the collapsed area is less than the second area threshold (S314: NO), the stability determination unit 2080 determines whether the collapsed area is greater than or equal to the third area threshold (S318). Here, the second area threshold > the third area threshold. If the collapsed area is greater than or equal to the third area threshold (S318: YES), the stability determination unit 2080 determines that the stability of the tunnel face 10 is at danger level 3B (S320). The situation in which the stability of the tunnel face 10 is at danger level 3B is when no excavation is taking place (no excavation sounds are included in the audio data 22), and there is some degree of collapse (the collapsed area is greater than or equal to the third area threshold but less than the second area threshold).

If the collapsed area is less than the third area threshold (S318: YES), the stability determination unit 2080 reserves judgment on the stability of the tunnel face 10 (S318). Here, the situation where the audio data 22 does not contain excavation sounds and the collapsed area is less than the third area threshold indicates that no excavation is taking place and the collapse is minor. For a tunnel face 10 in such a situation, it is difficult to determine the stability without verifying the magnitude of the collapse if excavation were to occur. Therefore, the stability determination unit 2080 reserves judgment. Note that "no judgment result" may also be expressed as "no judgment result."

Note that Figures 7 and 8 show only examples of judgment rules, and the judgment rules are not limited to those shown in Figure 7.

Here, if the collapse area is calculated for each sub-region, the stability of the tunnel face 10 is determined for each sub-region. Therefore, for example, the stability determination unit 2080 determines the risk level for each sub-region based on the determination rules shown in Figures 7 and 8.

<Output of results>
The tunnel face stability evaluation device 2000 outputs information indicating the stability of the tunnel face 10, as determined by the stability determination unit 2080 (hereinafter referred to as stability information). The functional component that outputs the stability information is called the output unit. Figure 9 is a block diagram illustrating the functional configuration of the tunnel face stability evaluation device 2000 having an output unit 2100. In Figure 9, stability information 200 is output from the output unit 2100.

The information contained in the stability information 200 is diverse. For example, the stability information 200 shows time-series data of the stability of the tunnel face 10 as identified by the stability identification unit 2080. Figure 10 is an example of the stability information 200. In Figure 10, the stability information 200 shows a display 210 representing the division pattern of the tunnel face 10, and a risk graph 220, which is a graph of the time-series data of stability.

Display 210 shows how the tunnel face 10 is divided into sub-regions. In the example in Figure 10, the tunnel face 10 is divided into four sub-regions R1 to R4. The tunnel face stability evaluation device 2000 identifies the risk level for each of these four divided regions at predetermined time intervals.

Risk Graph 220 shows time-series data of risk levels for each sub-region as a line graph. In Risk Graph 220, the horizontal axis represents time, and the vertical axis represents the risk level. Here, risk levels 3A and 3B are both treated as risk level 3.

In Figure 10, the stability information 200 displays two risk graphs 220. Risk graph 220-1 shows the risk level of the tunnel face 10 in real time. On the other hand, risk graph 220-2 shows the risk level of the tunnel face 10 in the past (for example, the previous day).

The timing at which the output unit 2100 generates the stability information 200 is not limited to real time. For example, the output unit 2100 may generate and output stability information 200 for the working face 10 for a specific period at regular intervals. A concrete example is that the output unit 2100 generates stability information 200 for the working face 10 for that day once a day.

The time axis scale of the risk graph 220 may be coarser than the time scale used by the face stability evaluation device 2000 to determine the stability of the tunnel face 10. In this case, for example, the face stability evaluation device 2000 calculates statistical values of the stability of the tunnel face 10 and plots them on the risk graph 220. For example, suppose the frame rate of the camera 30 is 30fps, and the stability of the tunnel face 10 is determined for each video frame of the video data 32. Also, suppose the time axis scale of the risk graph 220 is 1 second. In this case, the output unit 2100 calculates 30 statistical values of the stability of the tunnel face 10 every second and plots these statistical values on the risk graph 220.

The information contained in the stability information 200 is not limited to the risk graph 220. For example, the output unit 2100 may generate stability information 200 indicating a warning when a risk level above a predetermined level is identified. For example, a warning may be issued when a risk level above a predetermined level is identified at least once. Alternatively, a warning may be issued when a risk level above a predetermined level is identified at a predetermined frequency or higher.

The method of warning is arbitrary. For example, the stability information 200 indicating a warning includes a message indicating that the condition of the tunnel face 10 is dangerous. This message may be a text message, an image message, or an audio message. Alternatively, for example, the stability information 200 indicating a warning may also be a warning sound.

The output method of the stability information 200 is arbitrary. For example, the output unit 2100 stores the stability information 200 in an arbitrary memory unit. Alternatively, the output unit 2100 may display the stability information 200 on an arbitrary display device (such as a display device installed at the excavation site) or play it back through an arbitrary speaker (such as a speaker installed at the excavation site). Furthermore, the output unit 2100 may transmit the stability information 200 to an arbitrary terminal (for example, a terminal used by workers or managers at the excavation site).

The present invention has been described above with reference to the embodiments described above, but the present invention is not limited to the embodiments described above. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

In the above examples, the program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a tangible storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD), or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include electrical, optical, acoustic, or other forms of propagating signals.

Some or all of the above embodiments may also be described as follows, but are not limited to the following:
(Note 1)
An acquisition unit that acquires audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification unit that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation unit calculates the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated, using the aforementioned video data.
A face stability evaluation device having a stability determination unit that determines the stability of the face to be evaluated based on the type of sound identified and the calculated collapse area.
(Note 2)
The sound type identification unit identifies whether the sound represented by the audio data is of one of the following types: a sound that does not contain excavation noise, a light excavation noise, or a heavy excavation noise.
A heavy excavation sound is a sound with a larger high-frequency component compared to a light excavation sound, according to the face stability evaluation device described in Appendix 1.
(Note 3)
The collapse area calculation unit detects a dynamic region representing a moving object from the video frame of the video data, and performs semantic segmentation on the video frame on which the dynamic region is marked to detect the dynamic region representing the collapse area, as described in Appendix 1 or 2 of the face stability evaluation device.
(Note 4)
The collapse area calculation unit uses the video data to identify an excavation point, which is the location where the excavation machine is excavating; based on the excavation point, it identifies a deposit area, which is the area where soil from the rock mass crushed by the excavation machine is deposited; and excludes the deposit area from the collapse area, as described in any one of the appendices 1 to 3, for the face stability evaluation device.
(Note 5)
The collapse area calculation unit calculates the area of the collapse area included in each of the multiple sub-regions included in the face to be evaluated as the collapse area, according to any one of the appendices 1 to 4, in the face stability evaluation device.
(Note 6)
The face stability evaluation device according to any one of the appendices 1 to 5, wherein the stability determination unit determines that the stability of the face to be evaluated is the highest of multiple stability levels when the type of sound represented by the audio data is not a sound that does not include excavation noise, and the collapse area is greater than or equal to a threshold.
(Note 7)
The face stability evaluation device according to Appendix 6, wherein the stability determination unit makes the stability of the face to be evaluated higher when the type of sound represented by the audio data is a heavy excavation sound, in the case where the area of the collapse region is less than the threshold, than the stability of the face to be evaluated when the type of sound represented by the audio data is a light excavation sound.
(Note 8)
It has an output unit that outputs stability information, which is information regarding the stability of the tunnel face to be evaluated,
The face stability evaluation device according to any one of the appendices 1 to 7, wherein the stability information includes a graph showing the change in the stability of the face to be evaluated over time.
(Note 9)
A control method performed by a computer,
Acquisition step: Acquisition step of acquiring audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification step that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation step, which uses the aforementioned video data to calculate the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated,
A control method comprising: a stability determination step of determining the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area.
(Note 10)
In the sound type identification step, it is determined whether the sound represented by the audio data is of one of the following types: a sound that does not include excavation noise, a light excavation noise, or a heavy excavation noise.
A heavy digging sound is a sound with a larger high-frequency component compared to a light digging sound, as described in Appendix 9 of the control method.
(Note 11)
The control method according to Appendix 9 or 10, wherein in the step of calculating the collapse area, a region representing a moving object is detected from the video frame of the video data, and the region representing the collapse area is detected by performing semantic segmentation on the video frame in which the region representing the moving object is marked.
(Note 12)
The control method according to any one of the appendices 9 to 11, wherein in the step of calculating the collapse area, the video data is used to identify an excavation point, which is the location where the excavation machine is excavating; based on the excavation point, an accumulation area is identified, which is the area where the soil from the rock mass crushed by the excavation machine is deposited; and the accumulation area is excluded from the collapse area.
(Note 13)
The control method according to any one of the appendices 9 to 12, wherein in the step of calculating the collapse area, for each of the multiple sub-regions included in the face to be evaluated, the area of the collapse area included in that sub-region is calculated as the collapse area.
(Note 14)
The control method according to any one of the appendices 9 to 13, wherein in the stability determination step, if the type of sound represented by the audio data is not a sound that does not include excavation noise, and the collapse area is greater than or equal to a threshold, the stability of the face to be evaluated is determined to be the highest of multiple stability levels.
(Note 15)
The control method according to Appendix 14, wherein, in the stability determination step, when the area of the collapse region is less than the threshold, the stability of the face to be evaluated when the type of sound represented by the audio data is a heavy excavation sound is set higher than the stability of the face to be evaluated when the type of sound represented by the audio data is a light excavation sound.
(Note 16)
The system has an output step that outputs stability information, which is information regarding the stability of the tunnel face being evaluated.
The control method according to any one of appendices 9 to 15, wherein the stability information includes a graph showing the change in stability of the working face to be evaluated over time.
(Note 17)
On the computer,
Acquisition step: Acquisition step of acquiring audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification step that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation step, which uses the aforementioned video data to calculate the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated,
A program that performs a stability determination step, which determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area.
(Note 18)
In the sound type identification step, it is determined whether the sound represented by the audio data is of one of the following types: a sound that does not include excavation noise, a light excavation noise, or a heavy excavation noise.
A heavy digging sound is a sound with a larger high-frequency component compared to a light digging sound, as described in Appendix 17 of the program.
(Note 19)
The program according to Appendix 17 or 18, wherein in the step of calculating the collapse area, it detects a region representing a moving object from the video frame of the video data, and detects the region representing the collapse area by performing semantic segmentation on the video frame in which the region representing the moving object is marked.
(Note 20)
A program according to any one of the appendices 17 to 19, wherein in the step of calculating the collapse area, the program uses the video data to identify the drilling point, which is the location where drilling is being performed by the drilling machine, identifies the deposition area, which is the area where soil from the rock mass crushed by the drilling machine is deposited, based on the drilling point, and excludes the deposition area from the collapse area.
(Note 21)
The program described in any one of the appendices 17 to 20, wherein in the step of calculating the collapse area, for each of the multiple sub-regions included in the face to be evaluated, the area of the collapse area included in that sub-region is calculated as the collapse area.
(Note 22)
The program according to any one of the appendices 17 to 21, wherein in the stability determination step, if the type of sound represented by the audio data is not a sound that does not include excavation noise, and the collapse area is greater than or equal to a threshold, the program determines that the stability of the face to be evaluated is the highest of several stability levels.
(Note 23)
The program as described in Appendix 22, wherein, in the stability determination step, when the area of the collapse region is less than the threshold, the stability of the face to be evaluated when the type of sound represented by the audio data is a heavy excavation sound is set higher than the stability of the face to be evaluated when the type of sound represented by the audio data is a light excavation sound.
(Note 24)
The system has an output step that outputs stability information, which is information regarding the stability of the tunnel face being evaluated.
The stability information is a program described in any one of the appendices 17 to 23, which includes a graph showing the change in stability of the working face to be evaluated over time.

10. Tunnel face 20. Microphone 22. Audio data 30. Camera 32. Video data 100. Excavation location 110. Sedimentation area 200. Stability information 210. Display 220. Risk graph 500. Computer 502. Bus 504. Processor 506. Memory 508. Storage device 510. Input/output interface 512. Network interface 2000. Tunnel face stability evaluation device 2020. Acquisition unit 2040. Sound type identification unit 2060. Collapse area calculation unit 2080. Stability identification unit 2100. Output unit.

Claims (10)

An acquisition unit that acquires audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification unit that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation unit calculates the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated, using the aforementioned video data.
It has a stability determination unit that determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area,
The collapse area calculation unit detects motion regions representing motion from video frames of the video data, and performs semantic segmentation on the video frames on which the motion regions are marked to detect the motion regions representing the collapse area, thereby enabling the detection of the motion regions representing the collapse area. An acquisition unit that acquires audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification unit that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation unit calculates the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated, using the aforementioned video data.
It has a stability determination unit that determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area,
The collapse area calculation unit uses the video data to identify the excavation point, which is the location where the excavation machine is excavating, and based on the excavation point, identifies the deposition area, which is the area where the soil from the rock mass crushed by the excavation machine is deposited, and excludes the deposition area from the collapse area, thereby providing a face stability evaluation device. An acquisition unit that acquires audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification unit that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation unit calculates the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated, using the aforementioned video data.
It has a stability determination unit that determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area,
The stability determination unit determines that, if the type of sound represented by the audio data is not a sound that does not include excavation noise, and the collapse area is less than a threshold, the stability of the face to be evaluated is the highest of several stability levels, in this face stability evaluation device. An acquisition unit that acquires audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification unit identifies which type of sound represented by the aforementioned audio data is, among the following: a sound that does not contain excavation noise, a first type of excavation noise, and a second type of excavation noise that has a larger low-frequency component than the first type of excavation noise.
A collapse area calculation unit calculates the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated, using the aforementioned video data.
It has a stability determination unit that determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area,
The stability determination unit is a face stability evaluation device that, when the area of the collapse region is greater than or equal to a threshold , sets the stability of the face to be evaluated higher when the type of sound represented by the audio data is the second type of excavation sound than the stability of the face to be evaluated when the type of sound represented by the audio data is the first type of excavation sound. The collapse area calculation unit calculates the area of the collapse region included in each of the multiple sub-regions included in the face to be evaluated as the collapse area, according to any one of claims 1 to 4. It has an output unit that outputs stability information, which is information regarding the stability of the tunnel face to be evaluated,
The face stability evaluation device according to any one of claims 1 to 5, wherein the stability information includes a graph showing the change in the stability of the face to be evaluated over time. A control method performed by a computer,
Acquisition step: Acquisition step of acquiring audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification step that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation step, which uses the aforementioned video data to calculate the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated,
The system includes a stability determination step, which determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area.
A control method comprising the steps for calculating the collapse area, wherein the method detects a region representing a moving object from the video frame of the video data, and performs semantic segmentation on the video frame on which the region representing the moving object is marked, thereby detecting the region representing the collapse area. A control method performed by a computer,
Acquisition step: Acquisition step of acquiring audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification step that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation step, which uses the aforementioned video data to calculate the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated,
The system includes a stability determination step, which determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area.
A control method comprising the steps for calculating the collapse area, wherein the video data is used to identify an excavation point, which is the location where the excavation machine is excavating, and based on the excavation point, an accumulation area, which is the area where the soil from the rock mass crushed by the excavation machine is deposited, and the accumulation area is excluded from the collapse area. A control method performed by a computer,
Acquisition step: Acquisition step of acquiring audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification step that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation step, which uses the aforementioned video data to calculate the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated,
The system includes a stability determination step, which determines the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area.
A control method in which, in the stability determination step, if the type of sound represented by the audio data is not a sound that does not include excavation noise, and the collapse area is less than a threshold, the stability of the tunnel face to be evaluated is determined to be the highest of several stability levels. On the computer,
Acquisition step: Acquisition step of acquiring audio data obtained by recording sounds around the tunnel face to be evaluated, and video data obtained by imaging the tunnel face to be evaluated,
A sound type identification step that identifies the type of sound represented by the aforementioned audio data,
A collapse area calculation step, which uses the aforementioned video data to calculate the collapse area, which is the area of the collapsed region at the face of the tunnel being evaluated,
A stability determination step is performed to determine the stability of the tunnel face to be evaluated based on the type of sound identified and the calculated collapse area.
A program that, in the step of calculating the collapse area, detects motion regions representing motion from video frames of the video data, and performs semantic segmentation on the video frames on which the motion regions are marked, thereby detecting the motion regions representing the collapse area.