JP2023153686A - Detection device and detection method - Google Patents

Abstract

To acquire an excavation position of an excavator by means of an inexpensive method.SOLUTION: A detection device 100 comprises: an image acquisition unit 30 which acquires an object image obtained by imaging the foundation and an excavator 10; a processing detection unit 31 which detects a tip of the excavator in the object image by applying image processing to the object image; an energy information acquisition unit 32 which acquires information about an excavation energy value of excavating the foundation by the excavator in association with the object image; and a position specification unit 33 which specifies the position of the tip of the excavator 10 detected in the associated object image as an excavation position when the excavation energy value equal to or greater than a prescribed threshold is acquired.SELECTED DRAWING: Figure 2

Description

The present invention relates to a detection device and a detection method for detecting the excavation position of an excavator.

Patent Document 1 describes a method of evaluating the ground in front of a face using drilling energy measured by a drill jumbo during excavation of a mountain tunnel. Further, Patent Document 1 describes that drilling data such as drilling position, drilling energy, and drilling direction angle data during drilling are automatically obtained using a computer-controlled drill jumbo.

JP2021-127655A

In the method described in Patent Document 1, a drill jumbo is provided with various sensors and the drilling position is automatically obtained by sensing the operation of the drill jumbo.

However, multiple sensors are required to obtain the drilling position from the sensor attached to the drill jumbo. Additionally, it is necessary to measure the mounting position of the sensor and the shape of the drill jumbo, and calculate the position of the tip of the drill jumbo. As described above, the method of obtaining the drilling position by sensing the operation of the drill jumbo with a sensor is expensive, and it is desired to obtain the drilling position at a lower cost.

The present invention aims to obtain the excavation position of an excavator in an inexpensive manner.

The present invention is a detection device that detects the excavation position of the ground by an arm-shaped excavator, and includes an image acquisition unit that acquires a target image of the ground and the excavator, and a target image that performs image processing on the target image. a processing detection unit that detects the tip of the excavator in the ground; an energy information acquisition unit that obtains information on the excavation energy value for excavating the ground by the excavator in association with a target image; and a position specifying unit that, when acquired, specifies the position of the tip of the excavator detected in the associated target image as the excavation position.

The present invention is a detection method for detecting the excavation position of the ground by an arm-shaped excavator, in which a target image of the ground and the excavator is acquired, image processing is performed on the target image, and the excavator in the target image is Detects the tip of the excavator, and obtains information on the excavation energy value for excavating the ground by associating it with the target image, and when an excavation energy value greater than a predetermined threshold is acquired, it is detected in the associated target image. The position of the tip of the excavator is specified as the excavation position.

According to the present invention, excavation positions can be acquired at low cost.

1 is a schematic diagram of a detection system according to an embodiment of the present invention. 1 is a block diagram showing the configuration of a detection system according to an embodiment of the present invention. 3 is a flowchart showing a detection method according to an embodiment of the present invention. 3 is a flowchart showing an evaluation method according to an embodiment of the present invention. FIG. 3 is a diagram showing a state in which an excavator is detected in a target image in a detection method according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a method of estimating the position of the tip of an excavator using a target image in a detection method according to an embodiment of the present invention. FIG. 3 is a diagram showing a state in which a face is divided into a plurality of regions in a target image in an evaluation method according to an embodiment of the present invention. It is a figure showing the evaluation image in the evaluation method concerning an embodiment of the present invention.

Hereinafter, a detection system 101 including a detection device 100 and a detection method according to an embodiment of the present invention will be described with reference to the drawings.

The detection system 101 is applied to a working machine 102 that includes an arm-shaped excavator 10 and excavates the ground with the excavator 10, and detects the position where the excavator 10 excavates the ground.

First, the overall configuration of the work machine 102 will be described with reference to FIG.

In this embodiment, the working machine 102 is a drill jumbo that drills a blast hole H for loading explosives into a face S of a tunnel T as the ground to be excavated. The working machine 102 includes an arm-shaped excavator 10 and a mobile cart 1 on which the excavator 10 is mounted.

As shown in FIG. 1, the excavator 10 includes a rod 12 with a bit 11 attached to one end, a drifter 13 that supports the other end of the rod 12, and a guide cell 14 that guides the movement of the drifter 13. . When the drifter 13 reciprocates along the guide cell 14 by a drive mechanism (not shown), the rod 12 reciprocates along the guide cell 14.

The drifter 13 can apply rotational force to the rod 12 as well as impact force. By rotating the bit 11 while pressing the bit 11 against the rock at the face S and striking the rock using the bit 11, a blast hole H is bored. The work machine 102 normally includes a plurality of excavators 10, but in this embodiment, only a single excavator 10 is illustrated for convenience of explanation.

The movable trolley 1 supports the base end of a telescopic boom 15 so as to be swingable in the vertical and horizontal directions, and the guide cell 14 of the excavator 10 is supported on the tip of the boom 15 to swing in the vertical and horizontal directions. freely supported. The boom 15 swings relative to the movable trolley 1 and expands and contracts by a drive mechanism (not shown). As the boom 15 swings and expands and contracts, the excavator 10 moves vertically and horizontally. The guide cell 14 of the excavator 10 swings relative to the boom 15 by a drive mechanism (not shown). As the guide cell 14 swings, the direction of the excavator 10 (the direction of the rod 12) changes.

By operating the excavator 10 by an operator, the face S is drilled and the blast hole H is drilled. Normally, the movable trolley 1 does not move while the excavator 10 is drilling the face S, and the positional relationship between the movable trolley 1 and the face S does not change.

As shown in FIGS. 1 and 2, the detection system 101 includes a camera 20 as an imaging unit that images the face S, a measurement unit 25 that acquires the excavation energy value by the excavator 10, and a face S that captures the excavation face S. A detection device 100 for detecting a drilling position to S is provided.

The camera 20 is attached to the movable cart 1 so as to be able to image the entire face S drilled by the excavator 10. The camera 20 continuously images the face S and acquires image data in a moving image format. Note that the camera 20 may be one that captures still images intermittently. Image data acquired by camera 20 is input to detection device 100.

The measurement unit 25 measures parameters representing the operating state of the excavator 10 necessary to obtain the excavation energy value of the excavator 10. The measuring unit 25 measures parameters representing the operating states of each of the plurality of excavators 10 in a manner that allows them to be distinguished from each other. In the present embodiment, the measurement unit 25 is composed of various sensors that measure the rotational force and impact force applied to the rod 12 from the drifter 13. For example, although not shown, the measuring unit 25 includes a torque sensor that detects the rotational force of the rod 12, a pressure sensor that detects the striking force of the rod 12, and a speed sensor that detects the moving speed of the rod 12. The measuring unit 25 measures the rotational force, striking force, and moving speed of the rod 12 in accordance with the still image capturing interval (sampling interval) by the camera 20. That is, the measurement unit 25 measures each measurement item for each frame captured by the camera 20. The measurement results of the measurement unit 25 are input to the detection device 100. Note that the measuring unit 25 only needs to have a sampling interval set so as to be able to acquire at least the excavation energy when a target image, which will be described later, is captured.

The detection device 100 includes a CPU (Central Processing Unit) that executes a control program, etc., a ROM (Read-Only Memory) that stores the control program executed by the CPU, and a RAM (Random Access Unit) that stores the calculation results of the CPU. It is composed of a computer equipped with a memory (Memory), a communication device, and the like. The detection device 100 executes various functions of the detection device 100 described in this specification by executing a control program stored in the ROM by the CPU. The detection device 100 may be configured by one computer, or may be configured by multiple microcomputers so that each control is distributed among the multiple computers.

The detection device 100 is mounted, for example, on the mobile trolley 1, and is communicably connected to the camera 20 and the measurement unit 25 by wire or wirelessly. Note that the detection device 100 is not limited to being mounted on the movable trolley 1, and may be provided at a location other than the movable trolley 1. For example, the detection device 100 may be configured by a server computer wirelessly connected to the camera 20 and the measurement unit 25 through a network.

As shown in FIG. 2, the detection device 100 includes an image acquisition unit 30 that acquires a target image from image data input from a camera 20, and an image acquisition unit 30 that performs image processing on the target image to detect the tip of the excavator 10 in the target image. an energy information acquisition unit 32 that obtains information on the excavation energy value for drilling the face S by the excavator 10 in association with the target image; A position specifying unit 33 specifies the excavation position by the machine 10, and divides the imaged face S into a plurality of regions, and determines the excavation energy value in the region based on the excavation energy value at one or more excavation positions included in each region. An energy calculation unit 34 that calculates an excavation energy value is provided. Note that each configuration of the detection device 100 shown in FIG. 2 shows each function of the detection device 100 as a virtual unit, and does not mean that it physically exists.

The image acquisition unit 30 extracts still images at predetermined time intervals from image data input in video format. The extracted still image (hereinafter referred to as “target image”) is subjected to image processing by the processing detection unit 31. The interval at which still images are extracted from the image data may be the same as the sampling interval at which the camera 20 captures still images (that is, each frame of the video is extracted as a target image), or may be configured to It can be large. The detection method described later will be described assuming that each frame of a moving image is extracted as a target image.

The processing detection unit 31 detects the tip, rear end, and the entirety of the excavator 10 in the target image using a learned model that is machine-learned using a group of images in which the tip of the excavator 10 is specified as teacher data. . The tip of the excavator 10 here refers to the tip of the rod 12 to which the bit 11 is attached.

The trained model is created by labeling the front end, rear end, and the entirety of the excavator 10 in the images by an operator for a group of images taken in advance of the face S and the excavator 10, and then labeling the labeled image group. It is constructed by having a computer perform machine learning using deep learning as training data. Further, the trained model has been trained to be able to identify each of the plurality of excavators 10. For example, a learned model that identifies the excavators 10 can be generated by assigning a marker or the like to each excavator 10 and using a group of images labeled based on the markers as training data.

Through image processing and object recognition by the processing detection unit 31, rectangular areas (bounting boxes) B1, B2, and B3 corresponding to the front end, rear end, and the entire excavator 10 are extracted (see FIG. 5). ). Further, the processing detection unit 31 has a function of estimating and detecting the position of the tip portion based on the extracted rear end portion and the entire portion when the tip portion of the excavator 10 cannot be extracted. Details of this function will be explained later.

The energy information acquisition unit 32 acquires the excavation energy value of the excavator 10 when the target image was captured based on the measurement result of the measurement unit 25. The excavation energy value is calculated based on the rotational force, striking force, and moving speed of the excavator 10. As a method for calculating the excavation energy value from the rotational force, impact force, and movement speed of the excavator 10, a known method can be adopted, and detailed explanation will be omitted. The target image and the excavation energy value at the time the target image was captured are stored in association with each other and input to the position specifying unit 33.

If the drilling energy value of the target image input from the energy information acquisition unit 32 is larger than a preset threshold, the position specifying unit 33 determines the position of the tip of the excavator 10 in the target image as a face S. It is determined that the drilling position is for. In this way, the excavation position within the target image is specified. The threshold value is set to a value larger than zero so that fluctuations (fluctuations) in the excavation energy value that occur due to sensor errors and noise of the measurement unit 25 do not affect when the excavator 10 is not excavating the face S. Set.

After specifying the excavation position in the target image, the position specifying unit 33 compares the target image with a reference image in which the shape of the face S has been specified in advance in the image, and determines the shape of the face S in the target image. Identify. Thereby, the relative positional relationship between the detected excavation position and the face S is specified.

The energy calculation unit 34 creates a grid in the target image and divides the face S in the target image into a plurality of regions. The energy calculation unit 34 calculates the excavation energy value in each divided area based on the position of the face S in the target image specified by the position specifying unit 33, the excavation position, and the excavation energy value for excavating the excavation position. Calculate. The excavation energy value of each area is one excavation energy value (representative value) corresponding to each area. When a plurality of blast holes H are provided in a region, the drilling energy value of each region is the average value of the drilling energy values for drilling each blast hole H, or among the drilling energy values of the blast holes H in the region. can be the maximum value of In the evaluation method of the face S, which will be described later, a case will be explained using an example in which the maximum value is adopted as the representative value.

Furthermore, the energy calculation unit 34 generates an evaluation image by assigning color and brightness to a plurality of divided regions within the target image according to the magnitude of the excavation energy value. The generated evaluation image is output from the detection device 100 to an external monitor. Thereby, the operator can grasp the hardness and softness state of the face S by checking the distribution of excavation energy values expressed by differences in color and brightness in the evaluation image.

Next, with reference to FIGS. 3 to 8, a method of detecting an excavation position and a method of evaluating the face S using the detection system 101 will be described.

FIG. 3 is a flowchart showing a method of detecting an excavation position performed by the detection device 100, and FIG. 4 is a flowchart showing a method of evaluating the face S. The detection device 100 detects the excavation position by the excavator 10 by image processing the target image, and evaluates the state of the face S based on the excavation position and the excavation energy value required for excavation.

First, a method for detecting an excavation position will be explained. The detection device 100 executes the excavation position detection processing shown in FIG. 3 at predetermined processing intervals. In this embodiment, the processing interval of the detection processing performed by the detection device 100 matches the time interval of frames photographed by the camera 20. In other words, the detection system 101 executes the process shown in FIG. 3 for each frame of the video captured by the camera 20. Note that the processing interval executed by the detection system 101 may be set to be equal to or longer than the frame time interval.

In step S10, one frame of a still image is extracted from the video taken by the camera 20 and acquired as a target image.

In step S11, the tip, rear end, and entire area of the excavator 10 are extracted from the target image using the trained machine learning model. Specifically, as shown in FIG. 5, a rectangular area B1 indicating the tip of the excavator 10, a rectangular area B2 indicating the rear end, and a rectangular area B3 indicating the entire excavator 10 are extracted.

Here, since the leading end of the excavator 10 excavates by contacting the face S, in the method of recognizing objects in images using a machine learning model, the leading end of the excavator 10 excavates by touching the face S. Areas are difficult to extract. Therefore, although the rear end and the entire excavator 10 are detected from the target image, the tip may not be detected. Note that even if the tip of the entire excavator 10 is not detected, it exists within a certain range in the target image and the feature amount can be extracted, so it is easier to detect than the tip.

In such a case, the detection system 101 detects the tip by estimating the position of the tip based on the relative positional relationship between the rear end of the excavator 10 and the entire excavator 10. Specifically, as shown in FIG. 6, a rectangular area B2 indicating the rear end of the excavator 10 normally exists within a rectangular area B3 indicating the entire excavator 10. The tip of the excavator 10 extends from the rear end of the excavator 10 along the main body. Therefore, when only the tip of the excavator 10 is not detected, the detection device 100 detects the center point of the rectangular area B2 of the rear end with respect to the center point P3 of the rectangular area B3 indicating the entire excavator 10. A position that is point symmetrical with P2 is estimated as the position P1 of the tip in the target image. That is, on the virtual line (double-dotted line in FIG. 6) connecting the center point P3 of the rectangular area B3 indicating the entire excavator 10 and the center point P2 of the rectangular area B2 indicating the rear end, A point P1 that is equidistant from the center point P2 of the entire body to the center point P3 of the rear end portion is detected as the tip portion. Thereby, even if the tip of the excavator 10 cannot be directly detected from the target image, the position of the hole drilled by the excavator 10 can be detected with a certain degree of accuracy.

Steps S12 and S13 are executed in parallel with the execution of steps S10 and S11. Note that steps S10 and S11 and steps S12 and S13 are not limited to parallel processing, but may be processed in series, and the order before or after them may be any order. In step S12, the measurement result by the measurement unit 20 when the target image was photographed (the measurement result corresponding to the target image) is acquired.

In step S13, the excavation energy value of the excavator 10 is calculated from the measurement results obtained in step 12.

When the excavator 10 is extracted in step S11 and the excavation energy value is calculated in step S13, it is determined in step S14 whether the excavation energy is greater than or equal to a predetermined threshold value. If the excavation energy value is smaller than the threshold, the process is immediately terminated. If the excavation energy is equal to or greater than the threshold value, the process proceeds to step S15, where the target image (more specifically, the position of the tip of the excavator 10 within the target image) and the excavation energy value are stored in association with each other. Ru. The position of the tip of the excavator 10 at this time is detected as the hole drilling position by the excavator 10. When step S15 is completed, the process ends.

The detection device 100 detects such drilling positions until, for example, a predetermined number of blast holes H are formed. When a predetermined number of blast holes H are detected, the detection device 100 performs the evaluation process shown in FIG. 4 to evaluate the face S. Note that the detection device 100 may have a configuration that executes the process shown in FIG. 4 when there is an operation input by the operator, for example. In this way, the timing for executing the process shown in FIG. 4 can be set arbitrarily.

As shown in FIG. 4, in step S20, the shape of the ground (the shape of the face S) that is the excavation target in the target image is specified. Specifically, the detection system 101 stores in advance a reference image in which the shape (position within the image) of the face S is specified. The reference image is, for example, an image showing the design shape of the face S in excavating the tunnel T. By fitting the target image to the reference image, the shape of the face S in the target image is specified. Fitting involves moving and enlarging/reducing the reference image so that the face S in the reference image overlaps the face S in the target image, and superimposes the target image and the reference image. The fitting of the reference image to the target image may be performed by the detection device 100 or may be performed manually by an operator. Thereby, the excavation position, which was a coordinate position within the target image, is understood as a position relative to the face S.

Next, in step S21, as shown in FIG. 7, the face S is divided into a plurality of regions using grid lines in the target image. Note that the spacing and shape of the grid lines are not limited to the example shown in FIG. 6, and can be set arbitrarily. Further, each point shown in FIG. 7 indicates an excavation position by the excavator 10.

In step S22, one excavation energy value (representative value) is calculated for each cell C divided by the grid lines. When a plurality of blast holes H are formed in the cell C, the representative value of the excavation energy value is the maximum value of the excavation energy values for the plurality of blast holes H. Moreover, when the blast hole H is not formed in the cell C, the excavation energy value of the cell C is calculated as 0 (zero).

Next, in step S23, an evaluation image is generated in which the cells C are color-coded depending on the magnitude of the excavation energy value (see FIG. 8). Then, in step S24, the evaluation image is output to, for example, an external monitor, and the process ends. This allows the worker to determine the hardness or softness of the ground by checking the evaluation image.

According to the above embodiment, the following effects are achieved.

In this embodiment, the tip of the excavator 10 is detected by image processing, and based on the excavation energy value, it can be determined whether the excavator 10 is excavating the ground. That is, since the tip of the excavator 10 excavating the ground can be detected by image processing, the excavation position with respect to the ground can be specified. In this way, since the position of the tip of the excavator 10 can be specified by image processing without using a sensor, the number of sensors for sensing the operation of the excavator 10 can be reduced, and the excavation position can be acquired at a lower cost. I can do it.

Further, in the present embodiment, only when the excavation energy value is equal to or greater than the threshold value, the position of the tip of the excavator 10 at the time when the excavation energy value is calculated is detected as the excavation position. An evaluation image is generated based on the excavation position and excavation energy value. Thereby, the hardness and softness of the face S can be grasped. By understanding the hardness and softness of the face S, it is possible to manage the charge to the blast hole H, predict the hardness and softness of the ground that will be excavated from now on (such as a new face S that will be created after blasting), and support that will be installed inside the tunnel T. evaluation and management of engineering work, etc.

Next, a modification of this embodiment will be described.

In the embodiment described above, the leading end and the trailing end of the excavator 10 are detected in the target image using a machine-learned learning model. The detection system 101 may calculate the angle of the excavator 10 and thus the drilling angle based on the leading end and the trailing end. Since the total length of the excavator 10 can be known in advance, it is possible to calculate the excavation angle from the total length of the excavator 10 and the positions of the front end and rear end of the excavator 10. The excavation angle calculated in this manner may be used, for example, for evaluating the accuracy of drilling.

Further, in the above embodiment, the detection device 100 is applied to a drill jumbo that drills a face S, but is not limited to this, and is also applied to, for example, a road header that excavates the ground with the excavator 10. may be done.

Further, in the embodiment described above, when the excavation energy value becomes equal to or greater than the threshold value, the position of the tip of the excavator 10 at that time is acquired as the excavation position. This configuration does not prevent the position of the tip of the excavator 10 from being stored when the excavation energy value has not reached the threshold value.

Further, in the above embodiment, by comparing (fitting) the target image and the reference image, the shape of the face S in the target image is specified, and the relative position of the blast hole H with respect to the face S is grasped. In the above embodiment, the reference image is based on the designed shape of the face S, but for example, after actually imaging the face S, the shape of the face S in the image can be changed through image processing or processing. It may be detected manually by a person and used as a reference image. Moreover, the relative position of the blast hole H with respect to the face S is specified not limited to the method of comparing the target image and the reference image. Even without using a reference image, by understanding the actual shape of the face S and the relative position of the camera 20 with respect to the face S in advance, the face S in the target image captured by the camera 20 can be It is also possible to specify the shape of . It is also possible to perform image processing on the target image, detect the boundary between the face S and the inner wall of the tunnel T, and specify the shape of the face S in the target image.

Further, in the embodiment described above, the processing detection unit 31 detects the excavator 10 in the target image using a machine-learned learning model, but the detection of the excavator 10 is not limited to image processing using the learning model. In addition, when using a trained model, if the ground to be excavated is updated or changed, new training data is created by labeling the image of the ground, and new training data is created. It is desirable to perform additional learning on data.

Although the embodiments of the present invention have been described above, the above embodiments merely show a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. do not have.

The series of processes in the detection device 100 described above may be provided as a program for a computer to execute. Further, a program for executing the series of processes described above is provided by a storage medium readable by the detection device 100. Further, the program may be provided to the detection device 100 through a network line. Further, the various programs executed by the detection device 100 may be stored in a non-transitory recording medium such as a CD-ROM.

100 Detection device 10 Excavator 20 Camera (imaging unit)
25 Measuring unit 30 Image acquisition unit 31 Processing detection unit 32 Energy information acquisition unit 33 Position identification unit 34 Energy calculation unit

Claims (6)

A detection device for detecting the position of excavation in the ground by an arm-shaped excavator,
an image acquisition unit that acquires a target image of the ground and the excavator;
a processing detection unit that performs image processing on the target image to detect a tip of the excavator in the target image;
an energy information acquisition unit that acquires information on an excavation energy value for excavating the ground with the excavator in association with the target image;
a position specifying unit that specifies the position of the tip of the excavator detected in the associated target image as the excavation position when the excavation energy value that is equal to or greater than a predetermined threshold is acquired;
A detection device characterized by: The detection device according to claim 1,
The processing detection unit detects the tip of the excavator using a trained model that is machine learned using teacher data.
A detection device characterized by: The detection device according to claim 1 or 2,
The processing detection unit is configured to detect the front end, rear end, and the whole of the excavator, and when the front end cannot be detected, the position of the rear end and the whole is determined. detecting the position of the tip from the relationship;
A detection device characterized by: The detection device according to claim 1 or 2,
The position specifying unit specifies the shape of the ground in the target image by comparing the reference image in which the shape of the ground in the image is specified with the target image, and identifies the shape of the ground in the target image, and determines the relationship between the ground and the excavation position. identify the positional relationship of
A detection device characterized by: 5. The detection device according to claim 4,
Further comprising an energy calculation unit that divides the ground into a plurality of regions and calculates the excavation energy value in the region based on the excavation energy value at one or more excavation positions included in each region,
A detection device characterized by: A detection method for detecting the position of excavation in the ground by an arm-shaped excavator,
Obtaining a target image of the ground and the excavator;
performing image processing on the target image to detect the tip of the excavator in the target image; and acquiring information on an excavation energy value for excavating the ground by the excavator in association with the target image;
When the excavation energy value equal to or greater than a predetermined threshold is acquired, identifying the position of the tip of the excavator detected in the associated target image as the excavation position;
A detection method characterized by: