JP2023151682A - Shovel and shovel support system - Google Patents

Abstract

To provide a system that can more easily generate three-dimensional data of a feature surrounding a shovel.SOLUTION: A shovel support system 50 supports work by a shovel that includes a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and a camera 20 mounted on the upper rotating body. The shovel support system generates three-dimensional data of a feature around the shovel on the basis of a plurality of images acquired by the camera 20 while the shovel is in operation. The camera 20 may be a camera used for monitoring the surroundings of the shovel. The plurality of images may be stored in a drive recorder.SELECTED DRAWING: Figure 3

Description

TECHNICAL FIELD This disclosure relates to excavators and excavator support systems.

Conventionally, there has been known a work management system for construction machines that is configured to generate three-dimensional data of an area in which work by the construction machine has progressed at a work site based on images captured by a camera attached to an aircraft. (For example, see Patent Document 1).

International Publication No. 2017/170651

However, this system requires dedicated equipment such as a flying vehicle for acquiring images. Therefore, this system has a problem in that the introduction cost and operation cost are high.

Therefore, it is desirable to provide a system that can more easily generate three-dimensional data of features around the excavator.

A shovel support system according to an embodiment of the present invention supports work by a shovel including a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and a camera mounted on the upper rotating body. The excavator support system generates three-dimensional data of features around the excavator based on a plurality of images acquired by the camera while the excavator is in operation.

The excavator support system described above can more easily generate three-dimensional data of features around the excavator.

FIG. 1 is a side view of an excavator according to an embodiment of the present invention. FIG. 3 is a top view of the excavator. FIG. 1 is a block diagram showing an example of the configuration of a human detection system mounted on an excavator. It is a top view of the excavator which is moving forward. It is a figure showing an example of the figure displayed on the screen of a display device. 1 is a schematic diagram showing a communication network to which an excavator is connected; FIG.

First, with reference to FIGS. 1 and 2, a shovel 100 as an excavator according to an embodiment of the present invention will be described. FIG. 1 is a side view of the shovel 100, and FIG. 2 is a top view of the shovel 100.

As shown in FIGS. 1 and 2, an upper rotating body 3 is rotatably mounted on a lower traveling body 1 of an excavator 100 via a rotating mechanism 2. As shown in FIGS. A boom 4 is attached to the upper revolving body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4 is hydraulically driven by a boom cylinder 7, the arm 5 is hydraulically driven by an arm cylinder 8, and the bucket 6 is hydraulically driven by a bucket cylinder 9. The upper revolving body 3 is provided with a cabin 10 as a driver's cab, and is equipped with a power source such as an engine. Furthermore, a camera 20, a display device D1, an indoor alarm D2, a drive recorder D3, a controller 30, and the like are attached to the upper revolving body 3.

The camera 20 is configured to capture images of the surroundings of the excavator 100. The camera 20 is configured with an imaging device such as a CCD or CMOS, and is used to monitor the surroundings of the excavator 100. In the example shown in FIGS. 1 and 2, the camera 20 includes four monocular cameras attached to the upper surface of the upper rotating body 3. The four monocular cameras are a rear camera 20B attached to the rear end of the upper surface of the upper revolving body 3, a front camera 20F attached to the front end of the upper surface of the upper revolving body 3, and a left end of the upper surface of the upper revolving body 3. A left camera 20L is attached to the top surface of the upper rotating body 3, and a right camera 20R is attached to the right end of the upper surface of the upper rotating body 3. FIG. 2 schematically shows the imaging range CZ of the camera 20 with a dashed line. Specifically, the imaging range CZ includes an imaging range CZB of the rear camera 20B, an imaging range CZF of the front camera 20F, an imaging range CZL of the left camera 20L, and an imaging range CZR of the right camera 20R. The four monocular cameras are preferably attached to the revolving upper structure 3 so as not to protrude from the upper surface of the revolving upper structure 3, as shown in FIG. Note that at least one of the front camera 20F and the left camera 20L may be omitted.

The camera 20 is configured to be able to output the acquired image to any device. In the illustrated example, the camera 20 is configured to output images repeatedly acquired at a predetermined cycle to each of the display device D1 and the drive recorder D3. The camera 20 and a device that receives images captured by the camera 20 are connected to each other via a video cable. Note that the camera 20 and the device that receives images acquired by the camera 20 may be connected to each other via a dedicated communication cable.

The display device D1 is a device that displays various information, and is, for example, a liquid crystal display or an organic EL display. The indoor alarm device D2 is a device that outputs a warning to the operator inside the cabin 10, and is configured with, for example, a buzzer, a speaker, an LED lamp, a seat vibrator, etc. installed inside the cabin 10.

The drive recorder D3 is configured to be able to continuously record images acquired by the camera 20. In the example shown in FIG. 2, the drive recorder D3 is configured to record images acquired by the camera 20 at a predetermined recording cycle while the excavator 100 is in operation. "While the excavator 100 is in operation" means, for example, a period during which power is supplied to each of the camera 20 and the drive recorder D3. However, "while the excavator 100 is in operation" may be a period in which the engine 11 is in operation, or may be a period in which an operating device such as an operating lever is being operated.

Controller 30 is an example of a processing circuit, and functions as a control device for controlling shovel 100. In the illustrated example, the controller 30 is configured with a computer including a CPU, a volatile storage device, a nonvolatile storage device, and the like.

Next, with reference to FIG. 3, the shovel support system 50 mounted on the shovel 100 will be described. FIG. 3 is a block diagram showing a configuration example of the shovel support system 50.

The shovel support system 50 is a system that supports work by the shovel 100, and is mainly composed of a display device D1, an indoor alarm D2, a drive recorder D3, a camera 20, a controller 30, and the like. In the illustrated example, the shovel support system 50 is configured to be able to detect an object that has entered a predetermined detection range set around the shovel 100. The object may be a person, vehicle, work machine, animal, or the like. In the example shown in FIG. 3, the shovel support system 50 is configured to be able to detect people.

In the excavator support system 50 shown in FIG. 3, the controller 30, for example, runs programs corresponding to each functional element such as a person detection section 31, an image generation section 32, a three-dimensional data generation section 33, and a work amount calculation section 34 on a CPU. The functions corresponding to each functional element are realized by executing the function.

The person detection unit 31 is configured to detect a person who has entered a predetermined detection range set around the excavator 100. In the example shown in FIG. 3, the person detection unit 31 determines whether or not a person is present based on the image output by the camera 20. The human detection unit 31 can distinguish between humans and objects other than humans, such as trees or buildings. The person detection unit 31 determines the presence or absence of a person within a predetermined detection range by finding a person image in an image using, for example, an image feature such as a HOG feature. The human image is an image registered in advance as a human image. The person detection unit 31 may use other known image recognition techniques to determine the presence or absence of a person within a predetermined detection range.

The predetermined detection range is, for example, a space included in the imaging range CZ of the camera 20. In the example shown in FIG. 3, the predetermined detection range is a space included in the imaging range CZ of the camera 20 that can be reached by the toe of the bucket 6. The predetermined detection range may be a space corresponding to the imaging range CZ of the camera 20, or may be a space within a predetermined distance from the rotation axis of the excavator 100.

The predetermined detection range may include a plurality of partial detection ranges. For example, the rear detection range behind the shovel 100 is a space included in the imaging range CZB of the rear camera 20B, the rear detection range behind the shovel 100 is a space included in the imaging range CZF of the front camera 20F, and the left camera The left detection range to the left of the shovel 100 is a space included in the imaging range CZL of the right camera 20L, and the right detection range to the right of the shovel 100 is a space included in the imaging range CZR of the right camera 20R. It may also be configured in combination. Two adjacent partial detection ranges may be arranged so as to partially overlap, may be arranged adjacently so as not to overlap, or may be arranged with an interval between them.

The human detection unit 31 also detects people within a predetermined detection range based on the output of an object detection sensor such as an ultrasonic sensor, a laser radar sensor, a millimeter wave sensor, a pyroelectric infrared sensor, a bolometer infrared sensor, or an infrared camera. The presence or absence of . For example, when using the output of a pyroelectric infrared sensor, the human detection unit 31 determines that a person exists within the detection range when the energy change within the detection range exceeds a predetermined threshold. In this case, the predetermined detection range may be a space corresponding to the detection range of the pyroelectric infrared sensor.

When detecting a person, the person detection unit 31 outputs a control command to the indoor alarm D2 to activate the indoor alarm D2, and notifies the operator of the excavator 100 that a person has been detected. In the example shown in FIG. 3, the human detection unit 31 activates a buzzer installed in the cabin 10.

The image generation unit 32 is a functional element that generates an output image based on the image acquired by the camera 20. The output image is an image for display. In the example shown in FIG. 3, the image generation unit 32 generates an output image to be displayed on the display device D1 based on images acquired by each of the rear camera 20B, front camera 20F, left camera 20L, and right camera 20R. . Then, the generated output image is displayed on the display device D1. The output image may include a composite image composed of one or more images. The composite image is, for example, a viewpoint conversion image generated by performing viewpoint conversion processing on a composite image obtained by combining four images obtained by each of the rear camera 20B, front camera 20F, left camera 20L, and right camera 20R. It's okay. In this case, the viewpoint conversion image may be an overhead image showing the state around the excavator 100 when viewed from above. The bird's-eye view image may be, for example, an image that is arranged so as to surround a computer graphics (CG) image representing the excavator 100 and that is trimmed so that the outer contour line draws an arc. Note that the bird's-eye view image may be trimmed so that the outer contour line becomes a circular arc having an arbitrary curvature, or may be trimmed so that the outer contour line becomes an elliptical arc, or the outer contour line may be trimmed so that the outer contour line becomes an arbitrary convex curve. It may be trimmed to look like this. Further, the output image may include an image showing the state of the surroundings of the shovel 100 when viewed from the side (a through image). The through image shows, for example, the space behind the shovel 100 as viewed almost horizontally from the shovel 100 side. The through image may be cropped such that the outer contour line includes at least one straight line portion. For example, the through image may be trimmed so that the outer contour line becomes a square, a rectangle, a trapezoid, or the like. Alternatively, the through image may be the image itself acquired by the camera 20, that is, an image that has not been subjected to viewpoint conversion processing.

The three-dimensional data generation unit 33 is a functional element that generates three-dimensional data of features around the excavator 100. The terrestrial features include all objects that exist around the shovel 100. The three-dimensional data of the feature includes three-dimensional data of the ground around the excavator 100. In the example shown in FIG. 3, the three-dimensional data generation unit 33 uses photogrammetry technology such as SfM (Structure from Motion) technology based on the camera image acquired by the camera 20 to create a map of features around the excavator 100. Generate three-dimensional data.

Photogrammetry technology is a technology that uses a computer to analyze photographs taken of an object from various directions and generates a three-dimensional model. SfM technology is a type of photogrammetry technology that prepares a large number of images of the object to be measured from various positions and angles, and analyzes the correspondence between the images using software. This is a technology to acquire three-dimensional point cloud data.

In the example shown in FIG. 3, the three-dimensional data generation unit 33 generates three-dimensional data of features around the shovel 100 based on images recorded on the drive recorder D3 at a predetermined recording cycle. Specifically, at a predetermined timing, the three-dimensional data generation unit 33 generates three-dimensional data using all or part of the images stored in a predetermined storage area in the nonvolatile storage device included in the drive recorder D3. Generate data. The predetermined timing is, for example, when one day's work is finished. In this case, the three-dimensional data generation unit 33 may automatically start generating three-dimensional data at a predetermined timing. Note that the predetermined timing may be, for example, the time when a predetermined button provided in the cabin 10 is pressed, or the time when a predetermined operation is performed. In this case, the three-dimensional data generation unit 33 may start generating three-dimensional data in response to a predetermined operation.

More specifically, the three-dimensional data generation unit 33 uses a large number of images stored in a predetermined first storage area in chronological order during a first period from a predetermined start time to a predetermined end time. Generate three-dimensional data. The large number of images is, for example, tens to tens of thousands of images. The three-dimensional data generation unit 33 generates first three-dimensional data using a large number of images stored in the first storage area during the first period, and also generates first three-dimensional data using a large number of images stored in the first storage area during the first period. The second three-dimensional data may be generated using a large number of images stored in a second storage area that is separate from the first storage area. In this way, the three-dimensional data generation unit 33 can generate a plurality of time-series three-dimensional data.

The three-dimensional data generation unit 33 may generate two or more three-dimensional data regarding features around the excavator 100. The two or more three-dimensional data may include, for example, three-dimensional data at a time before starting one day's work and three-dimensional data at a time after finishing one day's work. In this case, the controller 30 compares the three-dimensional data before starting the day's work with the three-dimensional data at the time after finishing the day's work, and determines how the three-dimensional data has changed. By understanding how much the excavator 100 has done, it is possible to understand how the shapes of features around the excavator 100 have changed depending on the day's work.

Alternatively, two or more pieces of 3D data include 3D data at the time before starting the day's work, 3D data at the time after finishing the work in the morning, and 3D data about the day's work. It may also include three-dimensional data at a time after the end. In this case, the controller 30 compares the three-dimensional data at the time before starting the day's work with the three-dimensional data at the time after finishing the work in the morning of the day, to determine the surroundings of the excavator 100. You can see how the shape of the features changed as a result of the morning's work. In addition, the controller 30 compares the three-dimensional data at the point in time after finishing the work in the morning of the day with the three-dimensional data at the point in time after finishing the day's work. You can see how the shape of the feature changed as a result of the afternoon's work.

Note that the end time of the first period may be later than the start time of the second period. That is, the first period and the second period may partially overlap. In this case, the three-dimensional data generation unit 33 may generate three-dimensional data using a part of the image stored in the first storage area and a part of the image stored in the second storage area.

Alternatively, the three-dimensional data generation unit 33 may generate one or more three-dimensional data in real time. Specifically, each time a new image is recorded, the three-dimensional data generation unit 33 generates one piece of three-dimensional data using the previously recorded image and the newly recorded image. Alternatively, each time a new image is recorded, two or more three-dimensional data may be generated simultaneously using the previously recorded image and the newly recorded image. That is, the three-dimensional data generation unit 33 may reflect the content of the newly recorded image on the already generated three-dimensional data in real time.

The three-dimensional data generation unit 33 may display the generated three-dimensional data on the display device D1. In this case, the three-dimensional data can be expressed by any three-dimensional model such as a polygon model or a point cloud model. In this case, the display device D1 may be configured so that the operator of the excavator 100 can view the three-dimensional model from any virtual viewpoint. Further, the display device D1 may be configured so that the operator of the excavator 100 can view the three-dimensional model in an enlarged or reduced size.

The work amount calculation unit 34 is a functional element that calculates the work amount of the excavator 100. The amount of work performed by the shovel 100 is expressed, for example, by the volume of excavated earth and sand. In the example shown in FIG. 3, the work amount calculation unit 34 uses three-dimensional data (first three-dimensional data) of features around the shovel 100 at a first time point and data about features around the shovel 100 at a second time point. Based on the three-dimensional data (second three-dimensional data), the amount of work performed by the shovel 100 regarding the current work is calculated. The current work is, for example, today's excavation work using the shovel 100. In this case, the first time point is, for example, a time point after the previous work (yesterday's excavation work) is finished and before the current work (today's excavation work) is started. Further, the second time point is, for example, a time point after the current work (today's excavation work) is finished and before the next work (tomorrow's excavation work) is started.

Specifically, the work amount calculation unit 34 calculates the difference between the first three-dimensional data and the second three-dimensional data based on the first three-dimensional data and the second three-dimensional data. This difference corresponds to the volume difference between the volume of the feature around the shovel 100 at the first time point and the volume of the feature around the shovel 100 at the second time point. The work amount calculation unit 34 calculates this volume difference as the volume of earth and sand excavated by today's excavation work, that is, as the work amount of the shovel 100 for today's day. The same applies to cases where work other than excavation work is performed, such as loading work or backfilling work.

The workload calculation unit 34 may display information regarding the calculated workload on the display device D1. Further, the work amount calculation unit 34 displays the three-dimensional data of the excavated part, which is the difference between the first three-dimensional data and the second three-dimensional data, so that it can be distinguished from the three-dimensional data of other parts. It may be displayed on the device D1.

Further, the work amount calculation unit 34 may be configured to calculate the amount of completed work based on the calculated work amount. Amount of work is a contract price that corresponds to the amount of work that is completed. Finished form is the finished part of the object of work, that is, the part for which construction has been completed. The work amount calculation unit 34 can recognize the portions where construction has been completed by comparing the three-dimensional data generated by the three-dimensional data generation unit 33 with pre-stored design data. Note that the design data is, for example, three-dimensional data representing the shape of the ground when construction is completed.

Next, with reference to FIG. 4, images used by the controller 30 when generating three-dimensional data will be described. FIG. 4 is a top view of the excavator 100 while moving forward, showing changes in the position of the imaging range CZ of the camera 20. Specifically, for the sake of clarity, FIG. 4 shows only the transition of the position of the imaging range CZR of the right camera 20R, the imaging range CZB of the rear camera 20B, the imaging range CZF of the front camera 20F, and the left camera 20L. The illustration of the transition of each position of the imaging range CZL is omitted. However, the positions of the imaging range CZB of the rear camera 20B, the imaging range CZF of the front camera 20F, and the imaging range CZL of the left camera 20L also change in the same way as the position of the imaging range CZR of the right camera 20R.

More specifically, in the example shown in FIG. 4, the imaging range CZR of the right camera 20R includes a first imaging range CZR1 to a fourth imaging range CZR4. The first imaging range CZR1 represents the imaging range CZR of the right camera 20R at the first time, and the second imaging range CZR2 represents the imaging range CZR of the right camera 20R at the second time which is earlier than the first time by a predetermined recording cycle. represents. Similarly, the third imaging range CZR3 represents the imaging range CZR of the right camera 20R at a third time earlier than the second time by a predetermined recording cycle, and the fourth imaging range CZR4 represents the imaging range CZR at a predetermined recording period earlier than the third time. It represents the imaging range CZR of the right camera 20R at the fourth time, which is earlier by a period.

The right camera 20R chronologically records images acquired at a predetermined recording cycle in a predetermined storage area in a nonvolatile storage device included in the drive recorder D3.

If there is no change in the shape of the features around the excavator 100, the three-dimensional data generation unit 33 generates a three-dimensional No new information can be added to the original data. Therefore, each of the large number of images used by the three-dimensional data generation unit 33 to generate three-dimensional data may be an image acquired at a different viewpoint position, or even at the same viewpoint position, the camera 20 may be viewed in a different direction. It is desirable that the image be taken when the subject is facing the subject.

Specifically, as shown in FIG. 4, it is desirable that the image be acquired by camera 20 while shovel 100 is moving. Note that "while the shovel 100 is moving" refers not only to when the shovel 100 is moving forward as shown in FIG. 4, but also when the shovel 100 is moving backwards or when the shovel 100 is turning This includes when you are present.

Furthermore, when generating three-dimensional data, the three-dimensional data generation unit 33 may be configured not to redundantly use images acquired by the cameras 20 facing the same direction at the same viewpoint position. This is to reduce the processing load on the controller 30 regarding the generation of three-dimensional data. For example, the three-dimensional data generation unit 33 may be configured not to use images acquired when the shovel 100 is not moving when generating three-dimensional data. Note that "when the shovel 100 is not moving" includes when the shovel 100 is in a standby state. The standby state refers to a state of the excavator 100 when the engine 11 is in operation and no operating device such as an operating lever is operated.

Alternatively, the three-dimensional data generating section 33 may be configured to select an image to be used for generating three-dimensional data from among a large number of images recorded on the drive recorder D3. For example, the three-dimensional data generation unit 33 determines the position of the camera 20 when one image used for generating three-dimensional data was acquired, and the position of the camera 20 when one image used for generating three-dimensional data was acquired. The image used for generating the three-dimensional data may be selected such that the distance between the camera 20 and the current position of the camera 20 is sufficiently large. In this case, each of the many images stored in the drive recorder D3 may include a flag value indicating whether or not the image can be used to generate three-dimensional data. The flag may be configured, for example, to be set to "1" each time the shovel 100 moves by a predetermined distance, and set to "0" each time the camera 20 acquires an image. Alternatively, the flag may be configured to become "1" each time the shovel 100 turns by a predetermined angle, for example.

Next, with reference to FIG. 5, an example of a graphic displayed on the display device D1 will be described. FIG. 5 is a diagram showing an example of a graphic displayed on the screen of the display device D1. Specifically, the figure shown in FIG. 5 is a figure related to the three-dimensional data of features around the excavator 100 generated by the three-dimensional data generation unit 33, and the cross-section of the slope to be constructed before the slope is completed. It shows how the shape has changed. More specifically, the figure shown in FIG. 5 includes a ground line TL representing the position of the ground surface derived from the three-dimensional data and a shovel figure representing the current position of the shovel 100.

In the example shown in FIG. 5, the ground line TL includes a first ground line TL1 to a fourth ground line TL4. The first ground line TL1 represented by a solid line indicates the cross-sectional shape of the slope after the slope is completed. A second ground line TL2 represented by a broken line indicates the cross-sectional shape of the slope at the time before work starts on the day the slope is completed. A third ground line TL3 represented by a dashed-dotted line indicates the cross-sectional shape of the slope at the time before the work starts on the day before the slope is completed. A fourth ground line TL4 represented by a two-dot chain line indicates the cross-sectional shape of the slope at the time before the work starts two days before the slope is completed.

In addition, in the figure shown in FIG. 5, a diagonal line pattern is attached to the cross section of the ground located below the first ground line TL1, and the area between the first ground line TL1 and the second ground line TL2 (where the slope is The area (representing the soil excavated by the work on the day of completion) is marked with a fine dot pattern. In addition, in the figure shown in FIG. 5, the area between the second ground line TL2 and the third ground line TL3 (the area representing the earth and sand excavated by the work the day before the day when the slope is completed) has a coarse dot pattern. is added, and a coarser dot pattern is added to the area between the third ground line TL3 and the fourth ground line TL4 (the area representing the earth and sand excavated by the work two days before the day the slope is completed). has been done.

The operator of the excavator 100 who views this screen can grasp the progress of work from two days before the day on which the slope is completed to the day on which the slope is completed. Further, when the work volume calculation unit 34 calculates the volume of completed work, the controller 30 displays the volume of the day before the previous day, the volume of the previous day, and the volume of each day of the current day on the screen. It's okay.

Next, with reference to FIG. 6, another configuration example of the shovel support system 50 will be described. FIG. 6 is a schematic diagram showing a communication network 200 to which the excavator 100 is connected. In the example shown in FIG. 6, the excavator 100 includes a communication device D4.

The communication device D4 is configured to control communication with external equipment outside the excavator 100. In the example shown in FIG. 6, the communication device D4 controls communication with external devices via the communication network 200.

The communication network 200 is configured to interconnect the excavator 100, the base station 21, the server 22, and the communication terminal 23. Communication network 200 includes, for example, at least one of known communication networks such as a satellite communication network, a mobile phone communication network, and an Internet network. The base station 21 may be omitted.

The communication terminal 23 includes a mobile communication terminal 23a, a fixed communication terminal 23b, and the like. Excavator 100, base station 21, server 22, and communication terminal 23 are connected to each other using, for example, a communication protocol such as Internet protocol. The number of each of the excavator 100, base station 21, server 22, and communication terminal 23 connected via the communication network 200 may be one or more. The mobile communication terminal 23a may be a notebook computer, a tablet PC, a mobile phone, a smartphone, a smart watch, smart glasses, or the like.

The base station 21 is an external facility that receives information transmitted by the excavator 100. Information is transmitted and received between the base station 21 and the excavator 100 through at least one of, for example, a satellite communication network, a mobile phone communication network, and an Internet network.

The server 22 is configured to function as a management device for the excavator 100. In the example shown in FIG. 6, the server 22 is a device installed in an external facility such as a management center, and stores and manages information transmitted by the excavator 100. The server 22 is, for example, a computer equipped with a CPU, ROM, RAM, input/output interface, input device, display, and the like. Specifically, the server 22 acquires and stores information received by the base station 21 through the communication network 200, and manages the information so that the operator (administrator) can refer to the stored information as needed. The management device may include a monitor and an operating device for remote control. In this case, the operator remotely controls the shovel 100 using a remote control operating device. The operating device for remote control may be connected to the controller 30 of the excavator 100 through a communication network such as a mobile phone communication network, a satellite communication network, or a short-range wireless communication network.

The server 22 may be configured to allow an operator (administrator) to perform one or more settings regarding the excavator 100 via the communication network 200. Specifically, the server 22 transmits to the excavator 100 values related to one or more settings performed by the operator (administrator), and changes the values related to the one or more settings stored in the controller 30. You may.

The server 22 may transmit information regarding the excavator 100 to the communication terminal 23 via the communication network 200. Specifically, the server 22 transmits information regarding the excavator 100 to the communication terminal 23 when a predetermined condition is met or in response to a request from the communication terminal 23, and transmits information regarding the excavator 100 to the communication terminal 23. The information may be communicated to the operator of the terminal 23.

The communication terminal 23 functions as a support device for the excavator 100. In the example shown in FIG. 6, the communication terminal 23 is a device that can refer to information stored in the server 22, and is, for example, a computer equipped with a CPU, ROM, RAM, input/output interface, input device, display, etc. be. The communication terminal 23 may be connected to the server 22 through the communication network 200, for example, and may be configured so that an operator (administrator) can view information regarding the excavator 100. That is, the communication terminal 23 may be configured to receive information regarding the excavator 100 transmitted by the server 22, and allow the operator (administrator) to view the received information.

In the example shown in FIG. 6, the server 22 manages information regarding the shovel 100 that the shovel 100 has transmitted. Therefore, the operator (administrator) can view information regarding the excavator 100 at any time through the display attached to the server 22 or the communication terminal 23.

In such a communication network 200, functional elements such as the person detection section 31, image generation section 32, three-dimensional data generation section 33, and work amount calculation section 34, which constitute the excavator support system 50, are connected to devices other than the controller 30. It may also be realized by a processing circuit.

For example, the three-dimensional data generation unit 33 may be realized by a processing circuit installed in the server 22 or may be realized by a processing circuit installed in the communication terminal 23. The same applies to the person detection section 31, the image generation section 32, and the work amount calculation section 34.

Specifically, the three-dimensional data generation unit 33 realized by a processing circuit installed in the server 22 may receive an image acquired by the camera 20 installed in the excavator 100 through the communication network 200. The three-dimensional data generation unit 33 may then generate three-dimensional data of features around the shovel 100 based on the large number of received images. The communication terminal 23 refers to the three-dimensional data generated by the three-dimensional data generation unit 33 and stored in the server 22, and displays a three-dimensional model based on the three-dimensional data on the display device attached to the communication terminal 23. may be displayed.

As described above, the excavator support system 50 according to the embodiment of the present invention includes the lower traveling body 1, the upper rotating body 3 rotatably mounted on the lower traveling body 1, and the camera mounted on the upper rotating body 3. The shovel 100 is configured to support work by the shovel 100 including the The excavator support system 50 is configured to generate three-dimensional data of features around the excavator 100 based on a plurality of images acquired by the camera 20 while the excavator 100 is in operation.

With this configuration, the excavator support system 50 can generate three-dimensional data of features around the excavator 100 without requiring additional equipment such as a stereo camera or LIDAR. For example, the excavator support system 50 can generate three-dimensional data of the features around the excavator 100 using images acquired by the camera 20 and recorded on the drive recorder D3. Measurement of objects can be achieved at low cost.

The images acquired by the camera 20 are preferably one or more first images acquired before the topography around the shovel 100 changes, and one or more first images acquired after the topography around the shovel 100 changes. A second image of. In this case, the shovel support system 50 can compare the three-dimensional data generated based on the one or more first images by the controller 30 with the three-dimensional data generated based on the one or more second images. Make it. From this comparison, controller 30 can derive, for example, the volume of the area where the terrain has changed.

In the illustrated example, the camera 20 is a camera used to monitor the surroundings of the excavator 100. Specifically, the image acquired by the camera 20 is used to detect objects (people) existing around the excavator 100. With this configuration, the excavator support system 50 does not need to use an additional camera different from the existing camera 20, so it is possible to survey the features around the excavator 100 at low cost. However, the camera 20 may be a dedicated camera used to generate three-dimensional data of features around the excavator 100.

The three-dimensional data preferably includes first three-dimensional data generated before the topography around the shovel 100 changes, and second three-dimensional data generated after the topography around the shovel 100 changes. . In this case, the controller 30 may be configured to calculate the amount of work performed by the shovel 100 based on the first three-dimensional data and the second three-dimensional data. Further, the controller 30 may be configured to calculate the amount of completed work based on the calculated amount of work. Then, the controller 30 may display the calculated amount of work or output on the screen of the display device D1. By viewing the displayed information, the operator of the excavator 100 can confirm the daily amount of work or performance regarding completed work.

The controller 30 derives the position coordinates of the viewpoint of the camera 20 when the camera 20 acquires the image based on the output of the positioning device D5 (see FIG. 6 described later) mounted on the excavator 100, and calculates the position coordinates of the viewpoint. The positional coordinates of each point of the three-dimensional data may be derived based on the three-dimensional data.

With this configuration, the shovel support system 50 can more accurately grasp the positional relationship between the shovel 100 and the features around the shovel 100. Therefore, the shovel support system 50 can perform various functions using, for example, the distance between the shovel 100 and features around the shovel 100, and can more effectively support the work performed by the shovel 100.

The plurality of images acquired by the camera 20 may be stored in the drive recorder D3. With this configuration, the excavator support system 50 can generate three-dimensional data of features around the excavator 100 using a large number of images stored in the drive recorder D3.

Further, the plurality of images acquired by the camera 20 may be transmitted from the excavator 100 to a server 22 or the like located outside the excavator 100 via wireless communication by the communication device D4 shown in FIG. Alternatively, the plurality of images acquired by the camera 20 may be transmitted to the server 22 or the like via wired communication. Specifically, the plurality of images acquired by the camera 20 may be recorded on a removable recording medium (portable nonvolatile memory) such as a USB memory, a memory card, or a flash memory (registered trademark). In this case, the operator of the excavator 100 removes the flash memory inserted into the drive recorder D3 and inserts the flash memory into a device such as a flash memory reader installed in an office or the like, so that the camera 20 can be activated. The acquired image can be transmitted to the server 22 or the like.

With this configuration, the excavator support system 50 can generate three-dimensional data of features around the excavator 100 based on images acquired by the camera 20 outside the excavator 100.

The preferred embodiments of the present invention have been described above in detail. However, the invention is not limited to the embodiments described above. Various modifications or substitutions may be made to the embodiments described above without departing from the scope of the present invention. Further, features described separately can be combined as long as no technical contradiction occurs.

For example, the excavator 100 may be equipped with a positioning device D5 configured to measure the position of the upper revolving structure 3, as shown in FIG. For example, the positioning device D5 may be a GNSS compass. Further, the positioning device D5 may be configured to detect the position and orientation of the upper rotating body 3 and output the detected value to the controller 30. The controller 30 may be configured to be able to calculate the respective positions of the rear camera 20B, front camera 20F, left camera 20L, and right camera 20R based on the output of the positioning device D5. In this case, the controller 30 may utilize the output of a turning angle sensor or the like.

When the positioning device D5 is mounted on the excavator 100, the shovel support system 50 derives the position coordinates of the viewpoint of the camera 20 when the camera 20 acquired the image based on the output of the positioning device D5, and determines the position of the viewpoint. The positional coordinates of each point of the three-dimensional data may be derived based on the coordinates.

Further, when the positioning device D5 is mounted on the excavator 100, the three-dimensional data generation unit 33 generates three-dimensional data from among the many images recorded on the drive recorder D3 based on the output of the positioning device D5. It may be configured to select images to be used for generation. For example, the three-dimensional data generation unit 33 determines the position of the camera 20 when one image used for generating three-dimensional data was acquired, and the position of the camera 20 when one image used for generating three-dimensional data was acquired. The image used for generating the three-dimensional data may be selected such that the distance between the image and the current position of the camera 20 is a predetermined distance or more. That is, the three-dimensional data generation unit 33 converts another image acquired by the camera 20 before the camera 20 moves a predetermined distance or more from the position of the camera 20 when acquiring the image selected for generation of three-dimensional data into three-dimensional data. The image may be configured not to be selected as an image to be used for generating original data. In this case, each of the many images stored in the drive recorder D3 may include information regarding the position of the camera 20 when each image was acquired.

Furthermore, in the embodiment described above, the excavator support system 50 is configured to generate three-dimensional data of features around the excavator 100 based on images acquired by the camera 20 mounted on one excavator 100. ing. However, the excavator support system 50 may be configured to generate three-dimensional data of features around the excavator 100 based on images obtained by another camera mounted on another excavator. In this case, the excavator support system 50 uses an image obtained by the camera 20 mounted on the excavator 100 and an image obtained by another camera mounted on another excavator to identify features around the excavator 100. Three-dimensional data may be generated, or three-dimensional data of features around excavator 100 may be generated using only images acquired by another camera mounted on another excavator. Further, another camera mounted on another excavator may be a camera mounted on another machine such as a dump truck, a bulldozer, or a road roller.

1...Lower traveling body 2...Swivel mechanism 3...Upper rotating body 4...Boom 5...Arm 6...Bucket 7...Boom cylinder 8...Arm cylinder 9...・Bucket cylinder 10...Cabin 20...Camera 20B...Rear camera 20F...Front camera 20L...Left camera 20R...Right camera 21...Base station 22...Server 23. ...Communication terminal 23a...Mobile communication terminal 23b...Fixed communication terminal 30...Controller 31...Person detection section 32...Image generation section 33...Three-dimensional data generation section 34... Work amount calculation unit 50...Shovel support system 100...Shovel 200...Communication network D1...Display device D2...Indoor alarm D3...Drive recorder D4...Communication device D5...・Positioning device

Claims (11)

An excavator support system for supporting work by an excavator, comprising a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and a camera mounted on the upper rotating body,
generating three-dimensional data of features around the excavator based on a plurality of images acquired by the camera during operation of the excavator;
Excavator support system. The images include a plurality of first images acquired before the topography around the shovel changes, and a plurality of second images acquired after the topography around the shovel changes.
The excavator support system according to claim 1. The camera is a monocular camera used to monitor the surroundings of the excavator,
The excavator support system according to claim 1 or 2. The three-dimensional data includes first three-dimensional data generated before the topography around the shovel changes, and second three-dimensional data generated after the topography around the shovel changes.
The excavator support system according to claim 3. calculating the amount of work performed by the shovel based on the first three-dimensional data and the second three-dimensional data;
The excavator support system according to claim 4. Deriving the positional coordinates of the viewpoint of the camera when the camera acquired the image based on the output of the positioning device mounted on the excavator, and determining the position of each point of the three-dimensional data based on the positional coordinates of the viewpoint. derive the coordinates,
A shovel support system according to any one of claims 1 to 5. The plurality of images are stored in a drive recorder,
A shovel support system according to any one of claims 1 to 6. The plurality of images are transmitted from the excavator to a server external to the excavator via wireless communication or wired communication.
An excavator support system according to any one of claims 1 to 7. the plurality of images are stored in a portable non-volatile memory;
An excavator support system according to any one of claims 1 to 7. The three-dimensional data is generated by a processing circuit external to the excavator.
A shovel support system according to any one of claims 1 to 9. a lower running body;
an upper rotating body rotatably mounted on the lower traveling body;
a camera mounted on the upper rotating body;
a control device that generates three-dimensional data of features around the excavator based on a plurality of images acquired by the camera during operation of the excavator;
shovel.

Images