US20210272315A1 - Transport object specifying device of work machine, work machine, transport object specifying method of work machine, method for producing complementary model, and dataset for learning - Google Patents

Transport object specifying device of work machine, work machine, transport object specifying method of work machine, method for producing complementary model, and dataset for learning Download PDF

Info

Publication number
US20210272315A1
US20210272315A1 US17/260,069 US201917260069A US2021272315A1 US 20210272315 A1 US20210272315 A1 US 20210272315A1 US 201917260069 A US201917260069 A US 201917260069A US 2021272315 A1 US2021272315 A1 US 2021272315A1
Authority
US
United States
Prior art keywords
transport object
drop target
dump body
specifying
dimensional position
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US17/260,069
Other languages
English (en)
Inventor
Shun Kawamoto
Shintaro Hamada
Yosuke KAJIHARA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Komatsu Ltd
Original Assignee
Komatsu Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Komatsu Ltd filed Critical Komatsu Ltd
Publication of US20210272315A1 publication Critical patent/US20210272315A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F3/00—Dredgers; Soil-shifting machines
    • E02F3/04—Dredgers; Soil-shifting machines mechanically-driven
    • E02F3/28—Dredgers; Soil-shifting machines mechanically-driven with digging tools mounted on a dipper- or bucket-arm, i.e. there is either one arm or a pair of arms, e.g. dippers, buckets
    • E02F3/36—Component parts
    • E02F3/42—Drives for dippers, buckets, dipper-arms or bucket-arms
    • E02F3/43—Control of dipper or bucket position; Control of sequence of drive operations
    • E02F3/435—Control of dipper or bucket position; Control of sequence of drive operations for dipper-arms, backhoes or the like
    • E02F3/439—Automatic repositioning of the implement, e.g. automatic dumping, auto-return
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices
    • E02F9/261—Surveying the work-site to be treated
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • the present invention relates to a transport object specifying device of a work machine, a work machine, a transport object specifying method of a work machine, a method for producing a complementary model, and a dataset for learning.
  • Japanese Unexamined Patent Application, First Publication No. 2001-71809 discloses a technique of calculating a position of the center of gravity of a transport object based on the output of a weighting sensor provided on a transport vehicle and displaying a loaded state of the transport object.
  • the position of the center of gravity of a drop target such as the transport vehicle can be determined, but a three-dimensional position of the transport object in the drop target cannot be specified.
  • An object of the present invention is to provide a transport object specifying device of a work machine, a work machine, a transport object specifying method of a work machine, a method for producing a complementary model, and a dataset for learning capable of specifying a three-dimensional position of a transport object in a drop target.
  • a transport object specifying device of a work machine includes an image acquisition unit that acquires a captured image showing a drop target of the work machine in which a transport object is dropped, a drop target specifying unit that specifies a three-dimensional position of at least part of the drop target based on the captured image, a three-dimensional data generation unit that generates depth data which is three-dimensional data representing a depth of the captured image, based on the captured image, and a surface specifying unit that specifies a three-dimensional position of a surface of the transport object in the drop target by removing, from the depth data, a part corresponding to the drop target based on the three-dimensional position of the at least part of the drop target.
  • the transport object specifying device can specify the distribution of the transport object in the drop target.
  • FIG. 1 is a diagram showing a configuration of a loading place according to one embodiment.
  • FIG. 2 is an external view of a hydraulic excavator according to one embodiment.
  • FIG. 3 is a schematic block diagram showing a configuration of a control device according to a first embodiment.
  • FIG. 4 is a diagram showing an example of a configuration of a neural network.
  • FIG. 5 is an example of guidance information.
  • FIG. 6 is a flowchart showing a display method of the guidance information by the control device according to the first embodiment.
  • FIG. 7 is a flowchart showing a learning method of a feature point specifying model according to the first embodiment.
  • FIG. 8 is a flowchart showing a learning method of a complementary model according to the first embodiment.
  • FIG. 9 is a schematic block diagram showing a configuration of a control device according to a second embodiment.
  • FIG. 10 is a flowchart showing a display method of guidance information by the control device according to the second embodiment.
  • FIG. 11A is a diagram showing a first example of a method for calculating an amount of a transport object in a dump body.
  • FIG. 11B is a diagram showing a second example of the method for calculating the amount of the transport object in the dump body.
  • FIG. 1 is a diagram showing a configuration of a loading place according to one embodiment.
  • a hydraulic excavator 100 which is a loading machine and a dump truck 200 which is a transport vehicle are provided.
  • the hydraulic excavator 100 scoops a transport object L such as earth from the construction site and loads the transport object in the dump truck 200 .
  • the dump truck 200 transports the transport object L loaded by the hydraulic excavator 100 to a predetermined earth removable place.
  • the dump truck 200 includes a dump body 210 which is a container for accommodating the transport object L.
  • the dump body 210 is an example of a drop target in which the transport object L is dropped.
  • FIG. 2 is an external view of a hydraulic excavator according to one embodiment.
  • the hydraulic excavator 100 includes work equipment 110 that is hydraulically operated, a swing body 120 that supports the work equipment 110 , and a travel body 130 that supports the swing body 120 .
  • the swing body 120 is provided with a cab 121 in which an operator rides.
  • the cab 121 is provided in a front portion of the swing body 120 and is positioned on a left-side (+Y side) of the work equipment 110 .
  • the hydraulic excavator 100 includes a stereo camera 122 , an operation device 123 , a control device 124 , and a display device 125 .
  • the stereo camera 122 is provided in an upper portion of the cab 121 .
  • the stereo camera 122 is installed in an upper (+Z direction) and front (+X direction) portion of the cab 121 .
  • the stereo camera 122 captures an image in front (+X direction) of the cab 121 through a windshield on a front surface of the cab 121 .
  • the stereo camera 122 includes at least one pair of cameras.
  • the operation device 123 is provided inside the cab 121 .
  • the operation device 123 is operated by the operator to supply hydraulic oil to an actuator of the work equipment 110 .
  • the control device 124 acquires information from the stereo camera 122 to generate guidance information indicating a distribution of the transport object in the dump body 210 of the dump truck 200 .
  • the control device 124 is an example of a transport object specifying device.
  • the display device 125 displays the guidance information generated by the control device 124 .
  • the hydraulic excavator 100 may not necessarily include the stereo camera 122 and the display device 125 .
  • the stereo camera 122 includes a right-side camera 1221 and a left-side camera 1222 .
  • Examples of each camera include a camera using a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor.
  • CCD charge coupled device
  • CMOS complementary metal oxide semiconductor
  • the right-side camera 1221 and the left-side camera 1222 are installed at an interval in a left-right direction (Y-axis direction) such that optical axes of the cameras 1221 and 1222 are substantially parallel to a floor surface of the cab 121 .
  • the stereo camera 122 is an example of an imaging device.
  • the control device 124 can calculate a distance between the stereo camera 122 and a captured target by using an image captured by the right-side camera 1221 and an image captured by the left-side camera 1222 .
  • the image captured by the right-side camera 1221 is also referred to as a right-eye image.
  • the image captured by the left-side camera 1222 is also referred to as a left-eye image.
  • a combination of the images captured by respective cameras of the stereo camera 122 is also referred to as a stereo image.
  • the stereo camera 122 may be configured of three or more cameras.
  • FIG. 3 is a schematic block diagram showing a configuration of the control device according to the first embodiment.
  • the control device 124 includes a processor 91 , a main memory 92 , a storage 93 , and an interface 94 .
  • the storage 93 stores a program for controlling the work equipment 110 .
  • Examples of the storage 93 include a hard disk drive (HDD) and a non-volatile memory.
  • the storage 93 may be an internal medium directly connected to a bus of the control device 124 , or may be an external medium connected to the control device 124 through the interface 94 or a communication line.
  • the storage 93 is an example of a storage unit.
  • the processor 91 reads the program from the storage 93 , expands the program in the main memory 92 , and executes processing according to the program.
  • the processor 91 secures a storage area in the main memory 92 according to the program.
  • the interface 94 is connected to the stereo camera 122 , the display device 125 , and other peripheral devices, and transmits and receives signals.
  • the main memory 92 is an example of the storage unit.
  • the processor 91 includes a data acquisition unit 1701 , a feature point specifying unit 1702 , a three-dimensional data generation unit 1703 , a dump body specifying unit 1704 , a surface specifying unit 1705 , a distribution specifying unit 1706 , a distribution estimation unit 1707 , a guidance information generation unit 1708 , and a display control unit 1709 .
  • the storage 93 stores a camera parameter CP, a feature point specifying model M 1 , a complementary model M 2 , and a dump body model VD.
  • the camera parameter CP is information indicating a position relationship between the swing body 120 and the right-side camera 1221 and a position relationship between the swing body 120 and the left-side camera 1222 .
  • the dump body model VD is a three-dimensional model representing a shape of the dump body 210 .
  • three-dimensional data representing a shape of the dump truck 200 may be used instead of the dump body model VD.
  • the dump body model VD is an example of a target model.
  • the program may be for realizing part of functions to be exerted by the control device 124 .
  • the program may exert a function by a combination with another program already stored in the storage 93 or a combination with another program installed in another device.
  • the control device 124 may include a custom large scale integrated circuit (LSI) such as a programmable logic device (PLD) in addition to or instead of the above configuration.
  • LSI large scale integrated circuit
  • PLD programmable logic device
  • Examples of the PLD include a programmable array logic (PAL), a generic array logic (GAL), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA).
  • PAL programmable array logic
  • GAL generic array logic
  • CPLD complex programmable logic device
  • FPGA field programmable gate array
  • the data acquisition unit 1701 acquires the stereo image from the stereo camera 122 through the interface 94 .
  • the data acquisition unit 1701 is an example of an image acquisition unit.
  • the data acquisition unit 1701 may acquire a stereo image from a stereo camera provided in another work machine, a stereo camera installed at the construction site, or the like.
  • the feature point specifying unit 1702 inputs the right-eye image of the stereo image acquired by the data acquisition unit 1701 to the feature point specifying model M 1 stored in the storage 93 to specify positions of a plurality of feature points of the dump body 210 shown in the right-eye image.
  • Examples of the feature point of the dump body 210 include upper and lower ends of a front panel of the dump body 210 , an intersection of a guard frame of the front panel and a side gate, and upper and lower ends of a fixed post of a tailgate. That is, the feature point is an example of a predetermined position of the drop target.
  • the feature point specifying model M 1 includes a neural network 140 shown in FIG. 4 .
  • FIG. 4 is a diagram showing an example of a configuration of the neural network.
  • the feature point specifying model M 1 is realized by, for example, a trained model of deep neural network (DNN).
  • the trained model is configured of a combination of a training model and a trained parameter.
  • the neural network 140 includes an input layer 141 , one or more intermediate layers 142 (hidden layers), and an output layer 143 .
  • Each of the layers 141 , 142 , and 143 includes one or more neurons.
  • the number of neurons in the intermediate layer 142 can be set as appropriate.
  • the output layer 143 can be set as appropriate according to the number of feature points.
  • Neurons in the layers adjacent to each other are connected to each other, and a weight (connection load) is set for each connection.
  • the number of connected neurons may be set as appropriate.
  • a threshold value is set for each neuron, and an output value of each neuron is determined by whether or not a sum of products of an input value and the weight for each neuron exceeds the threshold value.
  • An image showing the dump body 210 of the dump truck 200 is input to the input layer 141 .
  • an output value indicating a probability of the pixel being the feature point is output to the output layer 143 .
  • the feature point specifying model M 1 is a trained model which is trained, when an image showing the dump body 210 is input, to output the positions of the feature points of the dump body 210 in the image.
  • the feature point specifying model M 1 is trained by using, for example, a dataset for learning with an image showing the dump body 210 of the dump truck 200 as training data and with an image obtained by plotting the positions of the feature points of the dump body 210 as teaching data.
  • the teaching data is an image in which a pixel related to the plot has a value indicating that the probability of the pixel being the feature point is 1, and other pixel has a value indicating that the probability of the pixel being the feature point is 0.
  • the teaching data may be information of which a pixel related to the plot has a value indicating that the probability of the pixel being the feature point is 1, and other pixel has a value indicating that the probability of the pixel being the feature point is 0, and may not be an image.
  • “training data” refers to data input to the input layer during training of the training model.
  • “teaching data” is data which is a correct answer for comparison with the value of the output layer of the neural network 140 .
  • “dataset for learning” refers to a combination of the training data and the teaching data.
  • the trained parameters of the feature point specifying model M 1 obtained by training are stored in the storage 93 .
  • the trained parameters include, for example, the number of layers of the neural network 140 , the number of neurons in each layer, the connection relationship between the neurons, the weight of each connection between the neurons, and the threshold value of each neuron.
  • the same or similar DNN configuration as a DNN configuration used for detecting a facial organ or a DNN configuration used for estimating a posture of a person can be used as the configuration of the neural network 140 of the feature point specifying model M 1 .
  • the feature point specifying model M 1 is an example of a position specifying model.
  • the feature point specifying model M 1 according to another embodiment may be trained by unsupervised learning or reinforcement learning.
  • the three-dimensional data generation unit 1703 generates a three-dimensional map representing a depth in an imaging range of the stereo camera 122 by stereo measurement using the stereo image and the camera parameters stored in the storage 93 . Specifically, the three-dimensional data generation unit 1703 generates point group data indicating a three-dimensional position by the stereo measurement of the stereo image.
  • the point group data is an example of depth data.
  • the three-dimensional data generation unit 1703 may generate an elevation map generated from the point group data as three-dimensional data instead of the point group data.
  • the dump body specifying unit 1704 specifies a three-dimensional position of the dump body 210 based on the positions of the feature points specified by the feature point specifying unit 1702 , the point group data specified by the three-dimensional data generation unit 1703 , and the dump body model VD. Specifically, the dump body specifying unit 1704 specifies three-dimensional positions of the feature points based on the positions of the feature points specified by the feature point specifying unit 1702 and the point group data specified by the three-dimensional data generation unit 1703 . Next, the dump body specifying unit 1704 fits the dump body model VD to the three-dimensional positions of the feature points to specify the three-dimensional position of the dump body 210 . In another embodiment, the dump body specifying unit 1704 may specify the three-dimensional position of the dump body 210 based on the elevation map.
  • the surface specifying unit 1705 specifies a three-dimensional position of a surface of the transport object L on the dump body 210 based on the point group data generated by the three-dimensional data generation unit 1703 and the three-dimensional position of the dump body 210 specified by the dump body specifying unit 1704 . Specifically, the surface specifying unit 1705 cuts out a part above a bottom surface of the dump body 210 from the point group data generated by the three-dimensional data generation unit 1703 to specify the three-dimensional position of the surface of the transport object L on the dump body 210 .
  • the distribution specifying unit 1706 generates a dump body map indicating a distribution of an amount of the transport object L on the dump body 210 based on the three-dimensional position of the bottom surface of the dump body 210 specified by the dump body specifying unit 1704 and the three-dimensional position of the surface of the transport object L specified by the surface specifying unit 1705 .
  • the dump body map is an example of distribution information.
  • the dump body map is, for example, an elevation map of the transport object L with reference to the bottom surface of the dump body 210 .
  • the distribution estimation unit 1707 generates a dump body map in which a value is complemented for a part of the dump body map that does not have a value of height data. That is, the distribution estimation unit 1707 estimates a three-dimensional position of a shielded part of the dump body map that is shielded by an obstacle to update the dump body map. Examples of the obstacle include the work equipment 110 , the tailgate of the dump body 210 , and the transport object L.
  • the distribution estimation unit 1707 inputs the dump body map into the complementary model M 2 stored in the storage 93 to generate a dump body map in which the height data is complemented.
  • the complementary model M 2 is realized by, for example, a trained model of DNN including the neural network 140 shown in FIG. 4 .
  • the complementary model M 2 is a trained model which is trained, when a dump body map including a grid without the height data is input, to output a dump body map in which all grids have the height data.
  • the complementary model M 2 is trained with a combination of a complete dump body map in which all grids have the height data, which is generated by simulation or the like, and an incomplete dump body map in which part of the height data is removed from the complete dump body map, as a dataset for learning.
  • the complementary model M 2 according to another embodiment may be trained by unsupervised learning or reinforcement learning.
  • the guidance information generation unit 1708 generates the guidance information from the dump body map generated by the distribution estimation unit 1707 .
  • FIG. 5 is an example of the guidance information.
  • the guidance information generation unit 1708 generates the guidance information for displaying a two-dimensional heat map indicating a distribution of the height from the bottom surface of the dump body 210 to the surface of the transport object L.
  • Granularity of vertical and horizontal divisions in the heat map shown in FIG. 5 is an example and is not limited thereto in another embodiment.
  • the heat map according to another embodiment may represent, for example, a ratio of a height of the transport object L to a height related to an upper limit of the loading of the dump body 210 .
  • the display control unit 1709 outputs a display signal for displaying the guidance information to the display device 125 .
  • the learning unit 1801 performs learning processing of the feature point specifying model M 1 and the complementary model M 2 .
  • the learning unit 1801 may be provided in a device separate from the control device 124 . In this case, the trained model which has been trained in the separate device will be recorded in the storage 93 .
  • FIG. 6 is a flowchart showing a display method of the guidance information by the control device according to the first embodiment.
  • the data acquisition unit 1701 acquires the stereo image from the stereo camera 122 (step S 1 ).
  • the feature point specifying unit 1702 inputs the right-eye image of the stereo image acquired by the data acquisition unit 1701 to the feature point specifying model M 1 stored in the storage 93 to specify the positions of the plurality of feature points of the dump body 210 shown in the right-eye image (step S 2 ).
  • the feature point of the dump body 210 include the upper and lower ends of the front panel of the dump body 210 , the intersection of the guard frame of the front panel and the side gate, and the upper and lower ends of the fixed post of the tailgate.
  • the feature point specifying unit 1702 may input the left-eye image to the feature point specifying model M 1 to specify the positions of the plurality of feature points.
  • the three-dimensional data generation unit 1703 generates the point group data of the entire imaging range of the stereo camera 122 by the stereo measurement using the stereo image acquired in step S 1 and the camera parameters stored in the storage 93 (step S 3 ).
  • the dump body specifying unit 1704 specifies the three-dimensional positions of the feature points based on the positions of the feature points specified in step S 2 and the point group data generated in step S 3 (step S 4 ). For example, the dump body specifying unit 1704 specifies, using the point group data, a three-dimensional point corresponding to the pixel showing the feature point in the right-eye image to specify the three-dimensional position of the feature point. The dump body specifying unit 1704 fits the dump body model VD stored in the storage 93 to the specified positions of the feature points to specify the three-dimensional position of the dump body 210 (step S 5 ).
  • the dump body specifying unit 1704 may convert a coordinate system of the point group data into a dump body coordinate system having a corner of the dump body 210 as the origin, based on the three-dimensional position of the dump body 210 .
  • the dump body coordinate system can be represented as, for example, a coordinate system composed of an X-axis extending in a width direction of the front panel, a Y-axis extending in a width direction of the side gate, and a Z-axis extending in a height direction of the front panel, with a lower left end of the front panel as the origin.
  • the dump body specifying unit 1704 is an example of a drop target specifying unit.
  • the surface specifying unit 1705 extracts, from the point group data generated in step S 3 , a plurality of three-dimensional points in a prismatic area, which is surrounded by the front panel, the side gate, and the tailgate of the dump body 210 specified in step S 5 and extends in the height direction of the front panel, to remove three-dimensional points corresponding to the background from the point group data (step S 6 ).
  • the front panel, the side gate, and the tailgate form a wall portion of the dump body 210 .
  • the surface specifying unit 1705 sets threshold values determined based on a known size of the dump body 210 on the X-axis, the Y-axis, and the Z-axis to extract three-dimensional points in an area defined from the thresholds.
  • the height of the prismatic area may be equal to the height of the front panel or may be higher than the height of the front panel by a predetermined length.
  • the transport object L can be accurately extracted even in a case where the transport object L is loaded higher than the height of the dump body 210 by making a height of the prismatic area higher than that of the front panel.
  • the prismatic area may be an area narrowed inward by a predetermined distance from the area surrounded by the front panel, the side gate, and the tailgate.
  • the dump body model VD is a simple 3 D model in which thicknesses of the front panel, the side gate, the tailgate, and the bottom surface are not accurate, an error in the point group data can be reduced.
  • the surface specifying unit 1705 removes three-dimensional points corresponding to the position of the dump body model VD from the plurality of three-dimensional points extracted in step S 6 to specify the three-dimensional position of the surface of the transport object L loaded on the dump body 210 (step S 7 ).
  • the distribution specifying unit 1706 generates the dump body map which is an elevation map representing the height in the height direction of the front panel with the bottom surface of the dump body 210 as a reference height, based on the plurality of three-dimensional points extracted in step S 6 and the bottom surface of the dump body 210 (step S 8 ).
  • the dump body map may include a grid without the height data.
  • the distribution specifying unit 1706 can generate the dump body map by obtaining an elevation map with an XY plane as the reference height and with the Z-axis direction as the height direction.
  • the distribution estimation unit 1707 inputs the dump body map generated in step S 7 into the complementary model M 2 stored in the storage 93 to generate the dump body map in which the height data is complemented (step S 8 ).
  • the guidance information generation unit 1708 generates the guidance information shown in FIG. 5 based on the dump body map (step S 9 ).
  • the display control unit 1709 outputs the display signal for displaying the guidance information to the display device 125 (step S 10 ).
  • steps S 2 to S 4 and steps S 7 to S 10 among the processing by the control device 124 shown in FIG. 6 may not be executed.
  • the positions of the feature points in the left-eye image may be specified from the positions of the feature points in the right-eye image by the stereo matching to specify the three-dimensional positions of the feature points using triangulation.
  • point group data only in the prismatic area which is surrounded by the front panel, the side gate, and the tailgate of the dump body 210 specified in step S 5 and extends in the height direction of the front panel may be generated. In this case, since it is not necessary to generate the point group data of the entire imaging range, the calculation load can be reduced.
  • FIG. 7 is a flowchart showing a learning method of the feature point specifying model M 1 according to the first embodiment.
  • the data acquisition unit 1701 acquires the training data (step S 101 ).
  • the training data in the feature point specifying model M 1 is an image showing the dump body 210 .
  • the training data may be acquired from an image captured by the stereo camera 122 .
  • the training data may be acquired from an image captured by another work machine.
  • An image showing a work machine different from the dump truck, for example, an image showing a dump body of a wheel loader may be used as the training data. It is possible to improve robustness of dump body recognition by using dump bodies of various types of work machines as the training data.
  • the learning unit 1801 performs training of the feature point specifying model M 1 .
  • the learning unit 1801 performs training of the feature point specifying model M 1 using the combination of the training data acquired in step S 101 and the teaching data which is the image obtained by plotting the positions of the feature points of the dump body, as the dataset for learning (step S 102 ).
  • the learning unit 1801 uses the training data as an input to perform calculation processing of the neural network 140 in a forward propagation direction. Accordingly, the learning unit 1801 obtains an output value output from the output layer 143 of the neural network 140 .
  • the dataset for learning may be stored in the main memory 92 or the storage 93 .
  • the learning unit 1801 calculates an error between the output value from the output layer 143 and the teaching data.
  • the output value from the output layer 143 is a value representing the probability of a pixel being the feature point, and the teaching data is the information obtained by plotting the position of the feature point.
  • the learning unit 1801 calculates an error of the weight of each connection between the neurons and an error of the threshold value of each neuron by backpropagation from the calculated error of the output value.
  • the learning unit 1801 updates the weight of each connection between the neurons and the threshold value of each neuron based on the calculated errors.
  • the learning unit 1801 determines whether or not the output value from the feature point specifying model M 1 matches the teaching data (step S 103 ). It may be determined that the output value matches the teaching data when an error between the output value and the teaching data is within a predetermined value. In a case where the output value from the feature point specifying model M 1 does not match the teaching data (step S 103 : NO), the above processing is repeated until the output value from the feature point specifying model M 1 matches the teaching data. As a result, the parameters of the feature point specifying model M 1 are optimized, and the feature point specifying model M 1 can be trained.
  • the learning unit 1801 records the feature point specifying model M 1 as a trained model including the parameters optimized by the training in the storage 93 (step S 104 ).
  • FIG. 8 is a flowchart showing a learning method of the complementary model according to the first embodiment.
  • the data acquisition unit 1701 acquires the complete dump body map in which all grids have the height data as teaching data (step S 111 ).
  • the complete dump body map is generated, for example, by simulation or the like.
  • the learning unit 1801 randomly removes a part of the height data of the complete dump body map to generate the incomplete dump body map as training data (step S 112 ).
  • the learning unit 1801 performs training of the complementary model M 2 .
  • the learning unit 1801 performs training of the complementary model M 2 with the combination of the training data generated in step S 112 and the teaching data acquired in step S 111 as the dataset for learning (step S 113 ).
  • the learning unit 1801 uses the training data as an input to perform calculation processing of the neural network 140 in a forward propagation direction. Accordingly, the learning unit 1801 obtains an output value output from the output layer 143 of the neural network 140 .
  • the dataset for learning may be stored in the main memory 92 or the storage 93 .
  • the learning unit 1801 calculates an error between the dump body map output from the output layer 143 and the complete dump body map as the teaching data.
  • the learning unit 1801 calculates an error of the weight of each connection between the neurons and an error of threshold value of each neuron by backpropagation from the calculated error of the output value.
  • the learning unit 1801 updates the weight of each connection between the neurons and the threshold value of each neuron based on the calculated errors.
  • the learning unit 1801 determines whether or not the output value from the complementary model M 2 matches the teaching data (step S 114 ). It may be determined that the output value matches the teaching data when an error between the output value and the teaching data is within a predetermined value. In a case where the output value from the complementary model M 2 does not match the teaching data (step S 114 : NO), the above processing is repeated until the output value from the complementary model M 2 matches the complete dump body map. As a result, the parameters of the complementary model M 2 are optimized, and the complementary model M 2 can be trained.
  • the learning unit 1801 records the complementary model M 2 as a trained model including the parameters optimized by the training in the storage 93 (step S 115 ).
  • the control device 124 specifies the three-dimensional positions of the surface of the transport object L and the bottom surface of the dump body 210 based on the captured image, and generates the dump body map indicating the distribution of the amount of the transport object L on the dump body 210 based on the three-dimensional positions. Accordingly, the control device 124 can specify the distribution of the transport object L on the dump body 210 . The operator can recognize the drop position of the transport object L for loading the transport object L on the dump body 210 in a well-balanced manner by recognizing the distribution of the transport object L on the dump body 210 .
  • the control device 124 estimates the distribution of the amount of the transport object L in the shielded part of the dump body map shielded by an obstacle. Accordingly, the operator can recognize the distribution of the amount of the transport object L even for a part of the dump body 210 that is shielded by the obstacle and cannot be captured by the stereo camera 122 .
  • the control device 124 specifies the distribution of the transport object L on the dump body 210 based on a type of the transport object L.
  • FIG. 9 is a schematic block diagram showing a configuration of a control device according to the second embodiment.
  • the control device 124 further includes a type specifying unit 1710 .
  • the storage 93 stores a type specifying model M 3 and a plurality of complementary models M 2 according to the type of the transport object L.
  • the type specifying unit 1710 inputs an image of the transport object L to the type specifying model M 3 to specify the type of the transport object L shown in the image.
  • Examples of the type of transport object include clay, sand, gravel, rock, and wood.
  • the type specifying model M 3 is realized by, for example, a trained model of deep neural network (DNN).
  • the type specifying model M 3 is a trained model which is trained, when an image showing the transport object L is input, to output the type of the transport object L.
  • a DNN configuration of the type specifying model M 3 for example, the same or similar DNN configuration as a DNN configuration used for image recognition can be used.
  • the type specifying model M 3 is trained, for example, using a combination of an image showing the transport object L and a label representing the type of the transport object L as teaching data.
  • the type specifying model M 3 is trained using a combination of an image showing the transport object L and label data representing the type of the transport object L as the teaching data.
  • the type specifying model M 3 may be trained by transfer learning of a general trained image recognition model.
  • the type specifying model M 3 according to another embodiment may be trained by unsupervised learning or reinforcement learning.
  • the storage 93 stores the complementary model M 2 for each type of the transport object L.
  • the storage 93 stores a complementary model M 2 for clay, a complementary model M 2 for sand, a complementary model M 2 for gravel, a complementary model M 2 for rock, and a complementary model M 2 for wood.
  • Each complementary model M 2 is trained, for example, using a combination of a complete dump body map generated by simulation or the like according to the type of the transport object L and an incomplete dump body map obtained by removing part of the height data from the dump body map as teaching data.
  • FIG. 10 is a flowchart showing a display method of the guidance information by the control device according to the second embodiment.
  • the data acquisition unit 1701 acquires the stereo image from the stereo camera 122 (step S 21 ).
  • the feature point specifying unit 1702 inputs the right-eye image of the stereo image acquired by the data acquisition unit 1701 to the feature point specifying model M 1 stored in the storage 93 to specify the positions of the plurality of feature points of the dump body 210 shown in the right-eye image (step S 22 ).
  • the three-dimensional data generation unit 1703 generates the point group data of the entire imaging range of the stereo camera 122 by the stereo measurement using the stereo image acquired in step S 21 and the camera parameters stored in the storage 93 (step S 23 ).
  • the dump body specifying unit 1704 specifies the three-dimensional positions of the feature points based on the positions of the feature points specified in step S 22 and the point group data generated in step S 23 (step S 24 ).
  • the dump body specifying unit 1704 fits the dump body model VD stored in the storage 93 to the specified positions of the feature points to specify the three-dimensional position of the bottom surface of the dump body 210 (step S 25 ).
  • the dump body specifying unit 1704 disposes, based on at least three specified positions of the feature points, the dump body model VD created based on the dimensions of the dump truck 200 to be detected in a virtual space.
  • the surface specifying unit 1705 extracts, from the point group data generated in step S 23 , a plurality of three-dimensional points in a prismatic area, which is surrounded by the front panel, the side gate, and the tailgate of the dump body 210 specified in step S 25 and extends in the height direction of the front panel, to remove three-dimensional points corresponding to the background from the point group data (step S 26 ).
  • the surface specifying unit 1705 removes three-dimensional points corresponding to the position of the dump body model VD from the plurality of three-dimensional points extracted in step S 6 to specify the three-dimensional position of the surface of the transport object L loaded on the dump body 210 (step S 27 ).
  • the distribution specifying unit 1706 generates the dump body map which is an elevation map with the bottom surface of the dump body 210 as a reference height, based on the plurality of three-dimensional points extracted in step S 27 and the bottom surface of the dump body 210 (step S 28 ).
  • the dump body map may include a grid without the height data.
  • the surface specifying unit 1705 specifies an area where the transport object L appears in the right-eye image based on the three-dimensional position of the surface of the transport object L specified in step S 27 (step S 29 ). For example, the surface specifying unit 1705 specifies a plurality of pixels in the right-eye image corresponding to the plurality of three-dimensional points extracted in step S 27 and determines an area composed of the plurality of specified pixels as the area where the transport object L appears.
  • the type specifying unit 1710 extracts the area where the transport object L appears from the right-eye image and inputs an image related to the area to the type specifying model M 3 to specify the type of the transport object L (step S 30 ).
  • the distribution estimation unit 1707 inputs the dump body map generated in step S 28 to the complementary model M 2 associated with the type specified in step S 30 to generate the dump body map in which the height data is complemented (step S 31 ).
  • the guidance information generation unit 1708 generates the guidance information based on the dump body map (step S 32 ).
  • the display control unit 1709 outputs the display signal for displaying the guidance information to the display device 125 (step S 33 ).
  • the control device 124 estimates the distribution of the amount of the transport object L in the shielded part based on the type of the transport object L. That is, characteristics (for example, the angle of repose) of the transport object L loaded on the dump body 210 differ depending on the type of the transport object L.
  • the third embodiment it is possible to more accurately estimate the distribution of the transport object L in the shielded part according to the type of the transport object L.
  • control device 124 is mounted on the hydraulic excavator 100
  • the present invention is not limited thereto.
  • the control device 124 according to another embodiment may be provided in a remote server device.
  • the control device 124 may be realized by a plurality of computers. In this case, part of the configuration of the control device 124 may be provided in the remote server device. That is, the control device 124 may be implemented as a transport object specifying system composed of a plurality of devices.
  • the drop target according to the above-described embodiment is the dump body 210 of the dump truck 200
  • the present invention is not limited thereto.
  • the drop target according to another embodiment may be another drop target such as a hopper.
  • the captured image according to the above-described embodiment is the stereo image
  • the present invention is not limited thereto.
  • the calculation may be performed based on one image instead of the stereo image.
  • the control device 124 can specify the three-dimensional position of the transport object L by using, for example, a trained model that generates depth information from the one image.
  • control device 124 complements the value of the shielded part of the dump body map by using the complementary model M 2
  • the present invention is not limited thereto.
  • the control device 124 according to another embodiment may estimate a height of the shielded part based on a rate of change or a pattern of change in the height of the transport object L near the shielded part. For example, in a case where the height of the transport object L near the shielded part becomes lower as it approaches the shielded part, the control device 124 may estimate the height of the transport object L in the shielded part to a value lower than the height near the shielded part based on the rate of change in the height.
  • the control device 124 may estimate the height of the transport object L in the shielded part by simulation in consideration of a physical property such as the angle of repose of the transport object L.
  • the control device 124 according to another embodiment may deterministically estimate the height of the transport object L in the shielded part based on cellular automaton in which each grid of the dump body map is regarded as a cell.
  • the control device 124 may not complement the dump body map and may display information related to the dump body map including a part where the height data is missing.
  • FIG. 11A is a diagram showing a first example of a method for calculating an amount of a transport object in a dump body.
  • FIG. 11B is a diagram showing a second example of the method for calculating the amount of the transport object in the dump body.
  • the dump body map according to the above-described embodiment is represented by a height from a bottom surface L 1 of the dump body 210 to an upper limit of the loading on the dump body 210 , but the present invention is not limited thereto.
  • the dump body map may represent a height from another reference plane L 3 with respect to the bottom surface to a surface L 2 of the transport object L.
  • the reference plane L 3 is a plane parallel to the ground surface and passing through a point of the bottom surface closest to the ground surface. In this case, the operator can easily recognize the amount of the transport object L until the dump body 210 is full, regardless of an inclination of the dump body 210 .
  • control device 124 may calculate the dump body map based on an opening surface of the dump body 210 , the surface of the transport object, and a height from the bottom surface to the opening surface of the dump body 210 . That is, the control device 124 may calculate the dump body map by subtracting, from the height from the bottom surface to the opening surface of the dump body 210 , a distance from an upper end surface of the dump body to the surface of the transport object L.
  • the dump body map according to another embodiment may be based on the opening surface of the dump body 210 .
  • the guidance information generation unit 1708 extracts the feature points from the right-eye image using the feature point specifying model M 1 , the present invention is not limited thereto.
  • the guidance information generation unit 1708 may extract the feature points from the left-eye image using the feature point specifying model M 1 .
  • the transport object specifying device can specify the distribution of the transport object in the drop target.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Civil Engineering (AREA)
  • Mining & Mineral Resources (AREA)
  • General Engineering & Computer Science (AREA)
  • Structural Engineering (AREA)
  • Mechanical Engineering (AREA)
  • Image Analysis (AREA)
  • Component Parts Of Construction Machinery (AREA)
  • Length Measuring Devices By Optical Means (AREA)
US17/260,069 2018-08-31 2019-07-19 Transport object specifying device of work machine, work machine, transport object specifying method of work machine, method for producing complementary model, and dataset for learning Pending US20210272315A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2018-163671 2018-08-31
JP2018163671A JP7311250B2 (ja) 2018-08-31 2018-08-31 作業機械の運搬物特定装置、作業機械、作業機械の運搬物特定方法、補完モデルの生産方法、および学習用データセット
PCT/JP2019/028454 WO2020044848A1 (fr) 2018-08-31 2019-07-19 Dispositif de spécification de cargaison supportée par une machine de construction, machine de construction, procédé de spécification de cargaison supportée par une machine de construction, procédé de production de modèle d'interpolation et ensemble de données pour apprentissage

Publications (1)

Publication Number Publication Date
US20210272315A1 true US20210272315A1 (en) 2021-09-02

Family

ID=69645231

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/260,069 Pending US20210272315A1 (en) 2018-08-31 2019-07-19 Transport object specifying device of work machine, work machine, transport object specifying method of work machine, method for producing complementary model, and dataset for learning

Country Status (5)

Country Link
US (1) US20210272315A1 (fr)
JP (1) JP7311250B2 (fr)
CN (1) CN112513563B (fr)
DE (1) DE112019003049T5 (fr)
WO (1) WO2020044848A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220195690A1 (en) * 2019-09-18 2022-06-23 Sumitomo Heavy Industries, Ltd. Shovel
US11414837B2 (en) * 2018-08-31 2022-08-16 Komatsu Ltd. Image processing system, display device, image processing method, method for generating trained model, and dataset for learning
US20220364873A1 (en) * 2021-05-12 2022-11-17 Deere & Company System and method for assisted positioning of transport vehicles for material discharge in a worksite

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7374142B2 (ja) * 2021-03-01 2023-11-06 日立建機株式会社 ホイールローダ
US11965308B2 (en) 2021-05-12 2024-04-23 Deere & Company System and method of truck loading assistance for work machines
JP2023088646A (ja) * 2021-12-15 2023-06-27 株式会社小松製作所 バケットに保持された掘削物の安息角を算出するための方法、バケットに保持された掘削物の安息角を算出するためのシステム、及び積込機械

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120114181A1 (en) * 2010-11-01 2012-05-10 Borthwick James R Vehicle pose estimation and load profiling
WO2018043301A1 (fr) * 2016-09-02 2018-03-08 株式会社小松製作所 Système d'affichage graphique pour engin de chantier

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH01156606A (ja) * 1987-12-15 1989-06-20 Matsushita Electric Works Ltd 光干渉式形状測定装置
JP3091648B2 (ja) * 1994-09-26 2000-09-25 沖電気工業株式会社 隠れマルコフモデルの学習方法
JPH11211438A (ja) * 1998-01-22 1999-08-06 Komatsu Ltd 荷台積載体積計測装置
JP3038474B2 (ja) * 1998-09-11 2000-05-08 五洋建設株式会社 土運船における積載土量の計測方法及び装置
JP2004061300A (ja) * 2002-07-29 2004-02-26 Asahi Shoji Ltd レーザー式角度検出装置、クランク軸のデフレクション計測装置、クランク軸のデフレクション計測方法及びクランク軸
JP2005220633A (ja) 2004-02-06 2005-08-18 Ohbayashi Corp ベルトコンベアの搬送土砂量検出装置およびその検出方法
JP5835881B2 (ja) * 2010-10-06 2015-12-24 株式会社東芝 医用画像処理装置、及び医用画像処理プログラム
CN104200657B (zh) * 2014-07-22 2018-04-10 杭州智诚惠通科技有限公司 一种基于视频和传感器的交通流量参数采集方法
US10208459B2 (en) * 2014-12-12 2019-02-19 Hitachi, Ltd. Volume estimation device and work machine using same
JP6567940B2 (ja) 2015-10-05 2019-08-28 株式会社小松製作所 施工管理システム
JP6674846B2 (ja) 2016-05-31 2020-04-01 株式会社小松製作所 形状計測システム、作業機械及び形状計測方法
CN106839977B (zh) * 2016-12-23 2019-05-07 西安科技大学 基于光栅投影双目成像技术的盾构渣土体积实时测量方法
CN106885531B (zh) * 2017-04-20 2018-12-18 河北科技大学 基于二维激光雷达的车斗形容器三维扫描系统标定方法
CN107168324B (zh) * 2017-06-08 2020-06-05 中国矿业大学 一种基于anfis模糊神经网络的机器人路径规划方法
CN108332682A (zh) * 2018-02-06 2018-07-27 黑龙江强粮安装饰工程有限公司 新型粮仓动态储藏容重监控系统及监控方法

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120114181A1 (en) * 2010-11-01 2012-05-10 Borthwick James R Vehicle pose estimation and load profiling
WO2018043301A1 (fr) * 2016-09-02 2018-03-08 株式会社小松製作所 Système d'affichage graphique pour engin de chantier

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11414837B2 (en) * 2018-08-31 2022-08-16 Komatsu Ltd. Image processing system, display device, image processing method, method for generating trained model, and dataset for learning
US20220195690A1 (en) * 2019-09-18 2022-06-23 Sumitomo Heavy Industries, Ltd. Shovel
US20220364873A1 (en) * 2021-05-12 2022-11-17 Deere & Company System and method for assisted positioning of transport vehicles for material discharge in a worksite
US11953337B2 (en) * 2021-05-12 2024-04-09 Deere & Company System and method for assisted positioning of transport vehicles for material discharge in a worksite

Also Published As

Publication number Publication date
CN112513563B (zh) 2023-01-13
JP7311250B2 (ja) 2023-07-19
JP2020034527A (ja) 2020-03-05
DE112019003049T5 (de) 2021-03-11
CN112513563A (zh) 2021-03-16
WO2020044848A1 (fr) 2020-03-05

Similar Documents

Publication Publication Date Title
US11414837B2 (en) Image processing system, display device, image processing method, method for generating trained model, and dataset for learning
US20210272315A1 (en) Transport object specifying device of work machine, work machine, transport object specifying method of work machine, method for producing complementary model, and dataset for learning
US12094151B2 (en) Image processing system, image processing method, learned model generation method, and data set for learning
US11417008B2 (en) Estimating a volume of contents in a container of a work vehicle
US11494930B2 (en) Techniques for volumetric estimation
CN110805093B (zh) 利用视觉传感器进行反馈回路控制的容器角度感测
Se et al. Vision based modeling and localization for planetary exploration rovers
US11591776B2 (en) Earth-moving machine sensing and control system
US8903689B2 (en) Autonomous loading
US9990543B2 (en) Vehicle exterior moving object detection system
KR20190120322A (ko) 작업 차량의 동작을 추정하기 위한 방법, 시스템, 학습된 분류 모델의 제조 방법, 학습 데이터, 및 학습 데이터의 제조 방법
Guevara et al. Point cloud-based estimation of effective payload volume for earthmoving loaders
JP2014228941A (ja) 地表面３次元サーフェス形状計測装置、走行可能領域検出装置およびそれを搭載した建設機械並びに走行可能領域検出方法
CN114170596A (zh) 姿态识别方法、装置、电子设备、工程机械和存储介质
Mandici et al. Aggregate road surface based environment representation using digital elevation maps
JP7175245B2 (ja) 作業機械
JP7501481B2 (ja) 距離推定装置、距離推定方法、および距離推定用コンピュータプログラム
CN118269993A (zh) 一种车辆控制方法、装置、设备、存储介质、产品及系统
CN116189141A (zh) 目标信息的确定方法和计算机设备

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED