JP2020178659A - Harvester - Google Patents

Abstract

To provide a harvester capable of detecting an obstacle accurately.SOLUTION: A harvester includes: a distance information generation part for generating three-dimensional point group data using stereo-matching from a captured image acquired by a stereo camera unit 81 for imaging a front in an advancing direction of a machine body in a farm field; a farm field reaping information creation part for creating farm field reaping information including a border between a reaped field and an unreaped field, and height of farm crops from the three-dimensional point group data; obstacle estimation means for estimating whether an obstacle is present in the reaped field or in the unreaped field; and an obstacle detection part for executing masking processing for excluding the three-dimensional point group data within a height region of the crops in the unreaped field from an obstacle detection object, and calculating positional relationships between the obstacle and the machine body based on the three-dimensional point group data and an estimation result by the obstacle estimation means.SELECTED DRAWING: Figure 3

Description

The present invention relates to a harvester that detects obstacles in a field using captured images acquired by a camera.

Since there is a possibility that workers other than the driver, animals, and obstacles such as utility poles exist in the field, the harvester that runs in the field is described in Patent Document 1, as disclosed in Patent Document 1. Some are equipped with an ultrasonic obstacle sensor for detecting obstacles. When an obstacle is detected by the obstacle sensor, the engine or the like is stopped to prevent danger.

The combine according to Patent Document 2 is provided with a laser scanner. This laser scanner is configured so that the laser is biased toward the right side of the cutting section in order to detect a person who has entered the area where most of the road surface is exposed due to the cutting of the object to be cut. ing. Further, Patent Document 2 describes that it is possible to use a millimeter-wave radar or image processing using a camera or the like instead of a laser scanner in order to detect an obstacle.

In the combine according to Patent Document 3, by inputting the photographed image sequentially acquired by the photographing unit provided in the airframe, the existence area where the recognition object shown in the photographed image exists is output together with the estimated probability. Will be done. When the recognition target is estimated with a high estimation probability, a warning is notified and running control is performed. Machine learning is used to estimate the recognition object, but in order to estimate the recognition object with high accuracy, the obstacle detection control unit is required to have an advanced learning function and a high calculation function.

Japanese Unexamined Patent Publication No. 9-76850 JP-A-2017-176007 Japanese Unexamined Patent Publication No. 2019-004773

In harvesting crops such as wheat, rice and soybeans, there is a high possibility that obstacles such as humans will be partially hidden by the crops, in which case the accuracy of obstacle detection will decrease. If the aircraft stops every time an obstacle is detected and the detection work of the detected obstacle is frequently performed, the workability is lowered. Under these circumstances, there is a demand for a harvester capable of detecting obstacles with high accuracy.

The harvester according to the present invention has a harvesting section for cutting agricultural products in the field, and uses a stereo camera unit that photographs the front of the aircraft in the traveling direction in the field and stereo matching from the captured image acquired by the stereo camera unit. A distance information generation unit that generates 3D point group data developed in a 3D coordinate system using a field scene as one reference plane, and a mowed land cut by the harvesting unit from the 3D point group data. A field mowing information creation unit that generates field mowing information including the boundary with the uncut land and the height of the agricultural product, and an obstacle that estimates whether the obstacle exists in the mowed land or the uncut land. The estimation means and the masking process for excluding the three-dimensional point group data in the height region of the agricultural product in the uncut land from the obstacle detection target are performed, and the three-dimensional point group data and the obstacle are performed. It is provided with an obstacle detection unit that detects the obstacle and calculates the positional relationship between the obstacle and the aircraft based on the estimation result by the estimation means.

According to this configuration, three-dimensional point group data is generated by performing stereo matching using the left and right captured images acquired by the pair of left and right cameras of the stereo camera unit. Since each point data constituting the 3D point cloud data indicates the position coordinates of each point in the captured image, the boundary between the mowed land and the uncut land (field mowing information) based on the distribution of the 3D point cloud data. It is possible to calculate the height of the crop (one of the above) and the height of the crop (the other one of the field cutting information). The mowed land is the field area where the crops are cut by the harvesting section, and the uncut land is the field area where the crops are still growing. From the difference between the point cloud indicating the cut land and the point cloud indicating the uncut land, the boundary between the cut land and the uncut land or the area between the cut land and the uncut land in the captured image is calculated. The height of the crop can be calculated from the point cloud indicating the crop in the uncut land. Furthermore, obstacles that enter the area of crops growing on the uncut land (the crop space required by the surface area of the uncut land and the height of the crop) are not shown in the photographed image, so that area is an obstacle. Treated as the target area of masking processing to be excluded from the object detection target. Furthermore, since there is an obstacle estimation means that can estimate whether the obstacle exists in the mowed land or the uncut land, the estimation result of this obstacle estimation means and the growing crops An obstacle is detected based on the three-dimensional point cloud data masking the area of the above, and the positional relationship between the obstacle and the aircraft is calculated with high reliability. In obstacle detection on uncut land, useless detection (noise) can be avoided by excluding the crop space from the obstacle detection target. Further, in the obstacle detection on the mowed land, substantially the entire area above the predetermined height from the field scene is the detection target, and various types of obstacles existing on the mowed land can be detected.

In one of the preferred embodiments of the present invention, the warning area setting unit that sets the obstacle warning area defined based on the position from the aircraft, and the obstacle detected by the obstacle detection unit are described. When it exists in the obstacle warning area, it is provided with a vehicle speed change command output unit that changes the vehicle speed according to the positional relationship. According to this configuration, when the detected obstacle is considered to exist in the obstacle warning area, the vehicle speed is changed according to the positional relationship between the obstacle and the aircraft. Therefore, the degree of change in the vehicle speed can be different depending on whether the possibility of interference within a predetermined time is large or the possibility of interference is small due to the positional relationship between the aircraft and the obstacle. At that time, for example, changing the vehicle speed when there is a high possibility of interference within a relatively short time due to the positional relationship between the aircraft and the obstacle will change the vehicle speed to zero, that is, stop the aircraft. .. As a result, even when the possibility of interference within a relatively short time is small due to the positional relationship between the machine and the obstacle, it is possible to reduce the possibility of stopping the machine and reducing workability. Furthermore, since the obstacle warning area is defined based on the position with respect to the aircraft, the possibility of interference between the aircraft and obstacles can be almost ignored even in the area within the shooting field of view of the camera. Areas where it can be removed can be removed from the obstacle warning area. This improves the efficiency of obstacle detection.

An obstacle estimation means capable of estimating whether an obstacle exists in a mowed land or an uncut land can be realized by a machine learning unit that inputs a captured image. In the detection of obstacles in the field, deep learning with high learning ability is preferable in machine learning. From this, in one of the preferred embodiments of the present invention, the obstacle estimation means is a deep learning unit, and the deep learning unit is acquired by any one camera constituting the stereo camera unit. The obstacle area where the obstacle is found is estimated using the captured image. In this configuration, by inputting a captured image, an obstacle frame (bounding box) showing the outline of an obstacle (an object of interest such as a person or an animal) is output. By applying this obstacle frame to the three-dimensional point cloud data, the three-dimensional position of the obstacle indicated by the obstacle frame, and as a result, the positional relationship between the obstacle and the aircraft can be obtained. By combining the 3D point cloud data by stereo matching and the well-learned deep learning unit, it is possible to detect the face of a person protruding from the upper end of the crop before cutting (growing crop). Has been confirmed experimentally.

It is an overall side view of a combine. It is a schematic diagram which shows the obstacle detection range by the stereo camera equipped in the front part of the fuselage. It is explanatory drawing explaining the flow of obstacle detection schematically. It is a top view which shows the 1st warning area and the 2nd warning area used for obstacle detection by a stereo camera. It is a functional block diagram of a control system in a combine. It is a flowchart which shows the flow of obstacle detection control.

Hereinafter, an embodiment of the combine as an example of the harvester according to the present invention will be described with reference to the drawings. The crops to be harvested are planted culms such as wheat and rice. In this embodiment, when the front-rear direction of the machine body 1 is defined, it is defined along the machine body traveling direction in the working state. The direction indicated by reference numeral (F) in FIG. 1 is the front side of the aircraft, and the direction indicated by reference numeral (B) in FIG. 1 is the rear side of the aircraft. When defining the left-right direction of the aircraft 1, the left-right direction is defined as viewed in the direction of travel of the aircraft. “Upper side (upper side)” or “lower side (lower side)” is a positional relationship of the aircraft 1 in the vertical direction (vertical direction), and indicates a relationship at a height above the ground.

As shown in FIG. 1, in the combine, a harvesting portion 2 which is a harvesting portion is connected to the front portion of the machine body 1 provided with a pair of left and right crawler traveling devices 10 so as to be movable up and down around the horizontal axis. Due to the speed difference between the left and right crawler traveling devices 10, the machine body 1 can turn left and right. The rear part of the machine body 1 is provided with a threshing device 11 and a grain tank 12 for storing grains in a state of being lined up in the width direction of the machine body 1. A boarding operation unit 14 is provided on the right side of the front portion of the aircraft 1, and an engine (not shown) is provided below the boarding operation unit 14.

As shown in FIG. 1, the threshing device 11 receives the harvested culm cut by the cutting unit 2 and transported to the rear inside, and sandwiches the stock of the culm by the feed chain 111 and the holding rail 112. The tip side is threshed by the culm 113 while being transported. Then, a grain sorting process for the threshed product is executed in a sorting section provided below the handling cylinder 113, and the grains sorted there are transported to the grain tank 12 and stored. Further, although not described in detail, a grain discharge device 13 for discharging grains stored in the grain tank 12 to the outside is provided.

The cutting unit 2 is provided with a clipper-type cutting device 22 for cutting the root of the raised planted culm, a culm transporting device 23, and the like. The grain culm transport device 23 transports the harvested culm in the vertical position in which the stock root has been cut toward the start end of the feed chain 111 while gradually changing the cut culm to the sideways position.

A satellite positioning module 80 is also provided on the ceiling of the boarding operation unit 14. The satellite positioning module 80 includes a satellite antenna for receiving GNSS (global navigation satellite system) signals (including GPS signals). In order to complement the satellite navigation by the satellite positioning module 80, an inertial navigation unit including a gyro acceleration sensor and a magnetic compass sensor may be incorporated in the satellite positioning module 80. Of course, the inertial navigation system can be placed elsewhere.

As shown in FIG. 2, the front part of the combine is provided with a stereo camera unit 81, which is a stereo camera having a pair of left and right cameras for photographing the front of the aircraft 1 in the traveling direction in the field for obstacle detection. There is. Further, in this embodiment, the captured image acquired by one of the cameras of the stereo camera unit 81 is used for a machine learning algorithm such as deep learning, and an obstacle such as a person in the captured image is detected. Will be done.

An example of obstacle detection will be described below with reference to the schematic control flow diagram shown in FIG. In the obstacle detection using the stereo camera unit 81, first, the captured images are acquired by the left and right cameras of the stereo camera unit 81 (#a). Next, stereo matching processing is performed on the acquired images taken by the left and right cameras (# b). Parallax data is generated through the stereo matching process, and a distance image is further generated (#c). Further, the three-dimensional point cloud data developed in the three-dimensional coordinate system with the field scene as one reference plane (bottom surface) is generated from the distance image (#d).

Further, from the three-dimensional point cloud data, field cutting information including the boundary between the cut land and the uncut land cut by the cutting unit 2 and the height of the planted culm is generated. An example of the algorithm for creating the field cutting information will be described below.

Based on the 3D point cloud data, the 3D space (space to be processed) in front of the traveling direction is modeled numerically, and the uncut land is estimated using the digital elevation model. In addition, the height of planted culms growing on this estimated uncut land is also estimated. The processing target space is, for example, about 5 m in width, 5 m in depth, and 3 m in height. The bottom surface of the processing target space is divided into unit sections (for example, 2 cm x 2 cm to 5 cm x 5 cm) in a grid pattern, and a rectangular parallelepiped having this unit section group as the bottom surface is further divided in the height direction to form cells. It becomes a structure. A point cloud indicated by three-dimensional point cloud data is assigned to each cell of this cell structure. By assigning the representative height (for example, maximum height or average height) of the point cloud in the cell belonging to the rectangular parallelepiped with the unit section as the bottom to the unit section, the unevenness of the field scene (height of the planted culm) An uneven image expressing (including the height of an obstacle) is obtained. The uneven image is binarized, expanded / contracted, and labeled, and finally the areas of uncut land and uncut land are estimated (#e). In FIG. 3, the uncut area is shown by a diagonal line, the cut area is shown by a cross line, and the boundary between the uncut area and the cut area is shown by a thick solid line. Assuming that the combine is in a straight line, the boundary is indicated by a straight line. The average of the representative heights of the point clouds in the uncut area is the height of the planted culm (for example, 1 m). As a result, the space to be processed in front of the traveling direction is the space where the planted culm of the uncut land exists, the space above the culm where the planted culm of the uncut land does not exist, and the free of the culm. It can be divided into space (#f).

Obstacles existing in the culm existence space are undetectable because they are shielded by the planted culm. Therefore, the estimated obstacle in the culm existence space is regarded as noise. Therefore, the three-dimensional point cloud data in the culm existence space, which is the height region of the planted culm in the uncut land, is excluded from the obstacle detection target by the masking process. Prior to the obstacle detection process, this masking process can be considered as a method of removing the three-dimensional point cloud data in the culm existence space from the processing target or a method of removing it by threshold comparison. , In the present application, it is treated as a masking process. The maximum height of obstacle detection in the space above the culm and the free space of the mowed land is about 2.5 m.

Obstacle detection (object detection) using the three-dimensional point cloud data that is the target of obstacle detection is performed by the point density obtained through filtering, binarization, expansion / contraction, and labeling. A group of cells having a point density higher than the threshold value for obstacle detection is regarded as an obstacle. At that time, since the point density in the distant region is coarser than the point density in the vicinity, the threshold value for obstacle detection is a function of the distance from the stereo camera unit 81 to each section, for example, the square of the distance. Is given in inverse proportion to.

In this embodiment, an obstacle estimation means for estimating whether an obstacle exists in a mowed land or an uncut land is used. This obstacle estimation means is realized by an algorithm using deep learning, and an obstacle frame (obstacle area) in which an obstacle is found is output (#g). That is, the size and position of the obstacle frame in which the obstacle is found in the captured image acquired by any one of the cameras constituting the stereo camera unit 81 is output by the deep learning algorithm. In this embodiment, YOLO is used as a deep learning algorithm for reasons such as cost effectiveness.

In this obstacle detection process, obstacle detection by stereo matching and obstacle frame by a deep learning algorithm are combined (#h). In obstacle detection by stereo matching, if the threshold value for obstacle detection is raised too much, the actual obstacle will not be detected, and if the threshold value for obstacle detection is lowered too much, the obstacle will be erroneously detected. Such a problem is solved by giving the area where the obstacle exists to the obstacle detection algorithm by stereo matching together with the reliability rate by the output of the obstacle frame showing the obstacle shown in the captured image by the deep learning algorithm. , Reduce. For example, it is experimental to correctly detect the face of a person protruding from the upper end of the planted culm as an obstacle, and to correctly detect a person crouching on the mowed land near the planted culm as an obstacle. Was possible.

When the detected obstacle may interfere with the aircraft 1, control of avoiding interference with the obstacle is performed. As shown in FIG. 4, the obstacle warning area used for this purpose is defined on the reference plane (XY plane) corresponding to the field scene extending forward in the traveling direction of the aircraft 1. Here, the obstacle warning area is the first warning area (darkly filled in FIG. 4 and given the code Z1) and the second warning area (lightly filled in FIG. 4) having different positional relationships with the aircraft 1. The symbol Z2 is assigned).

The first warning area is a stereo camera unit from a rectangle defined by the working width of the cutting unit 2 and the warning distance (indicated by reference numeral L1 in FIG. 4) from the machine 1 to the front in the traveling direction of the machine 1. It is a tip wedge-shaped region obtained by cutting out a blind spot region determined by the angle of view of 81. In other words, the first warning area is from a rectangle whose left and right sides of the harvesting section 2, which is substantially the width of the combine's traveling locus, is a predetermined distance (warning distance) forward from the aircraft 1 in the traveling direction. , The shape is obtained by cutting out a blind spot region determined by the angle of view of the stereo camera unit 81. Obstacles existing in this first alert area come into contact with the combine as the combine progresses. Therefore, the warning distance is preferably changed according to the vehicle speed of the combine, but in this embodiment, since the working running speed is substantially constant, it is set to several meters.

The second warning area is a shape obtained by cutting out the first warning area from the tip wedge-shaped area in which the blind spot area determined by the angle of view of the stereo camera unit 81 is cut out in the plane extending forward in the traveling direction of the aircraft 1. It has a width of about three times that of the first warning area in the left-right direction of the combine, and a little less than twice the length of the first warning area in front of the traveling direction of the aircraft 1 (indicated by reference numeral L2 in FIG. 4). ). The left-right spread of the second warning area is set in consideration of the turning of the combine, and the spread of the traveling direction of the second warning area has a margin in time until the aircraft 1 interferes with the obstacle, but the first warning area It is set to be larger than the area.

FIG. 5 shows a functional block diagram of the combine control system. The control system of this embodiment is composed of a large number of electronic control units called ECUs, various operating devices, sensor groups and switch groups, and a wiring network such as an in-vehicle LAN that transmits data between them. The notification device 84 is a device for notifying the driver or the like of a warning such as an obstacle detection result or a working running state, and is a buzzer, a lamp, a speaker, a display, or the like. Regarding the detection result of the obstacle, when the obstacle exists in the first warning area, a warning sound, a warning light, or a warning message indicating an imminent state is notified as an emergency warning notification, and the first warning is given. If it exists in the area, a gentle caution sound, a caution light, or a caution message is notified as a caution notification rather than an emergency warning notification.

The communication unit 85 is used for exchanging data with an external communication device. The control unit 6 is a core element of this control system and is shown as an aggregate of a plurality of ECUs. The positioning data from the satellite positioning module 80 and the captured image from the stereo camera unit 81 are input to the control unit 6 through the wiring network.

The control unit 6 includes an output processing unit 6B and an input processing unit 6A as input / output interfaces. The output processing unit 6B is connected to the vehicle traveling equipment group 7A and the working equipment group 7B. The vehicle traveling device group 7A includes control devices related to vehicle traveling, such as an engine control device, a shift control device, a braking control device, and a steering control device. The working equipment group 7B includes a cutting unit 2, a threshing device 11, a grain discharging device 13, a power control device in the grain transporting device 23, and the like.

A traveling system detection sensor group 8A, a working system detection sensor group 8B, and the like are connected to the input processing unit 6A. The traveling system detection sensor group 8A includes sensors that detect the state of the engine speed adjuster, the accelerator pedal, the brake pedal, the speed change operation tool, and the like. The work system detection sensor group 8B includes a sensor for detecting the device state in the cutting unit 2, the grain removal device 11, the grain discharge device 13, and the grain transfer device 23, and the state of the grain and the grain.

The control unit 6 is provided with a work travel control module 60, an obstacle detection unit 50, a vehicle speed change command output unit 65, an aircraft position calculation unit 66, and a notification unit 67 as control function units. It should be noted that these control function units are individually configured and connected by an in-vehicle LAN or the like.

The notification unit 67 generates notification data based on a request from each functional unit of the control unit 6 and gives the notification data to the notification device 84. The aircraft position calculation unit 66 calculates the aircraft position, which is the map coordinates (or field coordinates) of the aircraft 1, based on the positioning data sequentially sent from the satellite positioning module 80.

The combine of this embodiment can travel in both automatic driving (automatic steering) and manual driving (manual steering). The work travel control module 60 includes a work travel command unit 63 and a travel route setting unit 64 in addition to the travel control unit 61 and the work control unit 62. A traveling mode switch (not shown) for selecting between an automatic traveling mode for traveling by automatic steering and a manual steering mode for traveling by manual steering is provided in the boarding operation unit 14. By operating this travel mode switch, it is possible to shift from manual steering running to automatic steering running, or from automatic steering running to manual steering running.

The travel control unit 61 has an engine control function, a steering control function, a vehicle speed control function, and the like, and gives a travel control signal to the vehicle travel equipment group 7A. The work control unit 62 gives a work control signal to the work equipment group 7B in order to control the movements of the cutting unit 2, the threshing device 11, the grain discharge device 13, the grain culm transport device 23, and the like.

The travel route setting unit 64 develops a travel route for automatic travel in the memory. The travel route developed in the memory is used as a target travel route in sequential automatic driving. This travel route can be used for guidance for the combine to travel along the travel route even in manual travel.

The work travel command unit 63 generates an automatic steering command and a vehicle speed command as automatic travel commands and gives them to the travel control unit 61. The automatic steering command is generated so as to eliminate the directional deviation and the positional deviation between the traveling route and the own aircraft position calculated by the aircraft position calculation unit 66 by the traveling route setting unit 64. During automatic driving, the vehicle speed command is generated based on the vehicle speed value set in advance. During manual driving, the vehicle speed command is generated based on the manual vehicle speed operation. However, in the event of an emergency such as obstacle detection, the vehicle speed will be changed, including a forced stop, and the engine will be stopped automatically.

When the automatic driving mode is selected, the traveling control unit 61 controls the vehicle traveling equipment group 7A related to steering and the vehicle traveling equipment group 7A related to vehicle speed based on the automatic traveling command given by the work traveling command unit 63. When the manual driving mode is selected, the traveling control unit 61 generates a control signal based on the operation by the driver to control the vehicle traveling equipment group 7A.

In the obstacle detection unit 50, as described with reference to FIG. 3, the captured image acquired by the stereo camera unit 81 is used, and the obstacle detection by stereo matching and the obstacle detection by the deep learning algorithm are simultaneously performed. Therefore, the obstacle detection unit 50 includes an image acquisition unit 51, a distance information generation unit 52, a field cutting information creation unit 53, a deep learning unit 54, an obstacle detection unit 55, and a warning area setting unit 56. And are included. The image acquisition unit 51 gives the left and right captured images sent from the stereo camera unit 81 to the distance information generation unit 52, and sends one of the left and right captured images, for example, the right captured image, to the deep learning unit 54. .. The distance information generation unit 52 uses stereo matching from the left and right captured images to generate three-dimensional point cloud data developed in a three-dimensional coordinate system with the field scene as one reference plane. The field mowing information creation unit 53 generates field mowing information including the boundary between the mowed land and the uncut land mowed by the mowing unit 2 and the height of the planted culm from the three-dimensional point cloud data. The deep learning unit 54 has a function of estimating whether an obstacle exists in a mowed land or an uncut land. The trained deep learning unit 54 estimates the presence of an obstacle in the input captured image, and outputs the size and position of the obstacle frame showing the outline of the obstacle as an estimation result.

The obstacle detection unit 55 detects an obstacle based on the three-dimensional point cloud data, the field cutting information, the size of the obstacle frame and its position, and the distance between the obstacle and the aircraft 1 (obstacle). Distance) is calculated. The warning area setting unit 56 sets an obstacle warning area defined based on the position with respect to the aircraft 1. In this embodiment, the obstacle warning area is divided into a first warning area and a second warning area as described with reference to FIG. The obstacle detection unit 55 determines whether the detected obstacle is in the first warning area or the second warning area, and generates obstacle detection information. The components constituting the obstacle detection unit 50 shown in FIG. 5 are mainly separated for the purpose of explanation, and the components may be integrated or the components may be freely divided. ..

The obstacle detection information generated by the obstacle detection unit 50 is sent to the vehicle speed change command output unit 65. From the received obstacle detection information, the vehicle speed change command output unit 65 outputs the first vehicle speed change command including the vehicle speed change value to the work travel command unit 63 when the obstacle exists in the first warning area. When the obstacle exists in the second warning area, the second vehicle speed change command including the vehicle speed change value is output to the work travel command unit 63. The first vehicle speed change command is a command to reduce the vehicle speed to zero, that is, a command to stop the aircraft 1. Of course, the first vehicle speed change command may be a command for setting the vehicle speed to a value close to zero, or a command for gradually reducing the vehicle speed to zero. The second vehicle speed change command is a deceleration value for decelerating the vehicle speed to a very small speed, for example, about 1 km / h. As the second vehicle speed change command, a deceleration value determined by a ratio to the immediately preceding vehicle speed, such as decelerating to a fraction of the immediately preceding vehicle speed, may be used. In any case, the first vehicle speed change value by the first vehicle speed change command is larger than the second vehicle speed change value by the second vehicle speed change command, and is a command for emergency evacuation. Based on the vehicle speed change command determined by the vehicle speed change command output unit 65, the work travel command unit 63 gives a control command to the travel control unit 61 to urgently decelerate the aircraft 1. If the presence of the detected obstacle shifts from the first warning area to the second warning area, the stopped aircraft 1 moves forward at a slow speed specified by the second vehicle speed change command. Furthermore, if the obstacle is not detected in the caution area, the vehicle returns to normal working vehicle speed.

Next, FIG. 6 shows a flowchart showing an example of the control flow of obstacle detection and vehicle speed deceleration in the above-mentioned control system. When this control starts, initial settings such as resetting flags and variables are performed (# 01). The alert area is set, that is, the first alert area and the second alert area are set (# 02). When the first warning area is changed by the vehicle speed, the current vehicle speed is acquired and the first warning area is set based on the vehicle speed.

The captured image is captured from the stereo camera unit 81 (# 03). As shown in FIG. 3, the obstacle detection unit 50 performs the obstacle detection process using the captured image, and when the obstacle is detected, the obstacle detection information is output (# 04). ..

It is determined whether or not an obstacle is detected, and if an obstacle is detected (# 05Yes branch), the position of the obstacle in the obstacle warning area, that is, the two-dimensional position of the obstacle is calculated. (# 11).

It is checked if the obstacle is in the first alert area (# 12). If the obstacle is in the first warning area (# 12Yes branch), the vehicle speed change command output unit 65 gives a stop command to the work travel command unit 63 as the first vehicle speed change command (# 13). Furthermore, the stop flag is turned ON (# 14). In step # 14, when the deceleration flag is ON (when an obstacle moves from the second warning area to the first boundary area), the deceleration flag is turned OFF and the stop flag is turned ON. As a result, the aircraft 1 is stopped, and the control returns to step # 02.

If the obstacle is not in the first alert area (# 12No branch) in the check of step # 12, it is further checked whether or not it is in the second alert area (# 20). If the obstacle is in the second warning area (# 20Yes branch), the vehicle speed change command output unit 65 gives a deceleration command to the work travel command unit 63 as the second vehicle speed change command (# 21). Furthermore, the deceleration flag is turned ON (# 22). In step # 22, when the stop flag is ON (when an obstacle moves from the first warning area to the second boundary area), the stop flag is turned OFF and the deceleration flag is turned ON. As a result, the aircraft 1 travels at a very low speed (about 1 km / h), and the control returns to step # 02.

If the obstacle is not in the second warning area in the check of step # 20 (# 20 No branch), there is no obstacle in the warning area. In this way, when there is no obstacle in the warning area, or when no obstacle is detected in the first place (# 08No branch), the state of the stop flag and the state of the deceleration flag are checked, and the state corresponds to that state. The flag contents are rewritten. Specifically, at that time, if the stop flag is ON (# 30Yes branch), the stop flag is rewritten to OFF (# 31), the travel command becomes valid, and the aircraft 1 normally travels (# 30Yes branch). # 32). Control then returns to step # 02. At that time, if the stop flag is not ON (# 30No branch), the content of the deceleration flag is checked (# 40). If the deceleration flag is ON (# 40Yes branch), the deceleration flag is rewritten to OFF (# 41), the travel command becomes valid, and the aircraft 1 normally travels (# 42). The normal running is a work running when an obstacle or the like is not detected, and the vehicle speed is faster than the vehicle speed when the obstacle is in the second warning area. Control then returns to step # 02. If the deceleration flag is OFF (# 40 No branch), of course, the travel command is valid, and the aircraft 1 normally travels (# 43), and the control returns to step # 02 as it is.

By controlling such obstacle detection and vehicle speed change, when a field worker enters the warning area in front of the machine 1 of the harvester in the traveling direction, the harvester stops or decelerates, and the field worker leaves the warning area. Then, the harvester returns to working at normal vehicle speed again.

The vehicle speed deceleration control based on this obstacle detection can be executed regardless of whether the harvester is automatically running or manually running.

In the above-described embodiment, the first warning area and the second warning area are set, and when an obstacle exists in each warning area, the aircraft 1 is stopped or the vehicle speed is changed to a speed slower than the normal running. .. Instead of this, three or more warning areas may be provided, and the aircraft 1 may be stopped or changed to a slow speed according to each area. Further, the warning area may be set steplessly, and the aircraft 1 may be stopped or the vehicle speed may be changed steplessly to a vehicle speed slower than the normal traveling according to the distance between the aircraft 1 and the obstacle.

The configuration disclosed in the above embodiment (including another embodiment, the same shall apply hereinafter) can be applied in combination with the configuration disclosed in other embodiments as long as there is no contradiction. The embodiments disclosed in the present specification are examples, and the embodiments of the present invention are not limited thereto, and can be appropriately modified without departing from the object of the present invention.

The present invention is applied to a harvester that runs a work in a field.

1: Aircraft 2: Harvesting section (harvesting section)
50: Obstacle detection unit 51: Image acquisition unit 52: Distance information generation unit 53: Field mowing information creation unit 54: Deep learning unit (obstacle estimation means)
55: Obstacle detection unit 56: Warning area setting unit 65: Vehicle speed change command output unit 81: Stereo camera unit (stereo camera)

Claims (3)

A harvester that has a harvesting section that cuts crops in the field.
A stereo camera unit that captures the front of the aircraft in the field,
A distance information generator that uses stereo matching from the captured image acquired by the stereo camera unit to generate 3D point cloud data developed in a 3D coordinate system with a field scene as one reference plane.
From the three-dimensional point cloud data, a field cutting information creation unit that generates field cutting information including the boundary between the cut land and the uncut land cut by the harvesting part and the height of the crop,
An obstacle estimation means for estimating whether an obstacle exists in the mowed land or the uncut land, and
Masking processing is performed to exclude the three-dimensional point cloud data in the height region of the agricultural product in the uncut land from the obstacle detection target, and the three-dimensional point cloud data and the estimation by the obstacle estimation means are performed. An obstacle detection unit that detects the obstacle and calculates the positional relationship between the obstacle and the aircraft based on the result.
Harvesting machine equipped with. A warning area setting unit that sets an obstacle warning area defined based on the position from the aircraft,
The harvest according to claim 1, wherein when the obstacle detected by the obstacle detection unit exists in the obstacle warning area, a vehicle speed change command output unit for changing the vehicle speed according to the positional relationship is provided. Machine. The obstacle estimation means is a deep learning unit, and the deep learning unit estimates an obstacle region in which an obstacle is found by using a captured image acquired by any one of the cameras constituting the stereo camera unit. The harvester according to claim 1 or 2.