JP2021058099A - Agricultural implement - Google Patents

Abstract

To provide an agricultural implement in which an object detection technology is installed for accurately and quickly detecting an object in a farm field together with a distance from a machine body.SOLUTION: An agricultural implement includes an imaging unit 9A for imaging a specified area of a farm field, a neural network 4 which inputs an imaged image from the imaging unit 9A, recognizes a specified object imaged in the imaged image, and outputs object area information indicating an area occupied by the specified object in the imaged image, a space position measuring unit 52 which images the specified area and generates object position information indicating space positions of all the objects present in the specified area, a distance calculation unit 53 which determines the space position of the specified object on the basis of the object position information and the object area information, and calculates a specified object distance being a distance to the specified object, and an object detection signal generation unit which outputs an object detection signal on the basis of the specified object distance.SELECTED DRAWING: Figure 4

Description

The present invention relates to an agricultural work machine that runs while detecting an object existing in a field.

The agricultural work machine according to Patent Document 1 is provided with an ultrasonic obstacle sensor for detecting an obstacle existing in the field, and when an obstacle is detected by the obstacle sensor, in order to prevent danger. The engine etc. is stopped.

The combine according to Patent Document 2 is provided with a laser scanner. This laser scanner is configured so that the laser is biased to the right part of the cutting part in order to detect a person who has entered the area where most of the field scene is exposed due to the cutting object being cut. Has been done. Further, Patent Document 2 describes that it is possible to use a millimeter-wave radar instead of a laser scanner, or to use image processing using a camera or the like in order to detect an obstacle.

In the combine according to Patent Document 3, by inputting the photographed images sequentially acquired by the photographing unit provided in the machine body, the existence area where the recognition object in the photographed image exists is output together with the estimated probability. To. When the recognition target is estimated with a high estimation probability, warning notification and running control are performed. Machine learning is used to estimate this perceptible object, but the detailed structure of machine learning is not disclosed.

Japanese Unexamined Patent Publication No. 9-76850 JP-A-2017-176007 Japanese Unexamined Patent Publication No. 2019-004773

When working with a farming machine in a field, objects such as humans and animals may exist as running obstacles. When this object is near the aircraft in the direction of travel of the aircraft, it is necessary to avoid the object from interfering with the aircraft. For this purpose, it is necessary to detect an object that becomes a traveling obstacle with high accuracy and to calculate the distance to the aircraft with high accuracy. However, it has been difficult to detect an object in a field quickly and with high accuracy by the object detection technique mounted on a conventional agricultural work machine. From such a situation, there is a demand for an agricultural work machine equipped with an object detection technology that detects an object in a field quickly and with high accuracy together with a distance from the machine body.

The agricultural work machine according to the present invention works on a field, inputs a photographing unit for photographing a specific area of the field, and an image photographed from the photographing unit, recognizes a specific object reflected in the photographed image, and recognizes the specific object. A neural network that outputs object area information indicating the area occupied by the specific object in the captured image, and a spatial position that photographs the specific area and generates object position information indicating the spatial positions of all the objects existing in the specific area. The measurement unit, the distance calculation unit that determines the spatial position of the specific object based on the object position information and the object area information, and calculates the specific object distance that is the distance from the machine body to the specific object, and the specific object. It includes an object detection signal generation unit that outputs an object detection signal based on the object distance.

According to this configuration, the neural network recognizes a specific object existing in a specific area of the field by using the image taken from the photographing unit, and outputs the object area information indicating the area occupied in the photographed image of the specific object. The spatial position measuring unit generates object position information indicating the spatial positions of all the objects existing in the specific area of the photographed field. The spatial position of the area occupied by the specific object detected by the neural network is calculated by associating the object area information with the object position information, and further, the distance from the aircraft to the specific object is calculated. For example, when the specific object shown in the object area information is a traveling obstacle such as a human being, the distance from the aircraft to the traveling obstacle is calculated by associating the object area information with the object position information. It is possible to control the aircraft according to the distance. In the present invention, a neural network having excellent detection ability of a specific object even if it is not possible to calculate an accurate spatial position of a specific object, and an accurate space of some object (unknown object) even if it does not have the detection ability of a specific object. By combining with a spatial position measuring unit having excellent position calculation ability, an agricultural work machine capable of quickly and highly accurately detecting a specific object in a field together with a distance from the machine is provided.

In one of the preferred embodiments of the present invention, the neural network is used to recognize a specific object in the captured image and output object region information indicating a region of the specific object in the captured image from the captured image. It is composed of a segmentation neural network (semantic segmentation) composed of an encoder network and a decoder network, and the object area information includes a contour of the specific object. In this configuration, when a photographed image is input, the objects included in the photographed image are separated into at least one, and the identified images so that the separation can be understood, for example, different colors depending on the type of the object. The expressed image (label image) is output. Therefore, it is possible to extract data showing the outline of a specific object in the captured image from the object area information.

If the segmentation neural network is implemented as it is, the network structure becomes complicated, so that the calculation processing load increases and the processing speed decreases. In order to speed up the network, it is preferable to reduce the number of hidden layer units (also called perceptrons) constituting the network by pruning. Therefore, in one of the preferred embodiments of the present invention, the encoder network and the hidden layer constituting the decoder network include a convolutional layer, a pooling layer, and a batch normalization layer, and the batch is described. Pruning is performed based on the parameters in the normalization layer. In this configuration, the unit judged to have a small contribution rate for good output based on the parameters obtained by the batch normalization operation in the batch normalization layer is not used. This simplifies the network structure and increases the processing speed of the neural network.

Since the distance to the specific object is used for aircraft control to avoid interference with the specific object, it is necessary to calculate it quickly and accurately. In addition, it should be avoided as much as possible that the measured value, which is an error value due to disturbance or the like, is used for calculating the distance to a specific object. From such a viewpoint, in one of the preferred embodiments of the present invention, the specific object distance is the distance from the center of gravity of the specific object to the machine body calculated from the contour of the specific object, or within the contour. It is a statistical representative value of the distance measurement point included in. The position of the center of gravity is calculated from the coordinates (spatial position) of a large number of distance measurement points showing contours. Statistical representative values are intermediate, median, and average values, which are also calculated based on a large number of distance measurement points. In this way, by using the center of gravity of the specific object and the statistical representative value in the calculation of the distance to the specific object, it is possible to prevent the error value from occupying a large proportion in the calculation of the distance to the specific object.

In one of the preferred embodiments of the present invention, the spatial position measuring unit generates the object position information by using a stereo matching technique for the specific area. In this configuration, three-dimensional point cloud data is generated by stereo matching using a photographed image of a specific area of the field. Since each point in the point cloud data has a three-dimensional position, that is, a spatial position, object position information is generated based on the three-dimensional point cloud data. As the captured image used for stereo matching, the captured image used for the neural network can be diverted, but in order to obtain a captured image having a resolution suitable for stereo matching, an image captured by a dedicated camera is preferably used. It should be used. Of course, in that case, a stereo matching photographed image having a photographed field of view equal to or wider than the photographed field of view of the photographed image used for the neural network is used.

Among the various agricultural work machines, the fields where the harvesters that harvest the crops run include the uncut land where the crops that have grown high from the field scene exist, the uncut land left after cutting the crops, and the surrounding shores. Exists. Agricultural products such as rice and wheat are in a prone position in some areas, and their height varies from place to place. In addition, the areas of mowed and uncut land fluctuate as the mowing work progresses. In detecting specific objects such as humans and animals, it is important to distinguish between mowed land and uncut land, and to detect the height of crops growing on uncut land. From this, in one of the preferred embodiments of the present invention in the case where the agricultural work machine is a harvester having a harvesting part for cutting agricultural products, it was cut by the harvesting part based on the object position information. A field information creation unit for creating field harvesting information including the boundary between the cut land and the uncut land and the height of the crop is provided. By using this field mowing information, it is possible to consider a specific object detected in the area submerged in the crop or a specific object detected in the area above the field scene that is already mowed as a false detection. And can mask those areas. Further, since the object detected in the area far from the shore does not become a traveling obstacle, it is possible to mask such an area as well. This makes it possible to quickly and accurately detect traveling obstacles such as humans and animals and prevent the aircraft from interfering with them.

In one of the preferred embodiments of the present invention, there is a warning area setting unit that sets an obstacle warning area defined based on the position from the aircraft, and a vehicle speed change command output unit (object detection signal generation) that changes the vehicle speed. When the specific object is a traveling obstacle and the traveling obstacle exists in the obstacle warning area, the vehicle speed change command output unit is based on the specific object distance. And change the vehicle speed. According to this configuration, when the detected specific object, that is, the traveling obstacle is considered to exist in the obstacle warning area, the vehicle speed is changed according to the positional relationship between the traveling obstacle and the aircraft. Therefore, the degree of change in the vehicle speed can be different depending on whether the possibility of interference within a predetermined time is large or the possibility of interference is small due to the positional relationship between the aircraft and the traveling obstacle. At that time, for example, changing the vehicle speed when there is a high possibility of interference within a relatively short time due to the positional relationship between the aircraft and the traveling obstacle is to change the vehicle speed to zero, that is, to stop the aircraft. Become. As a result, even when the possibility of interference within a relatively short time is small due to the positional relationship between the machine body and the traveling obstacle, it is possible to reduce the possibility of stopping the machine body and reducing workability.

It is an overall side view of a combine. It is a schematic diagram which shows the specific object detection range by the camera equipped in the front part of the fuselage. It is a schematic diagram which shows the structure of a segmentation neural network. It is a schematic flow chart of the process of detecting a specific object and finding the spatial position of the specific object. It is a flow chart which shows another form of the flow chart according to FIG. It is a top view which shows the 1st warning area and the 2nd warning area used for the warning determination of the detected running obstacle. It is a functional block diagram of a control system in a combine. It is a flowchart which shows the flow of the traveling obstacle avoidance control.

Hereinafter, an embodiment of the combine as a harvester, which is an example of the agricultural work machine according to the present invention, will be described with reference to the drawings. The crops to be harvested are planted culms such as wheat and rice. In this embodiment, when the front-rear direction of the machine body 1 is defined, it is defined along the machine body traveling direction in the working state. The direction indicated by the reference numeral (F) in FIG. 1 is the front side of the aircraft, and the direction indicated by the reference numeral (B) in FIG. 1 is the rear side of the aircraft. When defining the left-right direction of the aircraft 1, the left-right direction is defined in the state of being viewed in the direction of travel of the aircraft. “Upper side (upper side)” or “lower side (lower side)” is a positional relationship of the aircraft 1 in the vertical direction (vertical direction), and indicates a relationship at a height above the ground.

As shown in FIG. 1, in the combine, a harvesting portion 2 which is a harvesting portion is connected to the front portion of the machine body 1 provided with a pair of left and right crawler traveling devices 10 so as to be movable up and down around the horizontal axis. Due to the speed difference between the left and right crawler traveling devices 10, the machine body 1 can turn left and right. The rear part of the machine body 1 is provided with a threshing device 11 in a state of being lined up in the width direction of the machine body, and a grain tank 12 for storing grains. A boarding operation unit 14 is provided on the right side of the front portion of the aircraft 1, and an engine (not shown) is provided below the boarding operation unit 14.

As shown in FIG. 1, the threshing device 11 receives the harvested culm cut by the cutting unit 2 and transported to the rear inside, and sandwiches the stock of the culm by the feed chain 111 and the sandwiching rail 112. The tip side is threshed by the culm 113 while being transported. Then, a grain sorting process for the threshed product is executed in the sorting section provided below the handling cylinder 113, and the grains sorted there are transported to the grain tank 12 and stored. Further, although not described in detail, a grain discharge device 13 for discharging grains stored in the grain tank 12 to the outside is provided.

The cutting unit 2 is provided with a clipper-type cutting device 22 for cutting the root of the raised planted culm, a culm transporting device 23, and the like. The grain culm transport device 23 transports the cut grain culm in the vertical position in which the stock root has been cut toward the start end of the feed chain 111 while gradually changing the cut grain culm to the sideways position.

A satellite positioning module 80 is also provided on the ceiling of the boarding operation unit 14. The satellite positioning module 80 includes a satellite antenna for receiving GNSS (global navigation satellite system) signals (including GPS signals). In order to complement the satellite navigation by the satellite positioning module 80, an inertial navigation unit including a gyro acceleration sensor and a magnetic orientation sensor may be incorporated in the satellite positioning module 80. Of course, the inertial navigation system can be placed elsewhere. An optical sensor unit 9 used for detecting obstacles is provided on the front portion of the combine. The optical sensor unit 9 includes a photographing unit 9A which is an RGB camera, a stereo vision camera 9B capable of measuring depth, and an illuminator 9C.

As shown in FIG. 2, each camera of the optical sensor unit 9 is set so that a specific region in front of the traveling direction of the machine body 1 in the field is set as a shooting field of view. In this embodiment, the horizontal viewing angle, the vertical viewing angle, and the oblique viewing angle of the stereo vision camera 9B are about 100 °, 70 °, and 100 °, respectively.

As shown in FIG. 3, the photographed image output from the photographing unit 9A is used as the input data of the neural network 4. The neural network 4 inputs a captured image, recognizes a specific object appearing in the captured image, and outputs object area information indicating an area occupied by the captured image of the specific object.

In this embodiment, the neural network 4 is configured as a segmentation neural network including an encoder network 4A and a decoder network 4B. The encoder network 4A is composed of an input layer 41 and a plurality of hidden layers 42, and the decoder network 4B is composed of a plurality of hidden layers 42 and an output layer 43. The hidden layer 42 includes a convolutional layer and a pooling layer. The output data output from the segmentation type neural network 4 is a label image for the input captured image, and each pixel is assigned a label that identifies the detected (sorted) specific object. .. In FIG. 3, a shade color is used as the label, and the planted grain culm (uncut land) and the cut trace (already cut land) as specific objects are classified by the shade color.

Further, in this embodiment, in order to speed up the neural network 4, a batch normalization layer 44 is interposed in the hidden layer 42, and pruning is performed based on the parameters in the batch normalization layer 44. ) Is performed. In this pruning, the number of units in the hidden layer 42 is reduced based on the parameters in the batch regularization layer 44, as described below.

とし、
出力を、

とすると、

となる。
ここで用いられたγがユニット削減のパラメータとして用いられる。つまり、学習処理において、パラメータ：γが小さいユニットが削除される。 In the explanation of pruning here, the input vector to the batch regularization layer is

age,
Output,

Then

Will be.
The γ used here is used as a parameter for unit reduction. That is, in the learning process, the unit having a small parameter: γ is deleted.

FIG. 4 shows a schematic flow chart of a process of detecting a specific object and obtaining a spatial position of the specific object using the image captured from the photographing unit 9A of the optical sensor unit 9 and the stereo vision image from the stereo vision camera 9B. It is shown in.

First, on the other hand, when the captured image from the photographing unit 9A is input to the trained neural network 4, it is estimated whether or not a specific object exists in the captured image, and it is estimated that the specific object is estimated. The object area information indicating the area occupied by the specific object is output (#a). In FIG. 4, the object area information is shown as a label image in which a person, a planted culm (uncut land), a cut trace (already cut land), and a shore are separated. The outline of the person, which is the area occupied by the detected person, can be obtained from this label image.

On the other hand, based on the stereovision image from the stereo vision camera 9B, the spatial position measuring unit 52 uses the stereo matching technique to spatially position all the objects existing in the specific area (depth data consisting of a large number of distance measuring points). Generates object position information indicating (a group). In FIG. 4, the object position information is output as an image called a depth image in which the depth of each pixel is represented by light and shade colors (# b). Each pixel corresponds to a stereo matching point, and since these point clouds have three-dimensional coordinate values, they are called three-dimensional point cloud data and can be expanded in a three-dimensional coordinate space (specific area).

Next, based on the object area information output from the neural network 4 and the object position information output from the spatial position measurement unit 52, from the aircraft 1 to a person or an animal (a person in FIG. 4) that becomes a traveling obstacle. The specific object distance, which is the distance of, is calculated by the distance calculation unit 53 (#c). In this embodiment, the specific object distance is calculated from the center of gravity of the human contour (indicated by P1 in FIG. 4) calculated by using a large number of distance measurement points included in the detected human contour. The distance to 1. As the reference point of the specific object distance, a statistical representative value (intermediate value, median value, average value, etc.) of a large number of distance measurement points included in the contour may be used instead of the center of gravity of the contour.

In the above description of FIG. 4, the object position information is given to the distance calculation unit 53 as it is, and is used for calculating the specific object distance. As another embodiment, instead of the method shown in FIG. 4, a method as shown in FIG. 5 may be adopted. In the method shown in FIG. 5, the field information creation unit 54 creates 3D point cloud data developed in a 3D coordinate system with the field scene as one reference plane (bottom surface) from the object position information (#). c1). Further, the field information creation unit 54 creates field cutting information including the shore, the planted culm, the cutting trace, and the boundary between the cut land and the uncut land from the three-dimensional point cloud data and the object area information ( # C2 and # c3). The distance calculation unit 53 determines a specific object that is a traveling obstacle based on the object position information, the object area information, and the field cutting information, and the center of gravity of the determined specific object (shown by P1 in FIG. 5). The distance between (is) and the aircraft 1 is calculated as the specific object distance (#c).

When there is a possibility that the detected traveling obstacle interferes with the aircraft 1, control of avoiding interference with the traveling obstacle is performed. As shown in FIG. 6, the obstacle warning area used for this purpose is defined on a reference plane (XY plane) corresponding to a field scene extending forward in the traveling direction of the aircraft 1. Here, the obstacle warning area is the first warning area (darkly painted and given the code Z1 in FIG. 6) and the second warning area (lightly painted in FIG. 6) having different positional relationships with the aircraft 1. The symbol Z2 is assigned).

The first warning area is a rectangular shape defined by the working width of the cutting unit 2 and the warning distance from the machine 1 to the front in the traveling direction (indicated by the reference numeral L1 in FIG. 6), and the photographing unit 9A. It is a tip wedge-shaped region obtained by cutting out a blind spot region determined by the camera angle of view of the stereo vision camera 9B. In other words, the first warning area is from a rectangle whose side is substantially the width of the cutting section 2 which is the width of the combine's traveling locus, and the other side is a predetermined distance (warning distance) from the aircraft 1 to the front in the traveling direction. , It is a shape that cuts out the blind spot area determined by the camera angle of view. The traveling obstacle existing in the first warning area comes into contact with the combine as the combine progresses. Therefore, the warning distance is preferably changed according to the vehicle speed of the combine, but in this embodiment, since the working running speed is substantially constant, it is set to several meters.

The second warning area is a shape obtained by cutting out the first warning area from the tip wedge-shaped area in which the blind spot area determined by the camera angle of view is cut out in the plane extending forward in the traveling direction of the aircraft 1. It has a width of about three times that of the first warning area in the left-right direction of the combine, and a little less than twice the length of the first warning area in front of the traveling direction of the aircraft 1 (indicated by reference numeral L2 in FIG. 6). ). The left-right spread of the second warning zone is set in consideration of the turning of the combine, and the spread of the traveling direction of the second warning zone has a time margin before the interference between the aircraft 1 and the traveling obstacle. It is set to be larger than the warning area.

FIG. 7 shows a functional block diagram of the combine control system. The control system of this embodiment is composed of a large number of electronic control units called ECUs, various operating devices, sensor groups and switch groups, and a wiring network such as an in-vehicle LAN that transmits data between them. The notification device 84 is a device for notifying a driver, a manager in the vicinity of a field, or the like of a warning such as a detection result of a running obstacle or a state of working running, and is a buzzer, a lamp, a speaker, a display, or the like. When the detected driving obstacle exists in the first warning area, a warning sound, a warning light, or a warning message indicating an imminent state is notified as an emergency warning notification and exists in the first warning area. If so, a gentle warning sound, a warning light, or a warning message is notified as a warning notification rather than an emergency warning notification.

The communication unit 85 is used for exchanging data with an external communication device. The control unit 6 is a core element of this control system and is shown as an aggregate of a plurality of ECUs. The positioning data from the satellite positioning module 80 and the image data from the optical sensor unit 9 are input to the control unit 6 through the wiring network.

The control unit 6 includes an output processing unit 6B and an input processing unit 6A as input / output interfaces. The output processing unit 6B is connected to the vehicle traveling equipment group 7A and the working equipment group 7B. The vehicle traveling device group 7A includes control devices related to vehicle traveling, such as an engine control device, a shift control device, a braking control device, and a steering control device. The working equipment group 7B includes a cutting unit 2, a threshing device 11, a grain discharging device 13, a power control device in the grain transporting device 23, and the like.

A traveling system detection sensor group 8A, a working system detection sensor group 8B, and the like are connected to the input processing unit 6A. The traveling system detection sensor group 8A includes a sensor that detects the state of the engine speed adjuster, the accelerator pedal, the brake pedal, the speed change operation tool, and the like. The work system detection sensor group 8B includes a sensor for detecting the device state in the cutting unit 2, the grain removing device 11, the grain discharging device 13, and the grain transporting device 23, and the state of the grain and the grain.

The control unit 6 is provided with a work travel control module 60, an obstacle detection unit 50, a vehicle speed change command output unit 65, an aircraft position calculation unit 66, and a notification unit 67 as control function units. It should be noted that these control function units are individually configured and connected by an in-vehicle LAN or the like.

The notification unit 67 generates notification data based on a request from each functional unit of the control unit 6 and gives the notification data to the notification device 84. The aircraft position calculation unit 66 calculates the aircraft position, which is the map coordinates (or field coordinates) of the aircraft 1, based on the positioning data sequentially sent from the satellite positioning module 80.

The combine of this embodiment can travel in both automatic driving (automatic steering) and manual driving (manual steering). The work travel control module 60 is provided with a work travel command unit 63 and a travel route setting unit 64 in addition to the travel control unit 61 and the work control unit 62. A traveling mode switch (not shown) for selecting between an automatic traveling mode in which the vehicle travels by automatic steering and a manual steering mode in which the vehicle travels by manual steering is provided in the boarding operation unit 14. By operating this travel mode switch, it is possible to shift from manual steering running to automatic steering running, or from automatic steering running to manual steering running.

The travel control unit 61 has an engine control function, a steering control function, a vehicle speed control function, and the like, and gives a travel control signal to the vehicle travel equipment group 7A. The work control unit 62 gives a work control signal to the work equipment group 7B in order to control the movements of the cutting unit 2, the threshing device 11, the grain discharge device 13, the grain culm transport device 23, and the like.

The travel route setting unit 64 develops a travel route for automatic travel in the memory. The travel route developed in the memory is used as a target travel route in sequential automatic driving. This travel route can be used for guidance for the combine to travel along the travel route even in manual travel.

The work travel command unit 63 generates an automatic steering command and a vehicle speed command as automatic travel commands and gives them to the travel control unit 61. The automatic steering command is generated so as to eliminate the directional deviation and the positional deviation between the traveling route and the own aircraft position calculated by the aircraft position calculation unit 66 by the traveling route setting unit 64. At the time of automatic driving, the vehicle speed command is generated based on the vehicle speed value set in advance. During manual driving, the vehicle speed command is generated based on the manual vehicle speed operation. However, in the event of an emergency such as detection of a running obstacle, the vehicle speed will be changed, including a forced stop, and the engine will be stopped automatically.

When the automatic driving mode is selected, the traveling control unit 61 controls the vehicle traveling equipment group 7A related to steering and the vehicle traveling equipment group 7A related to vehicle speed based on the automatic traveling command given by the work traveling command unit 63. When the manual driving mode is selected, the driving control unit 61 generates a control signal based on the operation by the driver to control the vehicle traveling device group 7A.

The obstacle detection unit 50 performs the object detection process described with reference to FIG. 4 or FIG. Therefore, the obstacle detection unit 50 includes an AI unit 51, a spatial position measurement unit 52, a distance calculation unit 53, a field information creation unit 54, an object detection signal generation unit 55, and a warning area setting unit 56. The above-mentioned components constituting the obstacle detection unit 50 are mainly separated for the purpose of explanation, and the components may be integrated or the components may be freely divided.

The AI unit 51 implements the trained model of the segmentation neural network 4 described in FIG. 3, and detects a traveling obstacle (specific object detection) by the principle as described in FIG. 4 or FIG.

The spatial position measurement unit 52, the distance calculation unit 53, and the field information creation unit 54 have the functions described with reference to FIGS. 4 and 5. The warning area setting unit 56 sets an obstacle warning area defined based on the position with respect to the aircraft 1. In this embodiment, the obstacle warning area is divided into a first warning area and a second warning area as described with reference to FIG.

The distance calculation unit 53 calculates the position of the detected traveling obstacle, particularly the distance from the aircraft 1 to the traveling obstacle (specific object distance) by linking with the AI unit 51, the space position measuring unit 52, and the like. .. The object detection signal generation unit 55 determines whether the detected traveling obstacle exists in the first warning area or the second warning area based on the distance from the aircraft 1 to the traveling obstacle. Then, an object detection signal for each warning area is generated. The object detection signal includes a traveling obstacle notification signal given to the notification unit 67 and an obstacle traveling signal given to the vehicle speed change command output unit 65, and the obstacle traveling signal includes a traveling obstacle. Signals for each alert area are included.

The vehicle speed change command output unit 65 outputs the first vehicle speed change command including the vehicle speed change value to the work travel command unit 63 from the received obstacle traveling signal when the traveling obstacle exists in the first warning area. When the traveling obstacle exists in the second warning area, the second vehicle speed change command including the vehicle speed change value is output to the work traveling command unit 63. The first vehicle speed change command is a command to reduce the vehicle speed to zero, that is, a command to stop the aircraft 1. Of course, the first vehicle speed change command may be a command to set the vehicle speed to a value close to zero, or a command to gradually reduce the vehicle speed to zero. The second vehicle speed change command is a deceleration value for decelerating the vehicle speed to a very small speed, for example, about 1 km / h. As the second vehicle speed change command, a deceleration value determined by a ratio to the immediately preceding vehicle speed, such as decelerating to a fraction of the immediately preceding vehicle speed, may be used. In any case, the first vehicle speed change value by the first vehicle speed change command is larger than the second vehicle speed change value by the second vehicle speed change command, and is a command for emergency evacuation. Based on the vehicle speed change command given from the vehicle speed change command output unit 65, the work travel command unit 63 gives a control command to the travel control unit 61 to urgently decelerate the aircraft 1. When the presence of the traveling obstacle shifts from the first warning area to the second warning area, the stopped aircraft 1 moves forward at a slow speed specified by the second vehicle speed change command. Furthermore, if the running obstacle is not detected in the caution area, the running returns to the normal working vehicle speed.

Next, FIG. 8 shows an example of a vehicle speed deceleration control routine when a traveling obstacle is detected in the above-mentioned control system. When this control routine starts, initial settings such as resetting flags and variables are performed (# 01). The alert area is set, that is, the first alert area and the second alert area are set (# 02). When the first warning area is changed by the vehicle speed, the current vehicle speed is acquired, and the first warning area is set based on the vehicle speed.

Image data (captured image) is captured from the optical sensor unit 9 (# 03). Using the captured image data, the obstacle detection unit 50 performs a specific object detection process (running obstacle detection process) (# 04).

It is determined whether or not a running obstacle is detected, and if a running obstacle is detected (# 05Yes branch), the position of the running obstacle in the obstacle warning area is calculated (# 11).

It is checked whether the running obstacle is in the first alert area (# 12). If the traveling obstacle is in the first warning area (# 12Yes branch), the vehicle speed change command output unit 65 gives a stop command to the working travel command unit 63 as the first vehicle speed change command (# 13). Furthermore, the stop flag is turned ON (# 14). In step # 14, when the deceleration flag is ON (when a traveling obstacle moves from the second warning area to the first boundary area), the deceleration flag is turned OFF and the stop flag is turned ON. As a result, the aircraft 1 is stopped, and the control returns to step # 02.

If the traveling obstacle is not in the first warning area (# 12No branch) in the check of step # 12, it is further checked whether or not it is in the second warning area (# 20). If the traveling obstacle is in the second warning area (# 20Yes branch), the vehicle speed change command output unit 65 gives a deceleration command to the working travel command unit 63 as the second vehicle speed change command (# 21). Furthermore, the deceleration flag is turned ON (# 22). In step # 22, when the stop flag is ON (when a traveling obstacle moves from the first warning area to the second boundary area), the stop flag is turned OFF and the deceleration flag is turned ON. As a result, the aircraft 1 travels at a very low speed (about 1 km / h), and the control returns to step # 02.

If the running obstacle is not in the second warning area in the check of step # 20 (# 20 No branch), there is no running obstacle in the warning area. In this way, when there is no running obstacle in the warning area, or when no running obstacle is detected in the first place (# 08No branch), the state of the stop flag and the state of the deceleration flag are checked, and the state is changed to that state. The flag contents are rewritten accordingly. Specifically, at that time, if the stop flag is ON (# 30Yes branch), the stop flag is rewritten to OFF (# 31), the travel command becomes valid, and the aircraft 1 normally travels (# 30Yes branch). # 32). Control then returns to step # 02. At that time, if the stop flag is not ON (# 30No branch), the content of the deceleration flag is checked (# 40). If the deceleration flag is ON (# 40Yes branch), the deceleration flag is rewritten to OFF (# 41), the travel command becomes valid, and the aircraft 1 normally travels (# 42). The normal running is a work running when a running obstacle or the like is not detected, and the vehicle speed is faster than the vehicle speed when the running obstacle is in the second warning area. Control then returns to step # 02. If the deceleration flag is OFF (# 40 No branch), of course, the travel command is valid and normal travel is performed (# 43). Then, the control returns to step # 02.

By controlling the traveling obstacle detection and the vehicle speed change, for example, when a field worker enters the warning area ahead in the traveling direction of the aircraft 1, the aircraft 1 stops or decelerates, and the field worker comes out of the warning area. As a result, the combine harvester returns to working at normal vehicle speed.

The control of vehicle speed deceleration based on the detection of a traveling obstacle can be executed regardless of whether the combine is automatic traveling or manual traveling.

In the above-described embodiment, the first warning area and the second warning area are set, and when a traveling obstacle exists in each warning area, the aircraft 1 is stopped or the vehicle speed is changed to a speed slower than the normal driving. It was. Instead of this, three or more warning areas may be provided, and the aircraft 1 may be stopped or changed to a slow speed according to each area. Further, the warning area may be set steplessly, and the aircraft 1 may be stopped or the vehicle speed may be changed steplessly to a vehicle speed slower than the normal traveling according to the distance between the aircraft 1 and the traveling obstacle.

The configuration disclosed in the above embodiment (including another embodiment, the same shall apply hereinafter) can be applied in combination with the configuration disclosed in other embodiments as long as there is no contradiction. The embodiments disclosed in the present specification are examples, and the embodiments of the present invention are not limited thereto, and can be appropriately modified without departing from the object of the present invention.

The present invention is applied to agricultural work machines such as harvesters, rice transplanters, and totactors that perform work running in the field.

1: Aircraft 2: Cutting unit 4: Segmentation neural network (neural network)
4A: Encoder network 4B: Decoder network 41: Input layer 42: Hidden layer 43: Output layer 44: Batch normalization layer 50: Obstacle detection unit 51: AI unit 52: Spatial position measurement unit 53: Distance calculation unit 54: Field Information creation unit 55: Object detection signal generation unit 56: Warning area setting unit 6: Control unit 61: Travel control unit 63: Working travel command unit 65: Vehicle speed change command output unit 67: Notification unit 9: Optical sensor unit 9A: Shooting Unit 9B: Stereo vision camera

Claims (7)

It is an agricultural work machine that runs in the field.
An imaging unit that photographs a specific area of the field, and
A neural network that inputs a photographed image from the photographing unit, recognizes a specific object reflected in the photographed image, and outputs object area information indicating an area occupied by the specific object in the photographed image.
A spatial position measuring unit that photographs the specific area and generates object position information indicating the spatial positions of all the objects existing in the specific area.
A distance calculation unit that determines the spatial position of the specific object based on the object position information and the object area information and calculates the specific object distance that is the distance from the machine body to the specific object.
An object detection signal generator that outputs an object detection signal based on the specific object distance,
Agricultural work machine equipped with. The agricultural work machine according to claim 1, wherein the neural network is composed of a segmentation neural network including an encoder network and a decoder network, and the object area information includes a contour of the specific object. The second aspect of claim 2, wherein the encoder network and the hidden layer constituting the decoder network include a convolutional layer, a pooling layer, and a batch normalization layer, and pruning is performed based on the parameters in the batch normalization layer. Agricultural work machine. The specific object distance is a distance from the center of gravity of the specific object to the aircraft calculated from the contour of the specific object, or a statistical representative value of a distance measurement point included in the contour, according to claim 2 or 3. Agricultural work machine described in. The agricultural work machine according to any one of claims 1 to 4, wherein the spatial position measuring unit generates the object position information by using the stereo matching technique for the specific area. The agricultural work machine is a harvester having a harvesting section for harvesting agricultural products.
A claim that includes a field information creating unit that creates field cutting information including the boundary between the cut land and the uncut land cut by the harvesting unit and the height of the crop based on the object position information. The agricultural work machine according to any one of 1 to 5. It is provided with a warning area setting unit that sets an obstacle warning area defined based on the position from the aircraft, and a vehicle speed change command output unit that changes the vehicle speed.
According to claims 1 to 6, when the specific object is a traveling obstacle and the traveling obstacle exists in the obstacle warning area, the vehicle speed change command output unit changes the vehicle speed based on the specific object distance. Agricultural work machine described in any one item.

Images