JP7353417B2 - Field work vehicle - Google Patents

Description

The present invention relates to a field work vehicle that performs work while automatically traveling in a field.

One method for automatically driving in a field uses a preset automatic driving route and a work vehicle position obtained by satellite positioning. For example, in the tractor disclosed in Patent Document 1, a work area to be worked on by automatic driving is formed inside an existing work area created by first manually driving around the outer periphery of a field. . A travel route for the tractor to travel through this work target area is calculated based on a route calculation algorithm. After that, the automatic driving control unit installed in the Totakuta calculates the positional deviation between the target driving route and the own vehicle position calculated from the satellite positioning data, and issues an automatic steering command to reduce this positional deviation. is generated.

Another method for automatically traveling in a field uses a target travel route of a field work vehicle obtained by processing images taken by a camera mounted on the field work vehicle. For example, in the grass mower disclosed in Patent Document 2, a boundary area between mowed land and unmown land in a lawn area such as a golf course is photographed by a photographing device, and from the photographed image, the boundary area between mowed land and unmown land is determined. The boundary line between the two is calculated, and the vehicle is automatically steered along this boundary line.

Japanese Patent Application Publication No. 2018-116608 JP2018-097526A

In order to implement automatic driving using satellite positioning data as disclosed in Patent Document 1, map information of the field (field coordinate system or earth coordinate system) is prepared, and using this map information, A driving route covering the field is calculated. The work vehicle automatically travels so as not to deviate from this travel route. The vehicle's position is calculated using satellite positioning, but when calculating the vehicle's position using satellite positioning, if the satellite radio waves are blocked by mountains or houses, the problem arises that positioning becomes impossible or the accuracy of positioning decreases.

As disclosed in Patent Document 2, automatic driving that uses the boundary line between mowed land (already worked land) and unmown land (unworked land) detected by image processing as a target travel route, This is an effective alternative to autonomous driving using satellite positioning. However, the field surface on which the field work vehicle travels is more uneven than a lawn, and the conditions vary depending on the location, which leads to a deterioration in steering accuracy. Furthermore, if there are agricultural products or residues of agricultural products in the field, their condition will vary depending on the location, so automatic driving may be required depending on the condition of the crops.

An object of the present invention is to provide a field work vehicle that can travel automatically without using satellite positioning and while taking into account the state of the field.

A field work vehicle according to the present invention that automatically travels in a field includes: a photographing unit that photographs the field including at least the advancing area of the vehicle body and outputs a field image; a traveling device for adjusting the vehicle speed and the amount of steering; An automatic driving calculation unit that is trained to input a field image and output automatic driving information for automatic driving based on the boundaries between unworked areas and already worked areas and the field condition; an automatic travel control unit that controls the traveling equipment based on the
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a second learning image group created by adding annotations to the first learning image group, which is the field image, as training data. , the first learning image group is an image group including the unworked area and the already worked area of the field, and the second learning image group is an image group including the unworked area and the already worked area, which serve as a reference for the driving route. a group of images to which an annotation indicating the boundary between the already worked area and the boundary line determined from the boundary is added;
The automatic driving information is information for automatically driving along the boundary .

With this configuration, the target running direction of the vehicle body is determined by estimating the boundary between the unworked area and the completed work area from the field image acquired by the photographing unit, and at the same time, the field condition is estimated from the field image. Then, a target driving behavior that is suitable for the field conditions is determined, and automatic driving information is output from the target driving direction and driving behavior. The automatic travel calculation unit is trained to output optimal automatic travel information including not only the boundaries between unworked areas and completed work areas but also field conditions. Therefore, when the traveling equipment is controlled along the boundary between the unworked area and the completed work area, field conditions (field scene conditions, agricultural product conditions, etc.) are also taken into consideration. For example, if there is mud or unevenness in front of the vehicle, it is possible to reduce the amount of steering to reduce slippage.

In order to avoid omitted work or redundant work (excessive overlap), it is necessary to accurately determine the deviation of the vehicle body from the target path and find a steering amount that will reduce the deviation. For this reason, it is important to determine the boundary between the unworked area and the completed area by a single line, that is, the boundary line, and to output a steering command that aligns the steering reference point of the vehicle body with this boundary line. be. Therefore, in one of the preferred embodiments of the present invention, the automatic driving information estimates a linear target driving route corresponding to the boundary line indicating the boundary, and combines this linear target driving route with the automatic driving information. Contains information for resolving discrepancies with the vehicle.

During field work such as plowing, rice planting, and harvesting, the outer circumferential area that has already been worked on is used as a direction change area to change the direction of the vehicle, and the area parallel to the inner area (unworked area) of the outer circumferential area is A driving style is adopted in which linear routes are connected by changing directions such as U-turns. In order to employ such a driving pattern in automatic driving according to the present invention, control is required to shift to direction-changing driving when the vehicle body reaches an outer peripheral area where it is necessary to change direction. Therefore, in one of the preferred embodiments of the present invention, the automatic driving information includes a direction change with a direction change travel route set in the vehicle body direction changeable area, which is the already worked area, as a target travel route. Contains information on how to do this.

There may be obstacles in the field, such as people, animals, or even wells or steel towers, so in order to perform automatic driving, it is necessary to detect these obstacles and prevent contact with them. Need to avoid. For this reason, in one of the preferred embodiments of the present invention, the automatic driving calculation unit uses collision avoidance information for avoiding contact with a traveling obstacle estimated from the field image as the automatic driving information. Output.

Countermeasures will differ depending on whether the obstacle is moving or fixed in the field. The work vehicle must avoid obstacles fixed in the field. Therefore, in one of the preferred embodiments of the present invention, the collision avoidance information includes avoidance travel information for avoiding the travel obstacle that is estimated to be a non-moving obstacle.

Obstacles that move in the field may move away from the path of the work vehicle over time and may no longer be obstacles to the vehicle. It is also possible to remove obstacles by issuing a warning or the like. Therefore, in one of the preferred embodiments of the present invention, the collision avoidance information includes at least information such as deceleration, stopping, and warning to avoid contact with the traveling obstacle estimated to be a moving obstacle. Contains information for doing one thing.

When field work is performed manually, the driver considers the current vehicle speed, engine speed, etc., and changes these appropriately to achieve optimal work driving. Such operations are also useful in automated driving. For this reason, one of the preferred embodiments of the present invention is provided with a running state detection unit that detects the running state from the state of the traveling equipment, and the running state information indicating the running state is sent to the automatic running calculation unit. The driving state information includes data indicating at least one of vehicle speed, gear shift position, engine rotation speed, and engine load factor, and the automatic driving information includes steering, vehicle speed, gear shifting, braking, A control command for commanding a change in at least one of the engine speeds is included.

Agriculture has developed through the many years of experience of farmers and technological innovations in field vehicles, and in order to advance the automation of field vehicles, it is necessary to reflect those years of experience in the automatic driving control of field vehicles. . To this end, it is preferable to introduce machine learning into the core of the control system, which allows field work vehicles to learn from the experiences of farmers. This allows the abilities of a skilled farmer to be realized on a computer mounted on a field work vehicle. For this reason, the automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a first learning image group and a second learning image group as learning images, and the first learning image group is an image group including the unworked area and the already worked area of the field, and the second learning image group includes an annotation indicating the boundary between the unworked area and the already worked area, This is a group of images to which an annotation indicating a positional shift with respect to a travel route for automatically traveling along the boundary is added. An annotation is a meaning given to a corresponding photographed image. For example, in a photographed image of a field, an annotation can be used to indicate the boundary between an unworked area and an already worked area, and furthermore to point out a positional deviation from the boundary with respect to the vehicle's travel route and an operation to eliminate the positional deviation. It is. This machine learning includes first learning and second learning. The first learning involves finding the boundary between the unworked area and the already worked area in the raw photographed image obtained from the perspective of the field work vehicle, and the boundary area as an annotation. This is a learning method that estimates boundary areas from raw captured images using annotated images. The second learning is based on the raw captured images and annotations that indicate the positional deviation of the vehicle body from the found boundary position (deviation from the target driving line) and the required operation to eliminate the positional deviation. This is a learning that uses images to estimate the positional deviation boundary area of the vehicle body from the raw captured image, and estimates the operation to eliminate the positional deviation. Through this learning, the automatic driving calculation unit can output the position of the boundary line in front of the vehicle body, the positional shift of the vehicle body, and, if necessary, the operation for eliminating the positional shift, from the acquired field image.

When a positional shift of the vehicle body occurs, steering is performed to align the vehicle body with the target travel route. In order to eliminate positional deviations caused by steering quickly and efficiently, control is performed by combining lateral positional deviations, which are deviations in the direction transverse to the driving route, and azimuth deviations, which are deviations in the vehicle's orientation relative to the direction of the driving route. It is preferable that the This is due to the fact that field work vehicles that travel in fields have more slips and lower vehicle speeds than vehicles that travel on roads. Therefore, in one of the preferred embodiments of the present invention, the positional deviation includes a lateral positional deviation that is a deviation in a transverse direction with respect to the traveling route, and an azimuth that is a deviation in the vehicle body orientation with respect to the direction of the traveling route. Includes misalignment.

Furthermore, in one of the preferred embodiments of the present invention, a third learning image and a fourth learning image are further used as the learning image, and the third learning image The fourth learning image is a group of images to which an annotation indicating the region where the vehicle direction can be changed is added. The automated driving calculation unit that has learned using such learning images creates a direction change area for changing the direction of the vehicle from the current work route and moving to the next work route, and the direction change area. Information for changing direction (turning) can be estimated from field images.

Furthermore, in one of the preferred embodiments of the present invention, a fifth learning image and a sixth learning image are further used as the learning image, and the fifth learning image is The learning image is a group of images including an object that becomes an obstacle, and the learning image is a group of images to which an annotation indicating the object that becomes an obstacle is added. The automatic driving calculation unit trained using such training images is able to estimate objects that will be obstacles to driving from field images, and provide the automatic driving information necessary to avoid contact with these obstacles. can be output.

FIG. 1 is a side view of a tractor that is an example of a field work vehicle. FIG. 2 is a schematic plan view of the tractor for explaining the mounting position of a photographing camera. It is an explanatory diagram explaining the flow of field work by a tractor. FIG. 3 is a functional block diagram of a control system. FIG. 3 is a functional block diagram of an automatic driving calculation section. FIG. 3 is a schematic diagram showing a learning image used in learning processing of the first calculation unit. FIG. 3 is a schematic diagram showing a learning image used in learning processing of the first calculation unit. FIG. 6 is a schematic diagram showing a learning image used for learning processing by a second calculation unit. FIG. 6 is a schematic diagram showing a learning image used for learning processing by a second calculation unit. It is a flowchart which shows an example of the learning process of an automatic driving calculation part. FIG. 3 is a functional block diagram of another embodiment of an automatic driving calculation unit.

Next, a field work vehicle that automatically travels in a field according to the present invention will be explained using the drawings. In the embodiment shown in FIG. 1, the work vehicle is a tractor that automatically travels in a field. In this tractor, a driving section 20 is provided in the center of a vehicle body 1 supported by front wheels 11 and rear wheels 12. A working device 24, which is a rotary tiller, is installed at the rear of the vehicle body 1 via a hydraulic lifting mechanism 23. The front wheels 11 function as steering wheels, and by changing the steering angle, the traveling direction of the tractor is changed. The steering angle of the front wheels 11 is changed by the operation of the steering mechanism 13. The steering mechanism 13 includes a steering motor 14 for automatic steering. During manual driving, the front wheels 11 are steered by operating a steering wheel 22 disposed in the driving section 20.

This tractor is equipped with a photographing unit 3 that photographs a field including at least the region where the vehicle body 1 is traveling and outputs a field image. In this embodiment, as shown in FIG. 2, the photographing unit 3 includes a front camera 31, a rear camera 32, a left camera 33, and a right camera 34. The front camera 31 is arranged at the front of the vehicle body 1 and photographs the front of the vehicle body 1. The rear camera 32 is arranged at the rear of the vehicle body 1 and photographs the rear of the vehicle body 1 including the working device 24. The left camera 33 and the right camera 34 are respectively arranged on the side of the vehicle body 1 and take images of the left side and the right side of the vehicle body 1. By converting and combining the images captured by the front camera 31, the rear camera 32, the left camera 33, and the right camera 34, it is possible to generate an overhead image with the viewpoint above the tractor. As the field image, a front photographed image by the front camera 31, a field image, or both are used. Note that the forward photographed image is used as a field image including the vehicle body advancing area in the case of forward movement, and the rear photographed image by the rear camera 32 is used as the field image including the vehicle body advancing area in the case of backward movement. Therefore, it is assumed that the field images described below include a front photographed image, a rear photographed image, and an overhead view image.

The general procedure for field work using a tractor is shown in Figure 3. In this example, when the tractor enters the field (#a), a peripheral work drive is performed manually or automatically to perform work along the outer periphery of the field, and an outer peripheral area, which is an already worked area, is formed on the outermost side of the field. (#b). For surrounding work driving, manual driving is preferred because turning around corners is complicated. Since the vehicle travels substantially in a straight line except for corner areas, automatic travel is used. At that time, the boundary line between the field scene and the ridges, furrows, etc., which is the outer circumference of the field, is recognized in real time by an image recognition program using machine learning, and the working end of the working device 24 is placed on the recognized boundary line. The tractor will be automatically steered so that the When the outer peripheral area formed by this surrounding work travel becomes large enough to allow the tractor to change direction (U-turn travel), automatic work travel to this unworked area is started. This automatic work driving includes forward work driving (#c) in which the tractor travels in a straight line, and vehicle body direction change driving (#c) in which the working device 24 is raised in the already worked area and the tractor is turned into a non-working state. d). By repeating forward work travel and vehicle direction change travel (#e), work on the entire field is completed. In addition, in forward work driving, the image recognition target is the boundary line between the unworked area and the completed work area, and in the vehicle direction change driving, the image recognition target is the field boundary line (outer line) and the unworked area and the completed work area. It is the border between the area. For this reason, it is preferable that different image recognition programs be used for forward work driving and vehicle direction change driving. In that case, automatic switching between the image recognition program for forward work travel and the image recognition program for vehicle direction change travel is performed based on the state of the work device 24 (for example, descending state and rising state, driving state and non-driving state). be able to. Note that depending on the type of field, the shape of the outer peripheral area is assumed in advance, and while the outer peripheral area remains unworked, #c, #d, and #e are placed first in the unworked area inside the outer peripheral area. #b is implemented after that.

FIG. 4 shows the control system built in this tractor. The control system of this embodiment includes a control unit 5 consisting of a group of in-vehicle ECUs and an input/output signal processing unit 50. The control unit 5 is equipped with an automatic driving calculation unit 4 constructed by applying machine learning in addition to a functional unit that executes normal work driving. The input/output signal processing unit 50 is constructed with a functional section that processes input/output signals handled during work travel of the tractor. The control unit 5 and the input/output signal processing unit 50 are connected by an in-vehicle LAN.

Connected to the input/output signal processing unit 50 are a traveling equipment group 91, a working equipment group 92, a notification device group 93, an automatic/manual switching operation tool 94, a traveling state detection sensor group 81, and a working state detection sensor group 82. . The traveling equipment group 91 includes a steering motor 14 (see FIG. 1) as a steering adjustment device, a speed adjuster as a vehicle speed adjustment device, an engine rotation adjuster, and the like. The work equipment group 92 includes control equipment for driving the work device 24 and the lifting mechanism 23 (see FIG. 1). The notification device group 93 includes a display, a speaker, a lamp, a buzzer, and the like. The automatic/manual switching operation tool 94 is a switch that selects either an automatic driving mode in which the vehicle travels with automatic steering or a manual steering mode in which the vehicle travels with manual steering.

The running state detection sensor group 81 includes sensors that detect running states such as the amount of steering, engine speed, and shift position. The working state detection sensor group 82 includes sensors that detect the posture and driving state of the working device 24.

In addition to the above, a photographing unit 3 is connected to the input/output signal processing unit 50. The photographing unit 3 includes an image processing section 30 that processes images taken by the four cameras 31 to 34 described above to generate a field image suitable for processing by the control unit 5. The field image output from the photographing unit 3 is input to the control unit 5 via the input/output signal processing unit 50.

The control unit 5 includes a manual travel control section 51, an automatic travel control section 52, a work control section 53, a travel state detection section 54, a work state detection section 55, and an automatic travel calculation section 4. The manual travel control unit 51 controls the travel equipment group 91 in the manual travel mode based on the operation by the driver. The automatic travel control section 52 controls the traveling equipment group 91 in the automatic travel mode based on the automatic travel information output from the automatic travel calculation section 4. The work control unit 53 manually or automatically controls the work equipment group 92. The running state detection unit 54 detects the running state based on the detection signal from the running state detection sensor group 81. The working state detection unit 55 detects the working state based on the detection signal from the working state detection sensor group 82.

In the learned state, the automatic driving calculation unit 4 inputs the field image, estimates the boundary between the unworked area and the worked area, and the field state, and outputs automatic driving information for automatic driving. . In this embodiment, the automatic driving calculation section 4 includes a first calculation section 41, a second calculation section 42, and a selection section 40, as shown in FIG. The first calculation section 41 includes a first front-stage calculation section 41a and a first rear-stage calculation section 41b. The second calculation section 42 includes a second front-stage calculation section 42a and a second rear-stage calculation section 42b.

The first front-stage calculation unit 41a estimates the boundary line indicating the boundary between the unworked area and the worked area and the field state from the field image. The first latter-stage calculation unit 41b calculates in real time a linear target travel route (forward work travel route) corresponding to the estimated boundary line. At that time, the working width of the working device 24, the working overtap amount, the distance from the vehicle body reference point to the working end of the working device 24, etc. are required, so these working device specification data are set in advance. Furthermore, the first pre-stage calculation unit 41a calculates a positional deviation of the steering reference point of the vehicle body 1 with respect to this target driving route (forward work driving path), and first automatic driving information that is information for eliminating this positional deviation. generate. This positional deviation includes a lateral positional deviation, which is a positional deviation in a transverse direction with respect to the target driving route, and an azimuth deviation, which is an angular deviation between the orientation of the target driving route and the vehicle body orientation. The first automatic travel information includes a steering amount for eliminating positional deviation. When generating the steering amount, field conditions such as the slope of the field and the unevenness of the field are taken into consideration. The first front-stage calculation section 41a and the first rear-stage calculation section 41b can be integrated.

The second front-stage calculation unit 42a estimates an area where the vehicle body direction can be changed, which is an already worked area, from the field image. The second latter-stage calculation unit 42b sets a turning travel route to be set in the vehicle body direction changeable region, defines the difference between the set turning travel route and the actual turning trajectory as a turning deviation, and determines the difference between the set turning travel route and the actual turning trajectory. Second automatic driving information, which is information (steering amount) for solving the problem, is generated. When generating the second automatic driving information, field conditions such as the slope of the field and the unevenness of the field are taken into consideration. The second front-stage calculation section 42a and the second rear-stage calculation section 42b can be integrated.

The selection unit 40 compares the reliability of the first automated driving information and the reliability of the second automated driving information, selects the information with higher degree of appropriateness as the driving control that must be performed at the present time, and selects the information that has a higher degree of appropriateness as the driving control that must be performed at the present time. It is output as information and given to the automatic travel control section 52. Alternatively, the selection unit 40 may select either the first automatic travel information or the second automatic travel information based on the work state detected by the work state detection unit 55. For example, if the state of the work device 24 is a lowered state, a driven state, or both, it is assumed that forward work travel is being performed, and the first automatic travel information is selected. If the state of the working device 24 is a raised state, a non-driving state, or both, it is considered that the vehicle is changing direction, and the second automatic travel information is selected. As yet another method, the use of either the first pre-processing section 41a or the second pre-processing section 42a may be determined based on the working state detected by the working state detecting section 55.

Note that in the automatic driving calculation section 4 shown in FIG. 5, as an additional function, driving state information detected by the driving state detection section 54 is inputted to the first calculation section 41 and the second calculation section 42. There is. The driving state information includes vehicle speed, gear shift position, engine rotation speed, engine load factor, and the like. When this function is added, the first rear-stage calculation unit 41b and the second rear-stage calculation unit 42b add vehicle speed control commands to the first automatic driving information and the second automatic driving information in addition to the information for eliminating positional deviation. , a shift control command, a brake control command, and an engine control command. This is because not only adjusting the amount of steering but also adjusting other driving conditions (vehicle speed, etc.) may be effective in eliminating positional deviation.

The first calculation section 41 is a machine learning control unit that outputs first automatic travel information from a field image, and the second calculation section 42 is a machine learning control unit that outputs second automatic travel information from a field image. Learning processing is necessary for the machine learning control unit to estimate the field work situation from the field image and output appropriate automatic driving information. For this learning process, the learning processing unit 6 is connected to the automatic driving calculation section 4 in FIG. 5 . Note that the learning processing unit 6 may be built in the automatic driving calculation section 4. Alternatively, a learning system may be adopted in which the learning processing unit 6 built in a computer system other than the control system of the tractor and the automatic travel calculation section 4 exchange data to execute the learning process. In this embodiment, the first calculation unit 41 and the second calculation unit 42 are constructed by a convolutional neural network, and the learning processing unit 6 performs learning processing that is compatible with the convolutional neural network.

The configuration of the learning processing unit 6 is well known, so a detailed explanation will be omitted here, but it has an error calculation function and a weight adjustment function. The learning processing unit 6 may include a convolutional neural network similar to the first computing section 41 or the second computing section 42, or may be directly connected to the first computing section 41 or the second computing section 42 before learning. , may be learned sequentially. In the former case, learned data such as weighting coefficients are set in the first calculation unit 41 and the second calculation unit 42 after the learning process is completed.

In the learning process of the first calculation unit 41, a first learning image group consisting of a large number of first learning images and a second learning image group consisting of a large number of second learning images are used.

The first learning image group is an image group including an unworked area and a completed work area of the field, and in this embodiment, an overhead image generated by the image processing section 30 of the photographing unit 3 is used. The second learning image group is an image group to which an annotation indicating a feature to be recognized in the first learning image group is added. An example of the first learning image is shown on the left side of FIGS. 6 and 7, and a second learning image group is shown on the right side of FIGS. 6 and 7. The first learning image in FIG. 6 shows a scene extending into an unworked area in front of a tractor located in an already worked area. In the second learning image in FIG. 6, there are annotations indicating the boundary between the unworked area and the worked area, which serve as the reference for the travel route, and the boundary line obtained from the boundary, and the boundary, which serves as the start reference for work travel, and the An annotation that shows the boundary line determined from the boundary, an annotation that shows the positional deviation when the boundary line is the target travel route, an annotation that shows that the surface condition of the unworked area is moderately uneven, and the required amount of steering. An annotation indicating . The first learning image in FIG. 7 is an overhead image of a tractor that has entered an unworked area and is traveling along the boundary between the working area and the completed working area. In this second learning image, there are annotations that indicate the boundary that serves as a reference for the driving route (forward and reverse) and a boundary line calculated from the boundary, an annotation that shows the positional deviation when the boundary line is the target driving route, and an annotation that shows the boundary line that is the reference for the driving route (forward and backward) An annotation indicating that the surface condition of the work area is moderately uneven and an annotation indicating the required amount of steering are provided.

In the learning process of the second calculation unit 42, the third learning image and the fourth learning image are used as learning images. The third learning image group is also an image group including an unworked area and a worked area of the field, but the already worked area in front of the vehicle body 1 is an area where the vehicle direction can be changed, and the fourth learning image is a group of images to which an annotation indicating a region in which the vehicle direction can be changed is to be recognized in the third group of learning images. An example of the third learning image is shown on the left side of FIGS. 8 and 9, and a fourth learning image group is shown on the right side of FIGS. 8 and 9. The third learning image in FIG. 8 shows a scene extending into the already worked area in front of the tractor during work travel. In the fourth learning image of FIG. 8, there is an annotation indicating that the completed work area extending ahead is an area where the vehicle direction can be changed, and an annotation indicating that the surface condition of the area where the vehicle body direction can be changed is moderately uneven. An annotation and an annotation indicating the required amount of steering are provided. Further, the fourth learning image shows a turning route for changing the vehicle direction that is set in the vehicle direction changeable area when the vehicle body direction changeable area is recognized. The third learning image in FIG. 9 is an overhead image near the end of the turning run for changing the direction of the vehicle body. In the fourth learning image in FIG. 9, there are annotations that indicate the boundary that serves as a reference for the driving route and a boundary line that is calculated from the boundary, an annotation that shows the positional deviation when the boundary line is set as the target driving route, and an annotation that shows the already worked area and An annotation indicating that the surface condition of the unworked area is moderately uneven and an annotation indicating the required amount of steering are provided.

FIG. 10 shows a flow of a learning process for making the first calculation unit 41 function to output first automatic driving information indicating the amount of steering, using the first learning image group and the second learning image group. It is shown.

First, a first learning image group is acquired using a test drive of a tractor, photographed images acquired by a camera-equipped drone, or a CG device (#01). Annotations indicating noteworthy feature regions are added to the first learning image group to create a second learning image group (#02).

Second learning images are sequentially extracted from the second learning image group and input to the convolutional neural network forming the first calculation unit 41 (#11). Automatic driving information estimated by the convolutional neural network is output using the second learning image (#21). The difference (error) between the estimated automatic driving information and the automatic driving information indicated by the annotation added to the second learning image is calculated (#22). If the calculated error is not less than the minimum value (#23 No branch), the weighting coefficient is corrected (#24), and the process returns to step #21 to repeat the process of reducing the error. If the calculated error is less than the minimum value (#23 Yes branch), the weighting coefficient at that time is recorded (#25).

It is checked whether the learning process using all the prepared second learning images has been completed (#26). If there are still second learning images to be used in the learning process (#26 Yes branch), the process returns to step #11 to continue the learning process. If the second learning image used for the learning process does not remain (#26 No branch), the final weighting coefficient is determined based on the recorded weighting coefficient (#31), and the weighting coefficient is used as the convolution neural It is set in the first calculation unit 41, which is a network (#32). Similar images from the second learning image group may be grouped, and a weighting coefficient that minimizes the error may be assigned to each group.

The learning process of the second arithmetic unit 42 using the third learning image group and the fourth learning image group is also performed in a manner similar to the learning process of the first arithmetic unit 41 described above. When the learned second calculation unit 42 recognizes that the front work area is an area where the vehicle direction can be changed, the second calculation unit 42 sets a turning route for changing the vehicle direction, changes the direction, and starts the next operation. It functions to output second automatic travel information indicating the amount of steering for entering the target travel route.

[Another embodiment]
(1) In the embodiment described above, the automatic travel calculation unit 4 is a first calculation unit that is trained to output first automatic travel information for performing work travel in a straight line (including a gentle curved line). 41, and a second calculation unit 42 that is trained to output first automatic driving information for changing direction at an appropriate location. For field work vehicles, it would be advantageous if the vehicle also had a function to automatically avoid contact with objects (traveling obstacles) that would impede the vehicle's travel. FIG. 11 shows a block diagram of the automatic travel calculation section 4 to which a third calculation section 43 having a function of avoiding traveling obstacles is added. The third calculation section 43 includes a third front-stage calculation section 43a and a third rear-stage calculation section 43b. The third front-stage calculation unit 43a estimates the presence of a running obstacle from the field image (front image or bird's-eye view image) sent from the photographing unit 3. The third latter-stage calculation unit 43b generates third automatic driving information including avoidance driving information (steering amount, deceleration command, stop command, warning, etc.) for avoiding contact between the estimated driving obstacle and the vehicle body 1. Output. At this time, when avoiding a traveling obstacle by steering, the avoidance traveling route is set as the target traveling route. Note that field conditions such as the slope of the field and the unevenness of the field may also be taken into consideration when generating the third automatic travel information. The third front-stage calculation section 43a and the third rear-stage calculation section 43b can be integrated.

Since the third front-stage calculation unit 43a and the third rear-stage calculation unit 43b are configured with a convolutional neural network, learning processing is required. The learning images include field images in which running obstacles exist in the field (fifth learning image group), annotations indicating running obstacles that should be recognized in the field images, and driving to avoid running obstacles. This is a group of images (sixth learning image group) to which an annotation indicating . Note that separate learning images are prepared for moving obstacles (people, animals, etc.) and immobile obstacles (steel towers, etc.) so that they can be recognized separately as traveling obstacles. The learning process is performed according to the flowchart in FIG.

(2) In the embodiment described above, the first automated driving information, the second automated driving information, and the third automated driving information included the steering amount, but instead of this, the operation command such as the steering amount is automatically The steering amount may be generated by another calculation unit that derives the steering amount from the driving information. In this case, learning to estimate the steering amount from field information is not necessary.

(3) In the embodiment described above, a convolutional neural network is used as machine learning for constructing the automatic driving calculation unit 4, but other artificial neural networks may be used. Further, the first-stage calculation section and the second-stage calculation section may be constructed using the same type of machine learning, or may be constructed using different types of machine learning.

(4) When the learning processing unit 6 is provided in the control system of each field work vehicle and is connected to the automatic driving calculation unit 4, the photographed image acquired by the photographing unit 3 during manual driving is used as the learning image. It can be used as Annotations can be added by the driver. Furthermore, annotations may be automatically added based on information such as operations performed by the driver and travel trajectories.

(5) In the embodiment described above, the traveling direction of the vehicle body 1 is changed by adjusting the steering angle of the front wheels 11, which are steered wheels, based on the steering command. In a work vehicle equipped with a crawler-type traveling mechanism, instead, the traveling direction of the vehicle body 1 is changed by adjusting the speeds of the left and right crawlers based on a steering command.

(6) In the above-described embodiment, a tractor is used as the field work vehicle, but a similar automatic travel calculation unit 4 can be adopted for other field work vehicles such as a rice transplanter or a combine harvester.

The present invention is applicable to field work vehicles that automatically travel in fields.

3: Photographing unit 30: Image processing section 31: Front camera 32: Rear camera 33: Left camera 34: Right camera 4: Automatic driving calculation section 40: Selection section 41: First calculation section 41a: First stage calculation section 41b: First rear-stage calculation section 42 : Second calculation part 42a : Second former-stage calculation part 42b : Second latter-stage calculation part 43 : Third calculation part 43a : Third former-stage calculation part 43b : Third rear-stage calculation part 5 : Control unit 50 : Input/output signal processing unit 52 : Automatic driving control part 54 : Driving state detection part 55 : Working state detection part 6 : Learning processing unit

Claims (11)

A field work vehicle that automatically travels in the field,
a photographing unit configured to photograph the field including at least the area in which the vehicle body advances and output a field image;
a traveling device for adjusting vehicle speed and steering amount;
an automatic driving calculation unit that is trained to input the field image, estimate the boundary between the unworked area and the completed work area, and the field state, and output automatic driving information for automatic driving;
an automatic driving control unit that controls the traveling equipment based on the automatic driving information,
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a second learning image group created by adding annotations to the first learning image group, which is the field image, as training data. , the first learning image group is an image group including the unworked area and the already worked area of the field, and the second learning image group is an image group including the unworked area and the already worked area, which serve as a reference for the driving route. a group of images to which an annotation indicating the boundary between the already worked area and the boundary line determined from the boundary is added;
The automatic driving information is information for automatically driving the field work vehicle along the boundary. A field work vehicle that automatically travels in the field,
a photographing unit configured to photograph the field including at least the area in which the vehicle body advances and output a field image;
a traveling device for adjusting vehicle speed and steering amount;
an automatic driving calculation unit that is trained to input the field image, estimate the boundary between the unworked area and the completed work area, and the field state, and output automatic driving information for automatic driving;
an automatic driving control unit that controls the traveling equipment based on the automatic driving information,
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a second learning image group created by adding annotations to the first learning image group, which is the field image, as training data. , the first learning image group is an image group including the unworked area and the already worked area of the field, and the second learning image group is an image group including the boundary serving as a reference for the driving route and the boundary. A group of images to which an annotation indicating a boundary line determined from the boundary line and an annotation indicating a positional deviation when the boundary line is used as a target travel route are added,
The automatic driving information is a field work vehicle that is a positional deviation with respect to a driving route for automatically driving along the boundary. The positional deviation includes a lateral positional deviation that is a deviation in a transverse direction with respect to the driving route, and an azimuth deviation that is a deviation of the vehicle body orientation with respect to the direction of the driving route,
The field work vehicle according to claim 2, wherein the automatic driving information is the lateral positional deviation and the azimuth deviation. A field work vehicle that automatically travels in the field,
a photographing unit configured to photograph the field including at least the area in which the vehicle body advances and output a field image;
a traveling device for adjusting vehicle speed and steering amount;
an automatic driving calculation unit that is trained to input the field image, estimate the boundary between the unworked area and the completed work area, and the field state, and output automatic driving information for automatic driving;
an automatic driving control unit that controls the traveling equipment based on the automatic driving information,
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a second learning image group created by adding annotations to the first learning image group, which is the field image, as training data. , the first learning image group is an image group including the unworked area and the already worked area of the field, and the second learning image group is an image group including the unworked area and the already worked area. It is a group of images to which an annotation indicating a positional shift when the boundary line between the two is set as the target travel route and an annotation indicating the required steering amount,
The automatic driving information is a steering amount for eliminating the positional deviation of the field work vehicle. A field work vehicle that automatically travels in the field,
a photographing unit configured to photograph the field including at least the area in which the vehicle body advances and output a field image;
a traveling device for adjusting vehicle speed and steering amount;
an automatic driving calculation unit that is trained to input the field image, estimate the boundary between the unworked area and the completed work area, and the field state, and output automatic driving information for automatic driving;
an automatic driving control unit that controls the traveling equipment based on the automatic driving information,
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a fourth learning image created by adding annotations to the third learning image group, which is the field image, as training data. , the third learning image group is an image group including the unworked area and the already worked area of the field, and the fourth learning image is a group of images including the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field. An image with an annotation indicating the area,
The automatic driving information indicates a field work vehicle in which the vehicle body direction can be changed. A field work vehicle that automatically travels in the field,
a photographing unit configured to photograph the field including at least the area in which the vehicle body advances and output a field image;
a traveling device for adjusting vehicle speed and steering amount;
an automatic driving calculation unit that is trained to input the field image, estimate the boundary between the unworked area and the completed work area, and the field state, and output automatic driving information for automatic driving;
an automatic driving control unit that controls the traveling equipment based on the automatic driving information,
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a fourth learning image created by adding annotations to the third learning image group, which is the field image, as training data. , the third learning image group is an image group including the unworked area and the already worked area of the field, and the fourth learning image is a group of images including the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field. It is an image with an annotation indicating a turning route for changing the direction of the vehicle body in the area,
The automatic driving information is a field work vehicle that is a turning route for changing the vehicle direction in the area where the vehicle direction can be changed. A field work vehicle that automatically travels in the field,
a photographing unit configured to photograph the field including at least the area in which the vehicle body advances and output a field image;
a traveling device for adjusting vehicle speed and steering amount;
an automatic driving calculation unit that is trained to input the field image, estimate the boundary between the unworked area and the completed work area, and the field state, and output automatic driving information for automatic driving;
an automatic driving control unit that controls the traveling equipment based on the automatic driving information,
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a fourth learning image created by adding annotations to the third learning image group, which is the field image, as training data. , the third learning image group is an image group including the unworked area and the already worked area of the field, and the fourth learning image is a group of images including the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field, and the fourth learning image is an image group that includes the unworked area and the already worked area of the field. This is an image with an annotation indicating the amount of steering necessary to change the direction of the vehicle body in the area,
The automatic driving information is a field work vehicle, which is the amount of steering necessary for changing the direction of the vehicle body in the area where the vehicle direction can be changed. A field work vehicle that automatically travels in the field,
a photographing unit configured to photograph the field including at least the area in which the vehicle body advances and output a field image;
a traveling device for adjusting vehicle speed and steering amount;
an automatic driving calculation unit that is trained to input the field image, estimate the boundary between the unworked area and the completed work area, and the field state, and output automatic driving information for automatic driving;
an automatic driving control unit that controls the traveling equipment based on the automatic driving information,
The automatic driving calculation unit is constructed by a convolutional neural network, and is trained using a second learning image group created by adding annotations to the first learning image group, which is the field image, as training data. , the first learning image group is a group of images in which a running obstacle is present in the field, and the second learning image group is annotated with an annotation indicating running to avoid the running obstacle. A group of images,
The automatic driving information is avoidance driving information for avoiding contact between the traveling obstacle and the vehicle body. The field work vehicle according to claim 8, wherein the avoidance travel information includes information for avoiding the travel obstacle estimated to be a non-mobile obstacle. 9. The avoidance driving information includes information for performing at least one of deceleration, stopping, and warning in order to avoid contact with the driving obstacle estimated to be a moving obstacle. The field work vehicle described in 9. A running state detection unit is provided that detects a running state from the state of the running equipment,
Driving state information indicating the driving state is input to the automatic driving calculation unit, and the driving state information includes data indicating at least one of vehicle speed, shift position, engine rotation speed, and engine load factor;
The farmland according to any one of claims 8 to 10, wherein the avoidance driving information includes a control command for instructing a change in at least one of steering, vehicle speed, gear change, braking, and the engine rotation speed. work vehicle.

Images