JP2022106892A - Field work vehicle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a field work vehicle provided with an automatic traveling arithmetic unit learned to input a field image and output automatic traveling information for automatically traveling a field.

SOLUTION: The field work vehicle includes: an automatic traveling arithmetic unit 4 that is learned to input a field image taken by a photographing unit 3 and output automatic traveling information for automatically traveling a field; and an automatic traveling control unit 52 that controls a traveling device 91 based on the automatic traveling information, wherein the automatic traveling arithmetic unit 4 is constructed by a convolution neural network and is learned to use an image group for second learning as teacher data created by adding an annotation to an image group for first learning that are the field images.

SELECTED DRAWING: Figure 5

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a field work vehicle that performs work while automatically traveling in the field.
Regarding.

In one of the methods of automatically traveling in the field, a preset traveling route for automatic traveling and a work vehicle position obtained by satellite positioning are used. For example, in the tractor disclosed in Patent Document 1, a work target area to be worked by automatic running is formed inside a working area created by first manually running around the outer periphery of the field. .. The travel route for the tractor to travel in this work target area is calculated based on the route calculation algorithm. After that, the automatic driving control unit provided in the totactor calculates the positional deviation between the target traveling route and the vehicle position calculated from the satellite positioning data, and automatically steers the vehicle so that this positional deviation becomes small. Is generated.

In another method for automatically traveling in the field, the target traveling route of the field work vehicle obtained by image processing the image taken by the photographing camera mounted on the field work vehicle is used. For example, in the mowing machine disclosed in Patent Document 2, the boundary area between the mowed land and the uncut land in a lawn such as a golf course is photographed by a photographing device, and the photographed image shows the mowed land and the uncut land. The boundary line between the two is calculated, and the vehicle body is automatically steered along this boundary line.

JP-A-2018-116608 Japanese Unexamined Patent Publication No. 2018-07526

In order to carry out automatic traveling using satellite positioning data as disclosed in Patent Document 1, map information of the field (field coordinate system or earth coordinate system) is prepared, and using this map information, A travel route covering the field is calculated. The work vehicle is automatically driven so that the position of the own vehicle does not deviate from this travel route. The position of the own vehicle is calculated using satellite positioning, but in the calculation of the position of the own vehicle by satellite positioning, if the satellite radio wave is blocked by a forest or a house, there arises a problem that positioning is impossible or the positioning accuracy is lowered.

The automatic driving with the boundary line between the mowed land (worked land) and the uncut land (unworked land) detected by image processing as the target traveling route as disclosed in Patent Document 2 is It is an effective alternative to autonomous driving using satellite positioning. However, the field scene, which is the running surface of the field work vehicle, has more unevenness than the lawn, and the state also differs depending on the place, which leads to deterioration of steering accuracy. Further, when a crop or a residue of a crop is present in the field, the state of the crop varies depending on the location, so that automatic driving may be required according to the condition of the crop.

An object of the present invention is to provide a field work vehicle capable of automatically traveling without using satellite positioning and considering the state of the field.

The field work vehicle according to the present invention that automatically travels in the field includes a photographing unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image, a traveling device for adjusting the vehicle speed and the steering amount, and the above. The automatic running calculation unit learned to input the field image and output the automatic running information for automatic running based on the boundary between the unworked area and the existing work area and the field state, and the automatic running information. It is provided with an automatic traveling control unit that controls the traveling device based on the above.

In this configuration, the target traveling direction of the vehicle body is determined by estimating the boundary between the unworked area and the existing work area from the field image acquired by the photographing unit, and at the same time, the field state is estimated from the field image. Then, the target traveling behavior suitable for the field condition is determined, and the automatic traveling information is output from the target traveling direction and the traveling behavior. The automatic driving calculation unit is learned to output optimum automatic driving information including not only the boundary between the unworked area and the existing work area but also the field state. Therefore, when the traveling equipment is controlled along the boundary between the unworked area and the existing work area, the field state (state of the field scene, state of the crop, etc.) is also taken into consideration. For example, if there is mud or unevenness in front of the vehicle, it is possible to slow down the steering amount and suppress slippage.

In order to avoid work omissions or overlapping work (excessive overlap), it is necessary to accurately grasp the deviation from the target path of the vehicle body and obtain the steering amount to reduce the deviation. For this reason, it is important to find the boundary between the unworked area and the existing work area with a single line, that is, the boundary line, and to output a steering command that matches the steering reference point of the vehicle body with this boundary line. be. From this, in one of the preferred embodiments of the present invention, the automatic travel information estimates a linear target travel route corresponding to the boundary line indicating the boundary, and the linear target travel route and itself. Includes information to eliminate deviations from the car.

In fields such as tilling work, rice planting work, and harvesting work, the outer peripheral area that has already been worked is used as a turning area that changes the direction of the vehicle body, and is parallel to the inner area (unworked area) of the outer peripheral area. A traveling mode in which linear routes are connected by a direction-changing traveling such as a U-turn traveling is adopted. In order to adopt such a traveling pattern also in the automatic traveling according to the present invention, it is necessary to control the vehicle body to shift to the direction-changing traveling when the vehicle body reaches the outer peripheral region where the vehicle body needs to change the direction. From this, in one of the preferred embodiments of the present invention, the automatic traveling information changes the direction with the direction-changing traveling route set in the vehicle body direction-changeable region, which is the already working area, as the target traveling route. Contains information to do.

There may be people and animals, as well as running obstacles such as wells and steel towers in the field, so in order to perform automatic driving, detect these running obstacles and make contact with the running obstacles. Need to avoid. Therefore, in one of the preferred embodiments of the present invention, the automatic driving calculation unit uses collision avoidance information for avoiding contact with a traveling obstacle estimated from the field image as the automatic driving information. Output.

Countermeasures differ depending on whether the running obstacle is moving or fixed in the field. Work vehicles need to avoid obstacles that are fixed to the field. Therefore, in one of the preferred embodiments of the present invention, the collision avoidance information includes avoidance traveling information for avoiding the traveling obstacle presumed to be a non-movable obstacle.

Obstacles moving in the field may move away from the destination of the work vehicle over time and become no longer a running obstacle. It is also possible to eliminate obstacles by an alarm or the like. Therefore, in one of the preferred embodiments of the present invention, the collision avoidance information includes at least deceleration, stoppage, and warning for avoiding contact with the traveling obstacle presumed to be a mobile obstacle. Contains information to do one.

When field work is performed by manual driving, the driver considers the current vehicle speed, engine speed, and the like, and appropriately changes and operates them to achieve optimum work driving. Such an operation is also useful in automatic driving. Therefore, in one of the preferred embodiments of the present invention, a traveling state detection unit that detects a traveling state from the state of the traveling device is provided, and the traveling state information indicating the traveling state is transmitted to the automatic traveling calculation unit. The input state information includes data indicating at least one of vehicle speed, shift position, engine speed, and engine load factor, and the automatic travel information includes steering, vehicle speed, shift, braking, and so on. A control command is included that commands a change in at least one of the engine speeds.

In agriculture, which has been developed through many years of experience of farmers and technological reform of field work vehicles, it is necessary to reflect many years of experience in the automatic running control of field work vehicles in order to promote automation of field work vehicles. .. For this purpose, it is preferable to introduce machine learning at the core of the control system so that the field work vehicle learns the experience of the agricultural worker. As a result, the ability of a skilled farmer can be realized on a computer mounted on a field work vehicle. Therefore, the automatic traveling calculation unit is constructed by a convolution neural network, and is trained using the first learning image group and the second learning image group as the learning image, and the first learning image group. Is an image group including the unworked area and the worked area of the field, and the second learning image group includes an annotation indicating the boundary between the unworked area and the worked area. It is a group of images to which an annotation indicating a positional deviation with respect to a traveling path for automatic traveling along the boundary is added. Annotation is a meaning for a corresponding photographed image. For example, in a photographed image of a field, pointing to a boundary between an unworked area and an existing working area, and further pointing to a position deviation from the boundary with respect to the traveling path of the vehicle body and an operation for eliminating the position deviation are annotations. Is. This machine learning includes a first learning and a second learning. In the first learning, the raw photographed image acquired from the viewpoint from the field work vehicle, the boundary between the unworked area and the already worked area in the photographed image are found, and the boundary area is shown as an annotation. It is a learning to estimate the boundary area from the raw photographed image by using the annotation image. The second learning is an annotation that shows the raw captured image and the operation to eliminate the position deviation of the vehicle body from the found boundary position (deviation from the target running line) and the required position deviation as annotations. This is learning to estimate the displacement boundary region of the vehicle body from a raw photographed image using an image, and to estimate an operation for eliminating the displacement. By this learning, the automatic traveling calculation unit can output the position of the boundary line in front of the vehicle body, the position deviation of the vehicle body, and, if necessary, the operation for eliminating the position deviation from the acquired field image.

When the vehicle body is displaced, steering is performed to match the vehicle body with the target traveling path. In order to eliminate the misalignment due to steering quickly and efficiently, control is performed by combining the lateral misalignment, which is the misalignment in the transverse direction with respect to the traveling path, and the azimuth deviation, which is the displacement of the vehicle body with respect to the direction of the traveling path. It is preferable to be done. This is because the field work vehicle traveling in the field has more slippage and the vehicle speed is lower than that of the vehicle traveling on the road. From this, in one of the preferred embodiments of the present invention, the misalignment includes a lateral position deviation, which is a deviation in the transverse direction with respect to the traveling path, and an orientation, which is a deviation in the vehicle body orientation with respect to the direction of the traveling path. The deviation is included.

Further, in one of the preferred embodiments of the present invention, a third learning image and a fourth learning image are further used as the learning image, and the third learning image is the already working area. The fourth learning image is an image group to which an annotation indicating the vehicle body direction changeable region is added. In the automatic travel calculation unit learned using such a learning image, a direction change area for changing the direction of the vehicle body from the current work travel path and shifting to the next work travel route, and a direction change area thereof. Information for turning (turning) in the field can be estimated from the field image.

Further, in one of the preferred embodiments of the present invention, the fifth learning image and the sixth learning image are further used as the learning image, and the fifth learning image is a running image in the field. It is an image group including an object that becomes an obstacle, and the learning image is an image group to which an annotation indicating the object that becomes an obstacle is added. In the automatic driving calculation unit learned using such a learning image, can estimate an object that becomes a traveling obstacle from the field image, and the automatic driving information necessary for avoiding contact with the obstacle. Can be output.

It is a side view of the tractor which is an example of a field work vehicle. It is the schematic plan view of the tractor for explaining the mounting position of a photographing camera. It is explanatory drawing explaining the flow of the field work by a tractor. It is a functional block diagram of a control system. It is a functional block diagram of the automatic driving calculation unit. It is a schematic diagram which shows the learning image used for the learning process of the 1st calculation part. It is a schematic diagram which shows the learning image used for the learning process of the 1st calculation part. It is a schematic diagram which shows the learning image used for the learning process of the 2nd calculation part. It is a schematic diagram which shows the learning image used for the learning process of the 2nd calculation part. It is a flowchart which shows an example of the learning process of the automatic driving calculation unit. It is a functional block diagram in another embodiment of the automatic driving calculation unit.

Next, a field work vehicle that automatically travels in the field according to the present invention will be described with reference to the drawings. In the embodiment shown in FIG. 1, the work vehicle is a tractor that automatically travels in the field. The tractor is provided with a driving unit 20 at the center of the vehicle body 1 supported by the front wheels 11 and the rear wheels 12. The rear part of the vehicle body 1 is equipped with a working device 24 which is a rotary tilling device via a hydraulic elevating mechanism 23. The front wheel 11 functions as a steering wheel, and the traveling direction of the tractor is changed by changing the steering angle thereof. The steering angle of the front wheels 11 is changed by the operation of the steering mechanism 13. The steering mechanism 13 includes a steering motor 14 for automatic steering. During manual driving, the front wheels 11 are steered by operating the steering wheel 22 arranged in the driving unit 20.

The tractor is provided with a photographing unit 3 that photographs a field including at least a traveling destination area of the vehicle body 1 and outputs a field image. In this embodiment, as shown in FIG. 2, the photographing unit 3 includes a front camera 31, a rear camera 32, a left camera 33, and a right camera 34.
The front camera 31 is arranged at the front portion of the vehicle body 1 and photographs the front of the vehicle body 1. The rear camera 32 is arranged at the rear of the vehicle body 1 and photographs the rear of the vehicle body 1 including the working device 24. The left camera 33 and the right camera 34 are arranged on the side of the vehicle body 1, respectively, and photograph the left side and the right side of the vehicle body 1. By converting the viewpoints of the images taken from the front camera 31, the rear camera 32, the left camera 33, and the right camera 34 and synthesizing them, it is possible to generate a bird's-eye view image with the upper part of the tractor as the viewpoint. As the field image, a front image taken by the front camera 31 and / or a field image is used. The forward captured image is used as a field image including the vehicle body traveling region in the case of forward movement, and the rearward captured image taken by the rear camera 32 is used as a field image including the vehicle body traveling region in the case of reverse movement. Therefore, it is assumed that the field image described below includes a front image, a rear image, and a bird's-eye view image.

A general procedure for field work with a tractor is shown in FIG. In this example, when the tractor enters the field (# a), the surrounding work running along the outer circumference of the field is performed manually or automatically, and the outer peripheral area, which is the existing work area, is formed on the outermost peripheral side of the field. Is done (# b). In the surrounding work running, the manual running is preferable because the turning run in the corner region is complicated. Since it is substantially straight running except for the corner area, automatic running is used. At that time, the boundary line between the field scene and the shore or ridge, which is the outer peripheral line of the field, is recognized in real time by an image recognition program using machine learning, and the work end of the work device 24 is recognized at the recognized boundary line. The tractor is automatically steered so that When the outer peripheral region formed by the surrounding work travel becomes large enough to enable the direction change (U-turn travel) of the tractor, the automatic work travel to the unworked region is started. This automatic work run includes a forward work run (# c) in which the tractor is made to work in a straight line, and a vehicle body direction change run (# c) in which the work device 24 is raised in the existing work area to make a U-turn of the tractor in a non-working state. d) and are included. By repeating the forward work run and the vehicle body direction change run (#e), the work of the entire field is completed. In the forward work run, the image recognition target is the boundary line between the unworked area and the existing work area, and in the vehicle body direction change run, the image recognition target is the boundary line (outer circumference line) of the field and the unworked area and the existing work. It is the boundary line with the area. Therefore, it is preferable that different image recognition programs are used for the forward work run and the vehicle body direction change run. In that case, the automatic switching between the forward work travel image recognition program and the vehicle body direction change travel image recognition program is performed based on the state of the work device 24 (for example, the descending state and the ascending state, the driven state and the non-driving state). be able to. Depending on the type of field, the shape of the outer peripheral region is assumed in advance, the outer peripheral region remains unworked, and # c, # d, and # e come first with respect to the unworked region inside the outer peripheral region. Is carried out, and then # b is carried out.

FIG. 4 shows a control system built on this tractor. The control system of this embodiment includes a control unit 5 including an in-vehicle ECU group and an input / output signal processing unit 50. The control unit 5 is provided with an automatic traveling calculation unit 4 constructed by applying machine learning, in addition to a functional unit that executes normal work traveling. The input / output signal processing unit 50 is constructed with a functional unit that processes input / output signals handled during the work running of the tractor. The control unit 5 and the input / output signal processing unit 50 are connected by an in-vehicle LAN.

A traveling device group 91, a working device group 92, a notification device group 93, an automatic / manual switching operation tool 94, a traveling state detection sensor group 81, and a working state detection sensor group 82 are connected to the input / output signal processing unit 50. .. The traveling device group 91 includes a steering motor 14 (see FIG. 1) as a steering adjusting device, a speed change adjusting device as a vehicle speed adjusting device, an engine rotation adjusting device, and the like. The work equipment group 92 includes a control device for driving the work device 24 and the elevating mechanism 23 (see FIG. 1). The notification device group 93 includes a display, a speaker, a lamp, a buzzer, and the like. The automatic / manual switching operation tool 94 is a switch for selecting either an automatic traveling mode in which the vehicle travels by automatic steering or a manual steering mode in which the vehicle travels by manual steering.

The traveling state detection sensor group 81 includes a sensor that detects a traveling state such as a steering amount, an engine speed, and a shifting position. The work state detection sensor group 82 includes sensors that detect the posture and drive state of the work device 24.

In addition, a photographing unit 3 is connected to the input / output signal processing unit 50. The photographing unit 3 includes an image processing unit 30 that processes images captured by the four cameras 31 to 34 described above to generate a field image suitable for processing by the control unit 5. The field image output from the photographing unit 3 is input to the control unit 5 via the input / output signal processing unit 50.

The control unit 5 includes a manual travel control unit 51, an automatic travel control unit 52, a work control unit 53, a travel state detection unit 54, a work state detection unit 55, and an automatic travel calculation unit 4. The manual travel control unit 51 controls the travel equipment group 91 based on the operation by the driver in the manual travel mode. In the automatic driving mode, the automatic driving control unit 52 controls the traveling equipment group 91 based on the automatic driving information output from the automatic driving calculation unit 4. The work control unit 53 manually or automatically controls the work equipment group 92. The traveling state detection unit 54 detects the traveling state based on the detection signal from the traveling state detection sensor group 81. The work state detection unit 55 detects the work state based on the detection signal from the work state detection sensor group 82.

The automatic driving calculation unit 4 inputs the field image in the learned state, estimates the boundary between the unworked area and the worked area and the field state, and outputs the automatic driving information for automatic driving. .. In this embodiment, as shown in FIG. 5, the automatic driving calculation unit 4 includes a first calculation unit 41, a second calculation unit 42, and a selection unit 40. The first calculation unit 41 includes a first first stage calculation unit 41a and a first second stage calculation unit 41b. The second calculation unit 42 includes a second front-stage calculation unit 42a and a second second-stage calculation unit 42b.

The first first-stage calculation unit 41a estimates from the field image the boundary line indicating the boundary between the unworked area and the worked area and the field state. The first second-stage calculation unit 41b calculates a linear target travel route (forward work travel route) corresponding to the estimated boundary line in real time. At that time, since the work width of the work device 24, the amount of work overtap, the distance from the vehicle body reference point to the work end of the work device 24, and the like are required, these work device specification data are set in advance. Further, the first front-stage calculation unit 41a calculates the positional deviation of the steering reference point of the vehicle body 1 with respect to the target traveling route (forward work traveling route), and the first automatic traveling information is information for eliminating this positional deviation. To generate. This misalignment includes a lateral misalignment, which is a lateral displacement with respect to the target travel path, and an azimuth deviation, which is an angular deviation between the orientation of the target travel path and the vehicle body orientation. The first automatic driving information includes a steering amount for eliminating the misalignment. When generating the steering amount, the field conditions such as the slope of the field and the unevenness of the field scene are taken into consideration. The first first-stage calculation unit 41a and the first second-stage calculation unit 41b can be integrated.

The second first-stage calculation unit 42a estimates the vehicle body direction changeable region, which is the work area, from the field image. The second rear-stage calculation unit 42b sets a direction-changing travel path set in the vehicle body direction changeable region, and sets a difference from the actual direction-change trajectory with respect to the set direction-change travel path as a turning deviation, and this turning deviation is defined as a turning deviation. The second automatic driving information, which is the information (steering amount) for eliminating the above, is generated. The field conditions such as the slope of the field and the unevenness of the field scene are also taken into consideration when generating the second automatic driving information. The second front-stage calculation unit 42a and the second rear-stage calculation unit 42b can be integrated.

The selection unit 40 compares the reliability of the first automatic driving information with the reliability of the second automatic driving information, selects the information with the higher appropriateness as the driving control that must be performed at the present time, and automatically travels. It is output as information and given to the automatic driving control unit 52. Alternatively, the selection unit 40 may select either the first automatic driving information or the second automatic driving information based on the work state detected by the work state detection unit 55. For example, if the state of the work device 24 is a lowered state, a driven state, or both, it is regarded as a forward work run, and the first automatic running information is selected. If the state of the working device 24 is an ascending state, a non-driving state, or both, it is regarded as a turning trip, and the second automatic running information is selected. As yet another method, the use of either the first pre-stage calculation unit 41a or the second pre-stage calculation unit 42a may be determined based on the work state detected by the work state detection unit 55.

In the automatic traveling calculation unit 4 shown in FIG. 5, as an additional function, the traveling state information detected by the traveling state detection unit 54 is input to the first calculation unit 41 and the second calculation unit 42. There is. The running state information includes vehicle speed, shift position, engine speed, engine load factor, and the like. When this function is added, the first rear-stage calculation unit 41b and the second rear-stage calculation unit 42b add vehicle speed control commands to the first automatic driving information and the second automatic driving information in addition to the information for eliminating the misalignment. It is also possible to include at least one of a shift control command, a braking control command, and an engine control command. This is because not only adjusting the steering amount but also adjusting other traveling conditions (vehicle speed, etc.) may be effective in eliminating the misalignment.

The first calculation unit 41 is a machine learning control unit that outputs the first automatic running information from the field image, and the second calculation unit 42 is a machine learning control unit that outputs the second automatic running information from the field image. In order for the machine learning control unit to estimate the situation of field work from the field image and output appropriate automatic driving information, learning processing is required. For this learning process, in FIG. 5, the learning process unit 6 is connected to the automatic traveling calculation unit 4. The learning processing unit 6 may be built in the automatic traveling calculation unit 4. Further, a learning system may be adopted in which the learning processing unit 6 built in the computer system other than the control system of the tractor and the automatic traveling calculation unit 4 exchange data and execute the learning processing. In this embodiment, the first arithmetic unit 41 and the second arithmetic unit 42 are constructed by a convolutional neural network, and the learning process suitable for the convolutional neural network is performed by the learning processing unit 6.

Since the configuration of the learning processing unit 6 is known, detailed description thereof will be omitted here, but it has an error calculation function and a weight adjustment function. The learning processing unit 6 may include a convolutional neural network similar to the first arithmetic unit 41 or the second arithmetic unit 42, or may be directly connected to the first arithmetic unit 41 or the second arithmetic unit 42 before learning. , May be sequentially learned. In the former case, the learned data such as the weighting coefficient is set in the first calculation unit 41 and the second calculation unit 42 after the learning process is completed.

In the learning process of the first calculation unit 41, a first learning image group composed of a large number of first learning images and a second learning image group composed of a large number of second learning images are used.

The first learning image group is an image group including an unworked area and an existing work area of the field, and in this embodiment, a bird's-eye view image generated by the image processing unit 30 of the photographing unit 3 is used. The second learning image group is an image group to which an annotation indicating a feature to be recognized in the first learning image group is added. An example of the first learning image is shown on the left side of FIGS. 6 and 7, and the second learning image group is shown on the right side of FIGS. 6 and 7. In the first learning image of FIG. 6, a scene spreading in the unworked area is shown in front of the tractor located in the worked area. In the second learning image of FIG. 6, the boundary between the unworked area and the existing work area, which is the reference of the traveling route, and the annotation indicating the boundary line obtained from the boundary, the boundary, which is the starting reference of the working traveling, and the said. Annotation indicating the boundary line obtained from the boundary, annotation indicating the positional deviation when the boundary line is set as the target traveling path, annotation indicating that the surface state of the unworked area is moderately uneven, and the required steering amount. Annotation indicating that is attached. The first learning image of FIG. 7 is a bird's-eye view image of a tractor that has entered an unworked area and is working along a boundary between a work area and an existing work area. In this second learning image, an annotation indicating a boundary that serves as a reference for a traveling route (forward and backward) and a boundary line obtained from the boundary, an annotation indicating a positional deviation when the boundary line is used as a target traveling route, and an annotation indicating a positional deviation are not yet available. An annotation indicating that the surface condition of the work area is moderately uneven and an annotation indicating the required steering amount are added.

In the learning process of the second calculation unit 42, a third learning image and a fourth learning image are used as learning images. The third learning image group is also an image group including an unworked area and an existing work area of the field, but the existing work area in front of the vehicle body 1 is an area in which the vehicle body direction can be changed, and the fourth learning image group. Is an image group to which an annotation indicating a vehicle body direction changeable region to be recognized in the third learning image group is added. An example of the third learning image is shown on the left side of FIGS. 8 and 9, and the fourth learning image group is shown on the right side of FIGS. 8 and 9. In the third learning image of FIG. 8, a scene spreading in the existing work area is shown in front of the tractor during the work running. In the fourth learning image of FIG. 8, an annotation indicating that the existing work area extending forward is a vehicle body direction changeable area and a surface state of the vehicle body direction changeable area are moderately uneven. Annotations and annotations indicating the required steering amount are added. Further, the fourth learning image shows a turning travel path for changing the vehicle body direction set in the vehicle body direction changeable region when the vehicle body direction changeable region is recognized. The third learning image of FIG. 9 is a bird's-eye view image near the end of the turning run for changing the direction of the vehicle body. In the fourth learning image of FIG. 9, an annotation indicating a boundary that serves as a reference for a traveling route and a boundary line obtained from the boundary, an annotation indicating a positional deviation when the boundary line is used as a target traveling route, an existing work area, and an annotation indicating that the boundary line is used as a target traveling route. An annotation indicating that the surface state of the unworked area is moderately uneven and an annotation indicating the required steering amount are added.

FIG. 10 shows a flow of learning processing for causing the first calculation unit 41 to function to output the first automatic running information indicating the steering amount by using the first learning image group and the second learning image group. It is shown.

First, the first learning image group is acquired by using the photographed image acquired by the test run of the tractor or the drone with a camera, or the CG device (# 01). An annotation indicating a notable feature region is added to the first learning image group to create a second learning image group (# 02).

The second learning image is sequentially extracted from the second learning image group and input to the convolutional neural network constituting the first calculation unit 41 (# 11). Using the second learning image, the automatic driving information estimated by the convolutional neural network is output (# 21).
The difference (error) between the estimated automatic driving information and the automatic driving information indicated by the annotation attached to the second learning image is calculated (# 22). If the calculated error is not less than the minimum value (# 23No branch), the weighting coefficient is corrected (# 24), the process returns to step # 21, and the process of reducing the error is repeated. If the calculated error is less than the minimum value (# 23Yes branch), the weighting coefficient at that time is recorded (# 25).

It is checked whether or not the learning process using all the prepared second learning images is completed (# 26). If the second learning image used for the learning process still remains (# 26Yes branch), the process returns to step # 11 and the learning process is continued. If there is no second learning image used for the learning process (# 26No branch), the final weighting factor is determined based on the recorded weighting factor (# 31), and the weighting factor is a convolutional neural network. It is set in the first calculation unit 41 which is a network (# 32). Similar images may be grouped from the second learning image group, and a weighting coefficient that minimizes the error may be assigned to each group.

The learning process of the second calculation unit 42 using the third learning image group and the fourth learning image group is also performed by a method similar to the learning process of the first calculation unit 41 described above. When the learned second calculation unit 42 recognizes that the existing work area in front is an area where the vehicle body direction can be changed, the learned second calculation unit 42 sets a turning travel path for changing the vehicle body direction, changes the direction, and then changes the direction to the next. It functions to output the second automatic driving information indicating the steering amount for entering the target traveling path.

[Another Embodiment]
(1) In the above-described embodiment, the automatic driving calculation unit 4 is a first calculation unit learned to output the first automatic driving information for performing work traveling in a straight line (including a gently curved line). It was provided with 41 and a second calculation unit 42 learned to output the first automatic driving information for performing a direction-changing driving at an appropriate place. In the field work vehicle, it would be convenient if there is a function of automatically avoiding contact with an object (running obstacle) that hinders running. FIG. 11 shows a block diagram of the automatic traveling calculation unit 4 to which the third calculation unit 43 having a function of avoiding a traveling obstacle is added. The third calculation unit 43 includes a third front-stage calculation unit 43a and a third rear-stage calculation unit 43b. The third first-stage calculation unit 43a estimates the presence of a traveling obstacle from the field image (front image or bird's-eye view image) sent from the photographing unit 3. Third rear stage calculation unit 43b Generates third automatic driving information including avoidance driving information (steering amount, deceleration command, stop command, warning, etc.) for avoiding contact between the estimated running obstacle and the vehicle body 1. Output. At that time, when avoiding a traveling obstacle by steering, the avoiding traveling route is set as the target traveling route. It should be noted that the field condition such as the slope of the field and the unevenness of the field scene may be taken into consideration when generating the third automatic traveling information. The third front-stage calculation unit 43a and the third rear-stage calculation unit 43b can be integrated.

Since the third first-stage calculation unit 43a and the third second-stage calculation unit 43b are composed of a convolutional neural network, learning processing is required. The learning image includes a field image (fifth learning image group) in which a running obstacle exists in the field, an annotation indicating a running obstacle to be recognized in the field image, and running for avoiding the running obstacle. This is an image group (sixth learning image group) to which an annotation indicating the above is added. Separate learning images are prepared so that mobile obstacles (humans and animals) and non-mobile obstacles (steel towers, etc.) can be recognized as traveling obstacles. The learning process is performed according to the flowchart of FIG.

(2) In the above-described embodiment, the steering amount is included in the first automatic driving information, the second automatic driving information, and the third automatic driving information, but instead, the operation command such as the steering amount is automatically. It may be generated by another calculation unit that derives the steering amount from the traveling information. In this case, learning to estimate the steering amount from the field information becomes unnecessary.

(3) In the above-described embodiment, the convolutional neural network is used as machine learning for constructing the automatic driving calculation unit 4, but other artificial neural networks may be used. Further, the first-stage calculation unit and the second-stage calculation unit may be constructed by the same type of machine learning, or may be constructed by different types of machine learning.

(4) When the learning processing unit 6 is provided in the control system of each field work vehicle and is connected to the automatic driving calculation unit 4, the captured image acquired by the photographing unit 3 during manual driving is used as a learning image. Can be used as. Annotation can be added by the driver.
Further, annotation may be automatically added from information such as an operation by the driver and a traveling locus.

(5) In the above-described embodiment, the traveling direction of the vehicle body 1 is changed by adjusting the steering angle of the front wheels 11 which are the steering wheels based on the steering command. In the work vehicle provided with the crawler type traveling mechanism, the traveling direction of the vehicle body 1 is changed by adjusting the speeds of the left and right crawlers based on the steering command instead.

(6) In the above-described embodiment, the tractor is treated as the field work vehicle, but the same automatic traveling calculation unit 4 can be adopted for other field work vehicles such as rice transplanters and combines.

The present invention is applicable to a field work vehicle that automatically travels in a field.

3: Shooting unit 30: Image processing unit 31: Front camera 32: Rear camera 33: Left camera 34: Right camera 4: Automatic running calculation unit 40: Selection unit 41: First calculation unit 41a: First front stage calculation unit 41b: 1st second stage calculation unit 42: 2nd calculation unit 42a: 2nd front stage calculation unit 42b: 2nd second stage calculation unit 43: 3rd calculation unit 43a: 3rd first stage calculation unit 43b: 3rd second stage calculation unit 5: control unit 50 : Input / output signal processing unit 52: Automatic running control unit 54: Running state detection unit 55: Working state detection unit 6: Learning processing unit

Claims (11)

It is a field work vehicle that automatically travels in the field.
An imaging unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image.
Traveling equipment for adjusting vehicle speed and steering amount,
An automatic driving calculation unit learned to input the field image, estimate the boundary between the unworked area and the existing working area, and the field state, and output the automatic driving information for automatic driving.
An automatic driving control unit that controls the traveling equipment based on the automatic driving information is provided.
The automatic running calculation unit is constructed by a convolutional neural network, and is trained using the second learning image group created by adding an annotation to the first learning image group which is the field image as teacher data. The first learning image group is an image group including the unworked area and the worked area of the field, and the second learning image group includes the unworked area and the worked area. It is a group of images to which an annotation indicating the boundary between them is added.
The automatic traveling information is a field work vehicle which is information for automatically traveling along the boundary. It is a field work vehicle that automatically travels in the field.
An imaging unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image.
Traveling equipment for adjusting vehicle speed and steering amount,
An automatic driving calculation unit learned to input the field image, estimate the boundary between the unworked area and the existing working area, and the field state, and output the automatic driving information for automatic driving.
An automatic driving control unit that controls the traveling equipment based on the automatic driving information is provided.
The automatic running calculation unit is constructed by a convolutional neural network, and is trained using the second learning image group created by adding an annotation to the first learning image group which is the field image as teacher data. The first learning image group is an image group including the unworked area and the worked area of the field, and the second learning image group includes the unworked area and the worked area. It is a group of images to which an annotation indicating a positional deviation with respect to a traveling path for automatic traveling along the boundary between the images is added.
The automatic traveling information is a field work vehicle which is a positional deviation with respect to a traveling route for automatic traveling along the boundary. The misalignment includes a lateral misalignment, which is a cross-sectional deviation with respect to the traveling path, and an azimuth deviation, which is a vehicle body orientation deviation with respect to the direction of the traveling path.
The field work vehicle according to claim 2, wherein the automatic traveling information is the lateral position deviation and the orientation deviation. It is a field work vehicle that automatically travels in the field.
An imaging unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image.
Traveling equipment for adjusting vehicle speed and steering amount,
An automatic driving calculation unit learned to input the field image, estimate the boundary between the unworked area and the existing working area, and the field state, and output the automatic driving information for automatic driving.
An automatic driving control unit that controls the traveling equipment based on the automatic driving information is provided.
The automatic running calculation unit is constructed by a convolutional neural network, and is trained using the second learning image group created by adding an annotation to the first learning image group which is the field image as teacher data. The first learning image group is an image group including the unworked area and the already worked area of the field, and the second learning image group is a running for automatically traveling along the boundary. It is a group of images to which an annotation indicating the steering amount for eliminating the misalignment with respect to the path is added.
The automatic traveling information is a field work vehicle which is a steering amount for eliminating the misalignment. It is a field work vehicle that automatically travels in the field.
An imaging unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image.
Traveling equipment for adjusting vehicle speed and steering amount,
An automatic driving calculation unit learned to input the field image, estimate the boundary between the unworked area and the existing working area, and the field state, and output the automatic driving information for automatic driving.
An automatic driving control unit that controls the traveling equipment based on the automatic driving information is provided.
The automatic running calculation unit is constructed by a convolutional neural network, and is trained using the fourth learning image group created by adding an annotation to the third learning image group which is the field image as teacher data. The third learning image group is an image group including the unworked area and the already worked area of the field, and the fourth learning image group can change the vehicle body direction in the already working area. A group of images with an annotation indicating the area.
The automatic traveling information is a field work vehicle which is an area where the vehicle body direction can be changed. It is a field work vehicle that automatically travels in the field.
An imaging unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image.
Traveling equipment for adjusting vehicle speed and steering amount,
An automatic driving calculation unit learned to input the field image, estimate the boundary between the unworked area and the existing working area, and the field state, and output the automatic driving information for automatic driving.
An automatic driving control unit that controls the traveling equipment based on the automatic driving information is provided.
The automatic running calculation unit is constructed by a convolutional neural network, and is trained using the fourth learning image group created by adding an annotation to the third learning image group which is the field image as teacher data. The third learning image group is an image group including the unworked area and the already worked area of the field, and the fourth learning image group can change the vehicle body direction in the already working area. It is a group of images with an annotation indicating a turning route for changing the direction of the vehicle body in the area.
The automatic traveling information is a field work vehicle which is a turning traveling route for changing the vehicle body direction in the area where the vehicle body direction can be changed. It is a field work vehicle that automatically travels in the field.
An imaging unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image.
Traveling equipment for adjusting vehicle speed and steering amount,
An automatic driving calculation unit learned to input the field image, estimate the boundary between the unworked area and the existing working area, and the field state, and output the automatic driving information for automatic driving.
An automatic driving control unit that controls the traveling equipment based on the automatic driving information is provided.
The automatic running calculation unit is constructed by a convolutional neural network, and is trained using the fourth learning image group created by adding an annotation to the third learning image group which is the field image as teacher data. The third learning image group is an image group including the unworked area and the already worked area of the field, and the fourth learning image group can change the vehicle body direction in the already working area. It is a group of images with an annotation indicating the amount of steering required for changing the direction of the vehicle body in the area.
The automatic traveling information is a field work vehicle which is a steering amount required for changing the direction of the vehicle body in the region where the direction of the vehicle body can be changed. It is a field work vehicle that automatically travels in the field.
An imaging unit that photographs the field including at least the traveling region of the vehicle body and outputs a field image.
Traveling equipment for adjusting vehicle speed and steering amount,
An automatic driving calculation unit learned to input the field image, estimate the boundary between the unworked area and the existing working area, and the field state, and output the automatic driving information for automatic driving.
An automatic driving control unit that controls the traveling equipment based on the automatic driving information is provided.
The automatic running calculation unit is constructed by a convolutional neural network, and is trained using the second learning image group created by adding an annotation to the first learning image group which is the field image as teacher data. The first learning image group is an image group of the field, and the second learning image group is given an annotation indicating a running obstacle in the field and an avoidance running route for avoiding the running obstacle. It is a group of images
The automatic traveling information is a field work vehicle which is avoidance traveling information for avoiding contact between the traveling obstacle and the vehicle body. The field work vehicle according to claim 8, wherein the avoidance traveling information includes information for avoiding the traveling obstacle presumed to be a non-movable obstacle. The avoidance driving information includes information for performing at least one of deceleration, stopping, and warning for avoiding contact with the traveling obstacle presumed to be a mobile obstacle. The field work vehicle according to 9. A traveling state detection unit that detects a traveling state from the state of the traveling device is provided.
The running state information indicating the running state is input to the automatic running calculation unit, and the running state information includes data indicating at least one of the vehicle speed, the shift position, the engine speed, and the engine load factor.
The field according to any one of claims 8 to 10, wherein the avoidance running information includes a control command for commanding steering, vehicle speed, shifting, braking, and changing at least one of the engine speeds. Work vehicle.

Images