JP2019004772A - Harvesting machine - Google Patents

Abstract

To provide a harvesting machine capable of efficiently performing a harvesting work support using an image imaged by an imaging part provided on a vehicle body part.SOLUTION: A harvesting machine comprises: a machine body position calculation part 66 for calculating a machine body position which is a map coordinate of a machine body based on positioning data from a satellite positioning module 80; an image recognition module 5 into which image data of an image being acquired continuously by an imaging part 70 which images a field in a harvesting work time, is input, which estimates an existence area where there is a recognition object in the image, and outputs recognition output data including the existence area and estimation probability when the existence area is estimated; and a recognition object position information generation part 51 for generating recognition object position information for indicating a position of the recognition object on the map from the machine body and the recognition output data when the image is acquired.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a harvester that harvests crops while traveling in a field and can support harvesting work based on a photographed image acquired by a photographing unit.

  In the harvesting work of a crop using a harvester in a field, it is necessary to pay attention to the growth state of locally different crops and the presence of obstacles including people. For this reason, for example, the combine according to Patent Document 1 includes a television camera and an image processing device that captures the cereals in front of the harvesting unit. The image processing device detects a specific state of the cereal by comparing an image from the television camera with an image indicating the planting state of various varieties stored in advance. For example, when the image processing apparatus detects that a portion of the grain ridge in front of the cutting part is lying down, the take-up reel is tilted with the grain fold down side down. Thereby, the cutting performance of the overturned cereal is improved. Moreover, in the combine by patent document 2 with which the television camera and the image processing apparatus are provided similarly to the combine of patent document 1, when it is detected that the grain cereal does not exist ahead of a cutting part, a combine is a headland. The power transmission to the cutter is cut off, the cutter is raised, and the traveling speed is reduced. Thereby, the collision of the cutter with the scissors or the like during the turning in the headland is avoided, and the heading of the combine headland is smoothly performed.

JP-A-11-155340 Japanese Patent Laid-Open No. 11-137062

  In the combine according to Patent Literature 1 and Patent Literature 2, the state of the cereals in front of the harvesting unit is detected using an image processing technique, and the operation of the work equipment is controlled based on the detection result. However, the actual position of the detected cereal field on the map is not considered. For this reason, even if the culm in a state that causes a problem in work traveling is detected and avoidance control is performed to solve the problem, the region indicating the culm state in question and the combine are far apart. In such a case, there arises a problem that the avoidance control is too early or too late. It is also difficult to appropriately determine the timing at which avoidance control should be terminated.

  In view of the above situation, there is a demand for a harvesting machine that effectively performs harvesting work support using a photographed image by a photographing unit provided on a vehicle body.

  A feature of the present invention is a harvesting machine that harvests crops while traveling in a field, and a machine body position calculation unit that calculates a machine body position that is map coordinates of the machine body based on positioning data from a satellite positioning module; An image capturing unit that captures a field during harvesting operation and image data of a captured image sequentially acquired by the image capturing unit sequentially are input, and an existing region where a recognition target object exists in the captured image An image recognition module that estimates and outputs recognition output data including the existence area and an estimated probability when the existence area is estimated, and the aircraft position and the recognition output data at the time when the captured image is acquired. A recognition object position information generation unit that generates recognition object position information indicating the position of the recognition object on the map.

  In the present invention, an image recognition module that estimates a region where a recognition target object exists in a captured image from image data that is a captured image is used, for example, using a technique such as neural network (including deep learning) or reinforcement learning. It is configured. In addition, since it is configured to be able to simultaneously output the estimated probability when the existence area is estimated, optimal control using the probability value is also possible. Further, the aircraft position indicated by the map coordinates at the time when the captured image is acquired is calculated by the aircraft position calculation unit, and the recognition target is obtained from the aircraft position and the recognition output data indicating the existence area of the recognition target object. Recognition object position information indicating the position of the object on the map is generated. From such a configuration, the harvesting work support that takes into consideration the estimated probability of estimating the existence area of the recognition object, the position of the recognition object on the map, and consequently the distance between the recognition object and the harvester Can be controlled.

  In the field-captured image captured by the imaging unit, the resolution for the recognition object far from the imaging unit is lower than the resolution for the recognition object near the imaging unit due to perspective. For this reason, the recognition object captured in a place far from the photographing unit has lower recognition reliability than the recognition target photographed in a place near the photographing unit. Accordingly, in one preferred embodiment of the present invention, the estimated probability of the recognition object is reduced as the recognition object is located farther from the imaging unit in the captured image. Has been.

  Since the generation of the recognition target object position information based on the captured image can be performed at a higher speed than the vehicle speed of the harvester, a plurality of recognition target object position information regarding the same recognition target object is generated. For this reason, it is possible to statistically calculate the respective estimated probabilities included in the plurality of recognition object position information regarding the same recognition object. Through this statistical calculation, recognition object position information with higher reliability can be generated. The statistical calculation here is an arithmetic average, a weighted average, an intermediate value calculation, or the like, and is an operation in which a more reliable data value is derived from a plurality of data values. By using such a statistical calculation for the estimated probability, it is possible to derive more appropriate recognition object position information from a plurality of recognition object position information. Therefore, in one preferred embodiment of the present invention, a plurality of pieces of recognition object position information are stored, and the plurality of pieces of recognition object position information stored are included in the corresponding recognition output data. It is configured to be corrected based on the result of statistical calculation of the estimated probability.

  Although the captured image acquired by the imaging unit provided in the airframe is an input source of the image recognition module, there is a possibility that the captured image may be inappropriate for recognition of the recognition target object depending on the imaging condition. As a result, even if the recognition object is recognized, the recognition probability of the recognition output data is low, so it is not preferable to use the recognition output data as it is for the subsequent processing. Therefore, in one of the preferred embodiments of the present invention, a data storage unit that temporarily and temporally stores the recognition output data sequentially output from the image recognition module as a recognition output data string, and a data A determination unit, wherein the data determination unit has the estimated probability that is lower than a predetermined level compared to the estimated probability of the recognized output data that is continuously in context in the recognized output data sequence The recognition output data is determined as inappropriate recognition output data. The recognition output data determined to be inappropriate recognition output data may be deleted, or the estimation probability of the recognition output data may be replaced with the estimation probability interpolated by the preceding and following recognition output data.

  Recognized objects that are important for a harvesting machine that harvests crops while traveling in a field are people, fallen grain straws, weeds, straws, and the like. Recognizing a person who becomes an obstacle to work traveling is a trigger for obstacle avoidance control and obstacle alarm notification, and is therefore important for safe traveling in a field. Recognizing the fallen cereals and weeds is important for carrying out high-quality harvesting work, because it becomes control of measures against fallen cereals and weeds in work control. Recognizing the cocoon is to detect the boundary line of the field and is important for headland travel control of the field.

It is a whole side view of a combine. It is a functional block diagram which shows the control system of a combine. It is explanatory drawing which shows typically the flow of the production | generation of the recognition output data by an image recognition module. It is a data flow figure showing a flow of data at the time of generating recognition subject position information from a picked-up image. It is a schematic diagram which shows the weed map obtained by mapping the weed position information as one of the recognition target object position information.

  Hereinafter, an embodiment of a combine as an example of a harvester according to the present invention will be described with reference to the drawings. In this embodiment, when the front-rear direction of the airframe 1 is defined, it is defined along the airframe traveling direction in the working state. The direction indicated by reference numeral (F) in FIG. 1 is the front side of the aircraft, and the direction indicated by reference numeral (B) in FIG. 1 is the rear side of the aircraft. When defining the left-right direction of the airframe 1, the left-right direction is defined in a state viewed in the airframe traveling direction.

  As shown in FIG. 1, in the combine, a cutting unit 2 is connected to a front part of an airframe 1 including a pair of left and right crawler travel devices 10 so as to be movable up and down around a horizontal axis X. At the rear of the machine body 1, a threshing device 11 and a grain tank 12 for storing grains are provided in a state of being arranged in the machine body width direction. A cabin 14 that covers the boarding operation part is provided at the right side of the front part of the airframe 1, and an engine 15 for driving is provided below the cabin 14.

  As shown in FIG. 1, the threshing device 11 receives the harvested cereal that has been harvested by the reaping unit 2 and conveyed backward, and the threshing stock of the cereal by the threshing feed chain 111 and the sandwiching rail 112. A threshing process is performed on the tip side by the handling cylinder 113 while being held and conveyed. And the grain selection process with respect to a threshing processed material is performed in the selection part with which the lower part of the handling cylinder 113 was equipped, and the grain selected there is conveyed to the grain tank 12, and is stored. Although not described in detail, a grain discharging device 13 for discharging the grains stored in the grain tank 12 to the outside is provided.

  The mowing unit 2 is provided with a plurality of triggering devices 21 that cause a fallen planted culm, a hair clipper-type cutting device 22 that cuts the stock of the raised planted culm, a culm transporting device 23, and the like. Yes. The grain feeder 23 is directed toward the start end of the threshing feed chain 111 of the threshing apparatus 11 located on the rear side of the machine body while gradually changing the vertical harvested grain cereal from which the stock has been cut into a laterally falling attitude. Transport.

  The cereal conveying device 23 sandwiches the stock of the collected harvested cereals while holding the gathering and conveying unit 231 that conveys a plurality of harvested cereals harvested by the cutting device 22 in the middle of the cutting width direction. The stock holding and conveying device 232 that conveys to the head, the tip locking and conveying device 233 that latches and conveys the tip of the harvested cereal rice bran, and the stock source of the harvested rice straw to the threshing feed chain 111 from the terminal end of the stock holding and conveying device 232 A supply conveyance device 234 and the like are provided for guidance.

  An imaging unit 70 including a color camera is provided at the front end of the ceiling of the cabin 14. In this embodiment, the photographic field of view of the photographing unit 70 extends in the front-rear direction almost from the front end region of the cutting unit 2 to the horizon. The breadth of the width of the field of view has reached about 10 m to several tens of m. The captured image acquired by the imaging unit 70 is converted into image data and sent to the combine control system.

  The imaging unit 70 images a farm field at the time of harvesting work, and various objects exist as imaging targets in the farm field. The control system of the combine has a function of recognizing a specific object as a recognition target object from the image data sent from the photographing unit 70. As such recognition objects, in FIG. 1, a normal planted cereal group indicated by a reference sign Z0, a weed group extending higher than the planted cereal part indicated by a reference sign Z1, and a lodging indicated by a reference sign Z2 A cereal group, a person indicated by a symbol Z3 is schematically shown.

  A satellite positioning module 80 is also provided on the ceiling of the cabin 14. The satellite positioning module 80 includes a satellite antenna for receiving GNSS (global navigation satellite system) signals (including GPS signals). In order to complement the satellite navigation performed by the satellite positioning module 80, an inertial navigation unit incorporating a gyro acceleration sensor and a magnetic bearing sensor is incorporated in the satellite positioning module 80. Of course, the inertial navigation unit can be located elsewhere. In FIG. 1, the satellite positioning module 80 is arranged at the rear part of the ceiling part of the cabin 14 for the convenience of drawing. For example, the satellite positioning module 80 is arranged as close as possible to the position directly above the central part of the left and right of the cutting device 22. It is preferable that the front end portion is disposed at a position closer to the center of the machine body.

  FIG. 2 shows a functional block diagram of the combine control system. The control system of this embodiment is composed of a number of electronic control units called ECUs, various operating devices, sensor groups and switch groups, and a wiring network such as an in-vehicle LAN that performs data transmission therebetween. The notification device 91 is a device for notifying a driver or the like of a work traveling state or various warnings, and includes a buzzer, a lamp, a speaker, a display, and the like. The communicator 92 is used by the combine control system to exchange data with the cloud computer system 100 and the portable communication terminal 200 installed at a remote location. Here, the portable communication terminal 200 is a tablet computer operated by a supervisor (including a driver) at a work traveling site. The control unit 6 is a core element of this control system, and is shown as an assembly of a plurality of ECUs. Positioning data from the satellite positioning module 80 and image data from the imaging unit 70 are input to the control unit 6 through a wiring network.

  The control unit 6 includes an output processing unit 6B and an input processing unit 6A as input / output interfaces. The output processing unit 6B is connected to the vehicle travel device group 7A and the work device device group 7B. The vehicle travel device group 7A includes control devices related to vehicle travel, such as engine control devices, shift control devices, braking control devices, steering control devices, and the like. The work device group 7B includes the mowing unit 2, the threshing device 11, the grain discharging device 13, the power control device in the cereal conveying device 23, and the like.

  A traveling system detection sensor group 8A, a work system detection sensor group 8B, and the like are connected to the input processing unit 6A. The traveling system detection sensor group 8A includes sensors that detect states of an engine speed adjusting tool, an accelerator pedal, a brake pedal, a shift operation tool, and the like. The work system detection sensor group 8B includes sensors for detecting the state of the reaper 2, the threshing device 11, the grain discharging device 13, and the cereal conveying device 23 and the state of the cereal and the cereal.

  The control unit 6 includes a work travel control module 60, an image recognition module 5, a data processing module 50, a body position calculation unit 66, and a notification unit 67.

  The notification unit 67 generates notification data based on commands from the respective functional units of the control unit 6 and gives the notification data to the notification device 91. The body position calculation unit 66 calculates the body position that is the map coordinates (or field coordinates) of the body 1 based on the positioning data sequentially transmitted from the satellite positioning module 80.

  The combine of this embodiment can be driven by both automatic driving (automatic steering) and manual driving (manual steering). The work travel control module 60 includes an automatic work travel command unit 63 and a travel route setting unit 64 in addition to the travel control unit 61 and the work control unit 62. A traveling mode switch (not shown) for selecting either an automatic traveling mode for traveling by automatic steering or a manual steering mode for traveling by manual steering is provided in the cabin 14. By operating this travel mode switch, it is possible to shift from manual steering travel to automatic steering travel, or from automatic steering travel to manual steering travel.

  The travel control unit 61 has an engine control function, a steering control function, a vehicle speed control function, and the like, and provides a travel control signal to the vehicle travel device group 7A. The work control unit 62 gives a work control signal to the work device group 7B in order to control the movement of the reaping unit 2, the threshing device 11, the grain discharging device 13, the cereal conveying device 23, and the like.

  When the manual steering mode is selected, the travel control unit 61 generates a control signal based on an operation by the driver, and controls the vehicle travel device group 7A. When the automatic steering mode is selected, based on the automatic travel command given by the automatic work travel command unit 63, the travel control unit 61 controls the vehicle travel device group 7A related to steering and the vehicle travel device group 7A related to vehicle speed. .

  The travel route setting unit 64 develops a travel route for automatic travel created in any of the control unit 6, the mobile communication terminal 200, the cloud computer system 100, and the like in the memory. The travel route expanded in the memory is sequentially used as a target travel route in automatic travel. This travel route can be used for guidance for the combine to travel along the travel route even in the case of manual travel.

  More specifically, the automatic work travel command unit 63 generates an automatic steering command and a vehicle speed command and gives them to the travel control unit 61. The automatic steering command is generated by the travel route setting unit 64 so as to eliminate the azimuth shift and the positional shift between the travel route and the vehicle position calculated by the body position calculation unit 66. The vehicle speed command is generated based on a vehicle speed value set in advance. Further, the automatic work travel command unit 63 gives a work device operation command to the work control unit 62 according to the own vehicle position and the travel state of the own vehicle.

  The image recognition module 5 receives image data of a captured image that is sequentially acquired by the imaging unit 70 over time. The image recognition module 5 estimates the existence area where the recognition target object exists in the photographed image, and outputs the recognition output data including the existence area and the estimated probability when the existence area is estimated as the recognition result. The image recognition module 5 is constructed using a neural network technique that employs deep learning.

  The flow of generating recognition output data by the image recognition module 5 is shown in FIGS. A pixel value of RGB image data is input to the image recognition module 5 as an input value. In the example of FIG. 3, the estimated authentication objects are weeds, lodging cereals and a person. Therefore, in the recognition output data as the recognition result, the existence area of the weeds (hereinafter referred to as the weed area) and its estimated probability, the existence area of the lodging cereal (hereinafter referred to as the lodging cereal area) and its estimated probability, A person existing area (hereinafter referred to as a person area) and its estimated probability are included.

  In FIG. 3, the estimation result is schematically shown, the weed area is indicated by a rectangular frame given a reference F1, the lodging cereal area is shown by a rectangular frame given a reference F2, and the person area is It is shown by a rectangular frame to which reference numeral F3 is given. Each region is linked to its estimated probability. The weed area, the overburden cedar area, and the person area are each defined by four corner points, and the coordinate positions on the captured image of the four corner points of each rectangle are also included in the estimation result. Of course, if the authentication target is not estimated, the existence area of the authentication target is not output, and the estimation probability is zero.

  In this embodiment, the image recognition module 5 sets the internal parameter so that the estimated probability of the recognition target object is reduced as the recognition target object is located farther from the photographing unit 70 in the captured image. . Thereby, the recognition of the recognition object in the imaging region where the resolution is low because it is far from the imaging unit 70 is made stricter, and erroneous recognition is reduced.

  The data processing module 50 processes the recognition output data output from the image recognition module 5. As shown in FIGS. 2 and 4, the data processing module 50 of this embodiment includes a recognition target object position information generation unit 51, a statistical processing unit 52, a data storage unit 53, a data determination unit 54, and a data correction unit 55. include.

  The recognition target object position information generation unit 51 generates recognition target object position information indicating the position of the recognition target object on the map from the body position at the time when the captured image is acquired and the recognition output data. The position on the map where the recognition objects (weeds, lodging cereals, and persons) included in the recognition output data are present is a rectangle indicating the existence area (weed area, lodging cereal area, and person area) of the authentication objects. These four corner points are obtained by converting the coordinate positions (camera coordinate positions) on the photographed image into coordinates on the map.

  Since the imaging unit 70 acquires captured images at a predetermined time interval, for example, 0.5 seconds, and inputs the image data to the image recognition module 5, the image recognition module 5 also receives the recognition output data at the same time interval. Output. Therefore, when the recognition object is in the photographing field of the photographing unit 70, the plurality of recognition output data includes the existence area for the same recognition object. As a result, a plurality of pieces of recognition object position information for the same recognition object are obtained. At that time, the estimation probability included in the recognition output data that is each original data, that is, the estimation probability of the existence area of the recognition target included in the recognition target object position information is between the imaging unit 70 and the recognition target. Since the positional relationship is different, the value is often different.

  Therefore, in this embodiment, such a plurality of recognition object position information is stored, and the estimated probability included in each of the stored plurality of recognition object position information is statistically calculated. The statistical processing unit 52 obtains a representative value of the estimated probability group using a statistical calculation for the estimated probabilities of the plurality of recognition object position information. Using the representative value, a plurality of pieces of recognition object position information can be corrected to one optimum recognition object position information (recognition object correction position information). An example of such correction is to calculate an arithmetic average value, a weighted average value, or an intermediate value of each estimated probability as a reference value (representative value), calculate a logical sum of existing regions having an estimated probability equal to or higher than the reference value, and To generate correction recognition target object position information having the optimal existence area as. Of course, it is also possible to generate one piece of recognition object position information with high reliability by using other statistical calculations. That is, a plurality of recognition target object position information is corrected based on the result of statistical calculation of the estimated probability included in the recognition output data corresponding to the recognition target object position information.

  Recognition object position information (weed position information, lodging cereal position information, person position) indicating the position on the map of the existence area (weed area, lodging cereal area, person area) of the authentication object obtained in this way By using (information), traveling work control and warning notification set in advance are performed at the time of weeds, lodging cereals, and person recognition.

  As described above, in the present invention, the estimated probability output from the image recognition module 5 is important for the generation of final recognition object position information. For this reason, in this embodiment, the recognition output data evaluation that examines the reliability of the estimation probability of the recognition output data output from the image recognition module 5 and determines the recognition output data having the unreliable estimation probability as the inappropriate recognition output data. The function is provided in the data processing module 50.

  The recognition output data evaluation function is realized by a data storage unit 53, a data determination unit 54, and a data correction unit 55, as shown in FIGS. The data storage unit 53 stores the recognition output data sequentially output from the image recognition module 5 temporarily and as a recognition output data string. In the recognition output data string, the data determination unit 54 compares the estimated probability of the recognition output data to be determined with the estimated probability of the recognition output data that is continuously related to the recognition output data. Recognition output data having an estimated probability lower than the level is determined as inappropriate recognition output data.

  Since the captured image as the recognition source is captured during the combine operation, the captured image may become unclear due to the sudden movement of the combine. Furthermore, since the combine changes its direction in the vicinity of the ridge, the shooting direction may be suddenly changed. Along with this, shooting with normal light and shooting with backlight may be mixed for the same recognition object. Recognition output data having an unexpectedly low estimated probability may occur due to such changes in shooting conditions, etc., but extraction of inappropriate recognition output data that is such recognition output data is performed by the above-described recognition output data evaluation function. Is possible.

  When the number of recognition output data is small, the estimated probability of the recognition output data determined to be inappropriate recognition output data may be interpolated and corrected with the estimated probability of the recognition output data before and after that. Accordingly, the inappropriate recognition output data is appropriately corrected and can be used, and an advantage that the number of data is secured can be obtained. The data correction unit 55 performs such interpolation correction. Of course, when the recognition output data determined to be inappropriate recognition output data is discarded, the interpolation correction is unnecessary, and the data correction unit 55 is omitted.

  In addition, each recognition target object position information generated by the recognition target object position information generation unit 51 can be mapped as shown in FIG. 5 for easy visual understanding. FIG. 5 illustrates a weed map in which weed position information is mapped. When the weed position information includes a weed presence area with different estimation probabilities, it is also possible to display the weed existence area in a form divided into patterns within a predetermined range of estimated probability values as shown in FIG. .

  Note that the configurations disclosed in the above-described embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with the configurations disclosed in the other embodiments as long as no contradiction arises. The embodiment disclosed in this specification is an exemplification, and the embodiment of the present invention is not limited to this. The embodiment can be appropriately modified without departing from the object of the present invention.

[Another embodiment]
(1) In the embodiment described above, the image recognition module 5 is constructed using a deep learning type neural network technique. Instead of this, an image recognition module 5 constructed using other machine learning techniques may be employed.
(2) In the above-described embodiment, the image recognition module 5 and the data processing module 50 are incorporated in the combine control unit 6, but part or all of the image recognition module 5 and the data processing module 50 is a control unit independent of the combine, for example, portable communication It can be constructed in the terminal 200 or the like.
(3) Each functional unit shown in FIG. 2 is divided mainly for the purpose of explanation. Actually, each functional unit may be integrated with other functional units, or may be further divided into a plurality of functional units.

  The present invention is not only a harvester that harvests rice, wheat, etc., but also a harvester that harvests other agricultural products such as corn, carrots, etc. It can also be applied to a harvesting machine that harvests.

1: Airframe 2: Cutting unit 5: Image recognition module 50: Data processing module 51: Recognition object position information generation unit 52: Statistical processing unit 53: Data storage unit 54: Data determination unit 55: Data correction unit 6: Control unit 60: Work travel control module 61: Travel control unit 62: Work control unit 66: Airframe position calculation unit 70: Imaging unit 80: Satellite positioning module

Claims (5)

A harvesting machine that harvests crops while traveling in the field,
An aircraft position calculation unit that calculates an aircraft position that is map coordinates of the aircraft based on positioning data from the satellite positioning module;
An imaging unit provided in the machine body for imaging a farm field during harvesting;
When image data of a photographed image sequentially acquired by the photographing unit is input, a presence area where a recognition target object exists in the photographed image is estimated, and the existence area and the existence area are estimated An image recognition module that outputs recognition output data including the estimated probability of
A recognition target object position information generating unit that generates recognition target object position information indicating a position of the recognition target object on a map from the aircraft position at the time when the captured image is acquired and the recognition output data;
Harvester with   The harvester according to claim 1, wherein the estimated probability of the recognition object is reduced as the recognition object is located farther from the imaging unit in the captured image.   A plurality of pieces of recognition object position information are stored, and the plurality of pieces of recognition object position information stored are corrected based on a result of a statistical calculation of the estimated probability included in the corresponding recognition output data. The harvesting machine according to claim 1 or 2. A data storage unit for temporarily and temporally storing the recognition output data sequentially output from the image recognition module as a recognition output data sequence, and a data determination unit,
The data determination unit inappropriately outputs the recognition output data having the estimated probability that is lower than a predetermined level compared to the estimated probability of the recognition output data that is continuously in a context in the recognition output data string. The harvester according to any one of claims 1 to 3, which is determined as data.   The harvesting machine according to any one of claims 1 to 4, wherein the recognition target object is at least one of a person, a fallen cereal cocoon, a weed, and a cocoon.

Images