US20220361392A1 - Method and system for driving view-based agricultural machinery and device for agricultural machinery applying method - Google Patents

Method and system for driving view-based agricultural machinery and device for agricultural machinery applying method Download PDF

Info

Publication number
US20220361392A1
US20220361392A1 US17/739,137 US202217739137A US2022361392A1 US 20220361392 A1 US20220361392 A1 US 20220361392A1 US 202217739137 A US202217739137 A US 202217739137A US 2022361392 A1 US2022361392 A1 US 2022361392A1
Authority
US
United States
Prior art keywords
driving
navigation route
image information
agricultural machinery
operation area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US17/739,137
Other languages
English (en)
Inventor
Di Wu
Yuan Yao
Wen-Quan Zhao
Bo Wang
Hong-Xin Li
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
FJ Dynamics Technology Co Ltd
Original Assignee
FJ Dynamics Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by FJ Dynamics Technology Co Ltd filed Critical FJ Dynamics Technology Co Ltd
Publication of US20220361392A1 publication Critical patent/US20220361392A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G05D1/0253—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow
    • A—HUMAN NECESSITIES
    • A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01B—SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL
    • A01B69/00—Steering of agricultural machines or implements; Guiding agricultural machines or implements on a desired track
    • A01B69/001—Steering by means of optical assistance, e.g. television cameras
    • A—HUMAN NECESSITIES
    • A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01B—SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL
    • A01B69/00—Steering of agricultural machines or implements; Guiding agricultural machines or implements on a desired track
    • A01B69/007—Steering or guiding of agricultural vehicles, e.g. steering of the tractor to keep the plough in the furrow
    • A01B69/008—Steering or guiding of agricultural vehicles, e.g. steering of the tractor to keep the plough in the furrow automatic
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0055—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots with safety arrangements
    • G05D1/0061—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots with safety arrangements for transition from automatic pilot to manual pilot and vice versa
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0214—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory in accordance with safety or protection criteria, e.g. avoiding hazardous areas
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0219—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory ensuring the processing of the whole working surface
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0276—Control of position or course in two dimensions specially adapted to land vehicles using signals provided by a source external to the vehicle
    • G05D1/0278—Control of position or course in two dimensions specially adapted to land vehicles using signals provided by a source external to the vehicle using satellite positioning signals, e.g. GPS
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G05D2201/0201—

Definitions

  • the present disclosure relates to the field of agriculture, in particular to a method and system for driving view-based agriculture machinery, and device applying method.
  • Control of agricultural machinery is dominated by manual driving.
  • Technologies such as Beidou satellite positioning and LIDAR have also been introduced into agricultural machinery operations to realize automatic driving.
  • agricultural machinery mainly uses automatic driving based on the global satellite positioning system (including but not limited to Beidou, GPS, GLONASS, etc.) and ground augmentation stations, with accurate positioning and accurate navigation operations.
  • satellite navigation operations are limited by the constraints of the operating terrain and environment. For complex areas of land, effective full-coverage operations cannot be carried out. Infrared, ultrasonic, and LIDAR sensors provide some assistance. These methods can improve the navigation effect to a certain extent. However, when the navigation accuracy too reduced to be navigated, there is no corresponding discovery and treatment, which can easily lead to accidents.
  • the present disclosure provides a method and system for driving view-based agriculture machinery, a device on an agricultural machine, and a non-volatile storage medium.
  • the method of the present disclosure can reduce accidents caused by automatic driving, the present disclosure further can seamlessly switch from automatic to manual driving when automatic driving cannot be carried out, does not affect the normal operation of the agricultural machinery, and improves the operation efficiency of the agricultural machinery.
  • the embodiment of the present disclosure provides a view-based agriculture machinery driving method, the method may include:
  • the identifying of a target operation area according to the ground image information may include:
  • the identifying of a target operation area according to the ground image information may include:
  • determining whether the ground image information meets identification requirements may include:
  • determining whether the navigation route is reliable may include:
  • determining the driving adjustment parameters according to the navigation route and current driving attitude of the agricultural machine may include:
  • the method further includes:
  • controlling the agricultural machine to stop working if an end-of-path sign is reached in the ground image information, or when the agricultural machine reaches an end position of the navigation route.
  • the embodiment of the present disclosure provides a view-based agriculture machinery system method, the system may include:
  • an image acquisition module is configured to collect ground image information
  • an operation area identification module is configured to identify a target operation area according to the ground image information
  • a navigation route determination module is configured to determine navigation route of the agricultural machinery within the target operation area
  • a reliability determination module is configured to determine whether the navigation route is reliable
  • a manual driving module can detect manual takeover and corresponding driving signal of the user and controlling the driving of the agricultural machinery to be manual if the navigation route is not reliable;
  • an automatic driving module is configured to determine driving adjustment parameters according to the navigation route and current driving attitude of the agricultural machine and if the navigation route is reliable controlling the driving of the agricultural machine according to the driving adjustment parameters.
  • the embodiment of the present disclosure provides a device for mounting on an agricultural machine
  • the device may include one or more processors and a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the view-based agricultural machinery driving method provided by any embodiment of the present disclosure.
  • the embodiment of the present disclosure provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by the processor, the view-based agricultural machinery driving method provided by any embodiment of the present disclosure is realized.
  • the ground image information as to the land environment around the agricultural machinery during the operation of agricultural machinery identify the target operation area for carrying out the operation of the agricultural machinery according to the ground image information, determines the navigation route of the agricultural machinery according to the identified target operation area, and switches to and from manual driving and automatic driving, according to whether the navigation route is reliable.
  • the navigation route is determined to be reliable
  • the manual driving signal of the user is detected and the driving of the agricultural machinery is done manually according to the manual driving signal.
  • the driving adjustment parameters are determined according to the navigation route and the current driving attitude of the agricultural machinery, so as provide automatic control of the driving of the agricultural machinery according to the driving adjustment parameters.
  • the method can plan the navigation route according to the collected ground image information in the process of the agricultural machinery operation, so as to carry out automatic driving operation, and make determinations as to reliability of navigation in the process of automatic driving.
  • the navigation route is found unreliable, it can be seamlessly switched to manual driving in time, which reduces the accidents caused by automatic driving to a certain extent. Normal operation of the agricultural machinery is not affected, and the operational efficiency of the agricultural machinery is improved.
  • FIG. 1 is a flowchart of a view-based method for providing automatic or manual driving of agricultural machinery as may be optimal from time to time according to a first embodiment of the present disclosure.
  • FIG. 2 is another flowchart of the view-based agricultural machinery driving method according to the first embodiment of the present disclosure.
  • FIG. 3 is a flowchart of the view-based agricultural machinery driving method according to a second embodiment of the present disclosure.
  • FIG. 4 is another flowchart of the view-based agricultural machinery driving method according to the second embodiment of the present disclosure.
  • FIG. 5 is a schematic diagram of a view-based agricultural machinery driving system according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic diagram of a device for mounting to agricultural machinery according to an embodiment of the present disclosure.
  • first”, “second” and the like may be used herein to describe various directions, actions, steps or elements, but these directions, actions, steps or elements are not limited by these terms. These terms are used only to distinguish a first direction, action, step, or element from another direction, action, step, or element.
  • the first module may be called the second module.
  • the second module can be called the first module. Both the first module and the second module are modules, but they are not the same module.
  • the terms “first”, “second”, etc. cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.
  • the features defining “first” and “second” may explicitly or implicitly include one or more of the features.
  • “a plurality of” means at least two, such as two, three, etc., unless otherwise specifically defined.
  • FIG. 1 illustrates a view-based agricultural machinery driving method in accordance with an embodiment of the present disclosure.
  • the method can be applied to the driving system of various agricultural machinery to lead or assist the driving of agricultural machinery.
  • the view-based agricultural machinery driving method of the embodiment of the present disclosure can be applied to grain harvesters, seed planters, plant sprayers and fertilizer spreaders, tractors, and other farmland devices.
  • the method includes the following steps:
  • step S 110 collecting ground information from images.
  • the ground image information is collected by an image acquisition module to determine the current working environment of an agricultural machinery.
  • the image acquisition module is usually set on the agricultural machinery to collect images based on known preset angle.
  • the known preset angle mentioned here allows correlation to be made between the location and position of the agricultural machinery and the collected ground image information, so as to facilitate the determination of the actual position of the agricultural machinery, it can be determined by the actual orientation (for example, if the image acquisition module is set on the agricultural machinery, it needs to be determined in combination with the orientation of the agricultural machinery and the orientation of the image acquisition module on the agricultural machinery) and fixed shooting angle of the image acquisition module (for example, setting up an image acquisition module on the agricultural machinery, and fixing the aim of the image acquisition module to one or more shooting angles relative to the agricultural machinery).
  • the image acquisition module is a camera (the specific model of the camera is not limited).
  • One or more cameras are set on the agricultural machinery to collect the ground image information around the agricultural machinery during the operation of the agricultural machinery.
  • the camera may be fixed in front of or on the side of the agricultural machinery, and the lens of the camera may face the driving direction of the agricultural machinery, the optical axis of the lens being at a fixed angle with the ground.
  • the 3D space conversion relationship of the camera installation position is stored in the agricultural machinery. More specifically, the one or more cameras can collect the ground image on the agricultural machinery based on one or more fixed angles (relative to the agricultural machinery), so as to cover a wider shooting angle and obtain more comprehensive ground image information.
  • step S 120 identifying a target operation area according to the ground image information.
  • the target operation area refers to the area where agricultural machinery is required to operate, such as farmland to be harvested.
  • the ground image information includes a farmland area and a background of the farmland area. Taking the grain harvester as an example, the grain harvester will continuously capture the ground image information through its camera during driving.
  • the captured ground image information includes a planting area of the farmland, a division area between adjacent planting areas, and non-farmland areas such as the sky.
  • the planting area can be further divided into an operable area and a non-operable area, and the non-operated area is the target operation area referred to in the embodiment.
  • the target operation area is the actual land on which operation tasks of the agricultural machinery are carried out, so as to facilitate the subsequent control of the agricultural machinery to traverse the target operation area to complete the operation.
  • the agricultural machinery includes a processing module, the processing module can adopt on-board computer.
  • the processing module stores relevant algorithms or models for processing the ground image to identify the features in the image, segments the image according to the recognized features to obtain different regions, and then identifies the target operation area.
  • step S 130 determining a navigation route of the agricultural machinery based on the target operation area.
  • the navigation route is a predicted route for planning the path of the agricultural machinery, which is used to guide agricultural machinery to traverse all target operation areas and complete operation tasks.
  • the navigation route needs to be determined according to the specific target operation area and the operation task of the agricultural machinery.
  • the target operation area determined based on the ground image information includes a planting area extending along the forward direction of the agricultural machine, the planting area is rectangular.
  • the determined navigation line is the center line of the target area.
  • the processing module of the agricultural machinery uses the preset vision algorithm to search the target operation area and predicts the navigation route of the agricultural machinery according to the specific operation task, so that the agricultural machinery can traverse the target operation area and complete the operation task according to the navigation route.
  • the operation modes of the agricultural machinery include edge operation, bundled operation and between-rows operations.
  • the navigation route is determined in different ways.
  • the processing module is preset with a variety of algorithms to formulate navigation routes for different operation types.
  • special algorithms can also be set in the processing module. The special algorithm is used to select the most suitable operation mode according to the ground image information and the operation tasks (such as harvesting and sowing) of the agricultural machinery.
  • step S 140 determining whether the navigation route is reliable. If the navigation route is reliable, step S 160 is implemented, otherwise step S 150 is implemented.
  • the navigation route is determined by algorithm, on the one hand, the actual agricultural operation environment is complex, on another hand, due to the possible limitations of the algorithm itself, it is not easy to make the target operation area geometrically neat and difficult to formulate the navigation route, or even if the navigation route is formulated, there is some possibility of error.
  • the actual operation put forward for the navigation route does not completely traverse the target operation area, or there may be accidents such as collision with obstacles when operating according to the navigation route, so the navigation route needs to be further verified.
  • the processing module includes a reliability verification algorithm to simulate the actual operation situation of the agricultural machinery according to the navigation route. If the agricultural machinery can traverse the target operation area according to the navigation route and complete the corresponding operation tasks, it is deemed to have passed the reliability verification, otherwise it will not pass. If the navigation route is not determined in step S 130 , it is directly determined that the navigation route is unreliable.
  • Determining whether the navigation route is reliable can include: determining whether the navigation route traverses the target operation area completely and whether the operation task in the target operation area can be carried out. If the navigation route traverses the target operation area completely and can complete the operation task corresponding to the target operation area, the navigation route is reliable.
  • step S 150 detecting manual driving signal of the user and controlling the driving of the agricultural machinery according to the manual driving signal.
  • the manual driving signal is generated by the user-driver.
  • the manual driving operation includes remote driving operation (through remote control terminal operation) and actual direct driving by operator.
  • the processing module verifies that the navigation route is unreliable, it needs to switch to the manual driving mode, and the user drives manually.
  • the agricultural machinery drives and operates according to the manual driving signals of the user.
  • the agricultural machinery includes a manual driving module
  • the manual driving module includes manual operation members (including steering wheel, etc.) and remote control components (including remote control terminal, etc.), so as to facilitate manual driving by the user, such as steering wheel and or remote control terminal, when the navigation route put forward by a computer is unreliable.
  • step S 160 determining driving adjustment parameters according to the navigation route and current driving attitude of the agricultural machine, and controlling the driving of the agricultural machine according to the driving adjustment parameters.
  • the driving adjustment parameters are used to control the automatic driving of the agricultural machinery according to the navigation route.
  • the driving adjustment parameters include driving steering angle and offset distance.
  • the driving steering angle is used to control the driving direction of the agricultural machinery, and the offset distance is used to determine the distance travelled by the agricultural machinery.
  • the agricultural machinery includes an automatic driving module (which can be a virtual module based on the processing module or a separate hardware module), the automatic driving module controls the agricultural machine to move according to the navigation route based on the current driving attitude of the agricultural machine after the processing module verifies that the navigation route is reliable.
  • the automatic driving module determines how to steer the agricultural machine from a current driving track to the required navigation line according to the current driving attitude and the navigation route of the agricultural machine, and the driving adjustment parameters reflect the deviation that needs to be corrected between the current driving track and the navigation route.
  • the processing module can adopt a variety of ways when identifying the target operation area according to the ground image information, for example, threshold segmentation method, adding genetic algorithm, and particle swarm optimization algorithm to auxiliary segmentation method and image semantic segmentation method based on deep convolution neural network, and different methods have different advantages and disadvantages.
  • the image semantic segmentation method based on deep convolution neural network is preferred, the method has the advantages of fast recognition, high accuracy, and can be continuously optimized during actual use.
  • a preset depth learning model needs to be configured in the processing module, and the preset depth learning model is obtained through a large number of images of ground for training.
  • the training ground images include ground images in various working environments, and the features used to realize image segmentation shall be marked on the training ground images (marked according to different ground areas of the ground image), based on the training ground image, the training times of the deep learning model are controlled according to the actual needs, and the preset deep learning model is obtained after the training. Accordingly, identifying the target operation area according to the ground image information includes: segmenting the ground image information through a preset depth learning model to obtain a plurality of ground areas, and identifying the target operation area in the plurality of ground areas.
  • the view-based agricultural machinery driving method of the embodiment of the present disclosure can collect the ground image information around the agricultural machinery during the operation of agricultural machinery, identify the target operation area that needs the operation of the agricultural machinery according to the ground image information, determine the navigation route of the agricultural machinery according to the identified target operation area, and switch between manual driving and automatic driving according to whether the navigation route is reliable at that moment.
  • the navigation route is determined to be reliable
  • the manual driving signal of the user is detected to control the driving of the agricultural machinery according to the manual driving signal.
  • the driving adjustment parameters are determined according to the navigation route and the current driving attitude of the agricultural machinery, so as to automatically control the driving of the agricultural machinery according to the driving adjustment parameters.
  • the method can plan the navigation route according to the collected ground image information in the process of the agricultural machinery operation, so as to carry out automatic driving operation, and introduce assessments of reliability in the process of automatic driving.
  • the navigation route is unreliable, it can be found and switched to manual driving in time, which reduces the accidents caused by automatic driving to a certain extent, and can be switched seamlessly when automatic driving cannot be carried out. This does not affect the normal operation of the agricultural machinery, and improves the operation efficiency of the agricultural machinery.
  • Embodiment 2 of the present disclosure further supplements and explains some contents in the Embodiment 1 on the basis of the Embodiment 1 of the present disclosure.
  • FIG. 2 illustrates a view-based agricultural machinery driving method in accordance with another embodiment of the present disclosure.
  • the method includes the following steps:
  • step S 210 collecting ground image information.
  • step S 220 determining whether the ground image information meets the identification requirements.
  • the image acquisition module will be affected by weather and other factors, such as rain and snow, the clarity of the ground image information will be affected.
  • identifying the target operation area according to the ground image information may not include obvious ground features, or the ground features may be discontinuous (for example, it is difficult to distinguish the target operation area when the snow covers the land).
  • the identification of the target operation area based on the ground image information is based on the ground features
  • the determining of whether the ground image information meets the identification requirements can include:
  • step S 230 obtaining GPS positioning information of the agricultural machinery and identifying the target operation area according to the GPS positioning information and the ground image information if the ground image information does not meet the identification requirements.
  • a global positioning system (GPS) positioning information is introduced to assist in the identification of the target operation area.
  • GPS global positioning system
  • the agricultural machinery further includes a satellite positioning module, the satellite positioning module is used to determine the GPS positioning information of the agricultural machinery, and modify and supplement the ground features information of the ground image information with the help of GPS positioning information, so as to make the ground features information of the ground image information accurate and continuous, and finally segment and identify the ground image based on the accurate and continuous ground features information, so as to determine the target operation area.
  • the satellite positioning module is used to determine the GPS positioning information of the agricultural machinery, and modify and supplement the ground features information of the ground image information with the help of GPS positioning information, so as to make the ground features information of the ground image information accurate and continuous, and finally segment and identify the ground image based on the accurate and continuous ground features information, so as to determine the target operation area.
  • step S 240 determining the navigation route of the agricultural machinery based on the target operation area.
  • step S 250 determining whether the navigation route is reliable. If the navigation route is reliable, step S 270 is implemented, otherwise step S 260 is implemented.
  • step S 260 detecting manual driving signal of the user and allowing manual driving of the agricultural machinery according to the manual driving signal.
  • step S 270 determining driving adjustment parameters according to the navigation route and current driving attitude of the agricultural machine, and controlling the driving of the agricultural machine according to the driving adjustment parameters.
  • step S 220 if it is determined in step S 220 that the ground image information meets the identification requirements, the target operation area can be identified directly according to the ground image information without introducing the GPS positioning information.
  • the other steps are consistent with steps S 240 -S 270 .
  • FIG. 3 illustrates a view-based agricultural machinery driving method in accordance with another embodiment of the present disclosure.
  • the method includes the following steps:
  • step S 310 collecting ground image information.
  • step S 320 identifying the target operation area according to the ground image information.
  • step S 330 determining the navigation route of the agricultural machinery based on the target operation area.
  • step S 340 determining whether the navigation route is reliable. If the navigation route is reliable, step S 360 is implemented, otherwise step S 350 is implemented.
  • step S 350 detecting manual driving signal of the user and allowing manual driving of the agricultural machinery according to the manual driving signal.
  • step S 360 detecting the current situation of the agricultural machinery on the land.
  • the two control objectives are the position and course of the vehicle.
  • the deviation between the expected course and the actual course must be known.
  • the expected course can be determined according to the navigation route
  • the actual course is the current course of the agricultural machinery.
  • the current course can be determined according to the orientation of the agricultural machinery, the orientation of the agricultural machinery in a certain direction is the current course of the agricultural machinery. The above is just an example, in fact, the moving direction of the agricultural machinery can also be taken as the current course.
  • step S 370 determining deviation between current course and the navigation route.
  • the driving course deviation is actually the difference between the current course and the expected course put forward and corresponding to the navigation route.
  • step S 380 determining driving steering angle and offset distance of the agricultural machine based on the deviation and the driving parameters of the agricultural machine.
  • the deviation is only used to describe how to achieve the desired course of the navigation route, but the agricultural machinery is often not on the navigation route, so it is also necessary to synchronize the position.
  • it when considering the course synchronization according to the driving course deviation, it will also calculate how to move according to the driving parameters (including speed and current position) of the agricultural machinery to realize synchronization with the navigation route in position and course.
  • the driving steering angle is used to determine the automatic driving direction of the agricultural machine
  • the offset distance is used to determine how far the agricultural machine needs to be offset according to the driving steering angle, which can be synchronized with the navigation route in position.
  • FIG. 4 illustrates a view-based agricultural machinery driving method in accordance with another embodiment of the present disclosure.
  • the method includes the following steps:
  • step S 410 collecting ground image information.
  • step S 420 identifying the target operation area according to the ground image information.
  • step S 430 determining the navigation route of the agricultural machinery based on the target operation area.
  • step S 440 determining whether the navigation route is reliable. If the navigation route is reliable, step S 460 is implemented, otherwise step S 450 is implemented.
  • step S 450 detecting manual driving signal of the user and allowing manual driving of the agricultural machinery according to the manual driving signal.
  • step S 460 determining driving adjustment parameters according to the navigation route and current driving attitude of the agricultural machine, and controlling the driving of the agricultural machine according to the driving adjustment parameters.
  • step S 470 determining whether there is an end-of-path stop sign in the ground image information, or determining whether the agricultural machine reaches an end position of the navigation route.
  • the stop sign is used to mark the areas where the agricultural machinery is not allowed to travel, such as planting areas where harvesting is not required and areas where the agricultural machinery should not travel, so as to avoid operation errors or risk of accidents.
  • the stop sign can be a sign specially set for the automatic driving of the agricultural machinery, or a special feature of the terrain, or traffic sign.
  • a stop sign is set on the ground in the area where the agricultural machinery is prohibited.
  • a stop sign identification algorithm is preset in the processing module of the agricultural machinery to identify such stop sign of the ground image information.
  • the deep learning model can also be used to identify the stop sign, the deep learning model used here is not the same as the deep model for identifying the target operation area.
  • step S 480 controlling the agricultural machinery to stop working if it meets a stop sign in the ground image information, or when the agricultural machine reaches the end position of the navigation route.
  • the processing module When it is determined that there is a stop sign in the ground image information or that the agricultural machinery has driven to the end position of the navigation route, it is necessary to stop the agricultural machinery in time.
  • the processing module detects a meeting with the stop sign or when the agricultural machinery has driven to the end position of the navigation route, it sends a termination signal in time to cut off the power output of the agricultural machinery and apply brake.
  • the view-based agricultural machinery driving method of the embodiment of the present disclosure can assist the process of the target operation area identification according to GPS positioning information when the ground image information does not meet the identification requirements, and further ensure the safety and stability of automatic driving.
  • the method can also determine the driving adjustment parameters according to the navigation route, and provide the process of stopping according to the stop sign or end of navigation route, so as to ensure that the agricultural machinery will not enter the non-operation area by mistake, improving the safety of automatic driving of the agricultural machinery.
  • FIG. 5 illustrates a view-based agricultural machinery driving system 500 in accordance with another embodiment of the present disclosure.
  • the view-based agricultural machinery driving system 500 provided in Embodiment 3 of the present disclosure can execute the view-based agricultural machinery driving method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.
  • the view-based agricultural machinery driving system 500 includes an image acquisition module 510 , an operation area identification module 520 , a navigation route determination module 530 , a reliability determination module 540 , a manual driving module 550 , and an automatic driving module 560 .
  • the image acquisition module 510 is used to collect the ground image information.
  • the operation area identification module 520 is used to identify the target operation area according to the ground image information.
  • the navigation route determination module 530 is used to determine the navigation route of the agricultural machinery based on the target operation area.
  • the reliability determination module 540 is used to determine whether the navigation route is reliable.
  • the manual driving module 550 is used to detect the manual driving signal of the user to control the driving of the agricultural machinery according to the manual driving signal.
  • the automatic driving module 560 is used to determine the driving adjustment parameters according to the navigation route and the current driving attitude of the agricultural machine, so as to control the driving of the agricultural machine according to the driving adjustment parameters.
  • the operation area identification module 520 is specifically used for: segmenting semantics the ground image information through a preset depth learning model to obtain a plurality of ground areas, and identifying the target operation area in the plurality of ground areas.
  • the operation area identification module 520 is specifically used for: determining whether the ground image information meets the identification requirements, obtaining the GPS positioning information of the agricultural machinery and identifying the target operation area based on the GPS positioning information and the ground image information if the ground image information does not meet the identification requirements.
  • the determining of whether the ground image information meets the identification requirements can include: determining whether the ground feature information of the ground image information is accurate and continuous. If information concerning the ground features of the ground image information is inaccurate or discontinuous, the ground image information does not meet the identification requirements.
  • the reliability determination module 540 is specifically used for: determining whether the navigation route traverses the target operation area completely and can complete the operation task corresponding to the target operation area, determining that the navigation route is reliable if the navigation route traverses the target operation area completely and can complete the operation task corresponding to the target operation area.
  • the automatic driving module 560 is specifically used for: detecting the current course of the agricultural machinery, determining driving course deviation between the current course and the navigation route, and determining the driving steering angle and the offset distance of the agricultural machine based on the driving course deviation and the driving parameters of the agricultural machine.
  • the view-based agricultural machinery driving system 500 further includes a termination detection module and a stop module.
  • the termination detection module is used to determine whether there is a stop sign in the ground image information, or when the agricultural machine reaches the termination of the navigation route.
  • the stop module is used to control the agricultural machinery to stop working if there is a stop sign in the ground image information, or when the agricultural machine reaches the termination of the navigation route.
  • the view-based agricultural machinery driving system of the embodiment of the present disclosure can plan the navigation route according to the collected ground image information in the process of agricultural machinery operation, so as to carry out automatic driving operation.
  • the system introduces assessments of reliability in the process of automatic driving, and finds and switches to manual driving in time when the navigation route is unreliable, reduces the accidents caused by automatic driving to a certain extent, and switches seamlessly to manual driving when automatic driving cannot be carried out, does not affect the normal operation of the agricultural machinery, and improves the operation efficiency of the agricultural machinery.
  • FIG. 6 illustrates an agriculture machinery device 12 in accordance with another embodiment of the present disclosure.
  • FIG. 6 shows a block diagram of an exemplary agricultural machinery device 12 suitable for implementing an embodiment of the present disclosure.
  • the agricultural machinery device 12 shown in FIG. 6 is only an example and should not impose any restrictions on the function and scope of use of the embodiment of the present disclosure.
  • the agricultural machinery device 12 is represented in the form of general computing device.
  • the components of the agricultural machinery device 12 may include, but are not limited to, one or more processors 16 or a processing unit, a system storage device 28 , and a bus 18 connecting different system components (including the system storage device 28 and the processor 16 ).
  • the bus 18 represents one or more of several types of bus structures, including storage device bus or storage device controller, peripheral bus, graphics acceleration port, processor or local bus, using any of a variety of bus structures.
  • these architectures include but are not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC) bus, enhanced ISA bus, video electronics standards association (VESA) local bus, and peripheral component interconnection (PCI) bus.
  • ISA industry standard architecture
  • MAC microchannel architecture
  • VESA video electronics standards association
  • PCI peripheral component interconnection
  • the agricultural machinery device 12 typically includes a variety of computer system readable storage medium. These mediums can be any available medium that can be accessed by the agricultural machinery device 12 , including volatile medium and non-volatile medium, removable medium and non-removable medium.
  • the system storage device 28 may include a computer system readable medium in the form of volatile storage device, for example, random access memory (RAM) 30 and/or cache memory 32 .
  • the agricultural machinery device 12 may further include other removable/non removable, volatile/nonvolatile computer system storage medium.
  • the storage system 34 may be used to read and write non removable, non-volatile magnetic medium (not shown in FIG. 6 , commonly referred to as “hard disk drive”).
  • a disk drive for reading and writing a removable nonvolatile disk such as a “floppy disk”
  • an optical disk drive for reading and writing a removable nonvolatile optical disk (such as a CD-ROM, DVD-ROM, or other optical medium) may be provided.
  • each drive may be connected to bus 18 through one or more data medium interfaces.
  • the storage device 28 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of various embodiments of the present disclosure.
  • the utility 40 has a set of programs of the program module 42 that can be stored, for example in the storage device 28 .
  • Such program modules 42 include, but are not limited to, an operating system, one or more applications, other program modules, and program data, and each or some combination of these examples may include an implementation of a network environment.
  • the program module 42 typically performs functions and/or methods in the embodiments described in the present disclosure.
  • the agricultural machinery device 12 can also communicate with one or more external devices 14 (such as keyboard, pointing device, display 24 , etc.), one or more devices that enable the user to interact with the agricultural machinery device 12 , and/or any device that enables the agricultural machinery device 12 to communicate with one or more other computing devices (such as network card, modem, etc.), the communication being carried out through the input output interface 22 .
  • the agricultural machinery device 12 may also communicate with one or more networks (such as local area network (LAN), wide area network (WAN) and/or public network, such as the Internet) through the network adapter 20 . As shown in FIG. 6 , the network adapter 20 communicates with other modules of the agricultural machinery device 12 through the bus 18 .
  • LAN local area network
  • WAN wide area network
  • public network such as the Internet
  • the processing unit 16 executes various functional applications and data processing by running programs stored in the system storage device 28 , to realize the view-based agricultural machinery driving method provided by the embodiment of the present disclosure:
  • Embodiment 5 of the present disclosure also provides a computer readable storage medium.
  • a computer program is stored on the computer readable storage medium, and when the computer program is executed by the processor or the processing unit, realizes the view-based agricultural machinery driving method provided by all the embodiments of the present disclosure:
  • the computer readable storage medium of the embodiment of the present disclosure can adopt any combination of one or more computer readable media.
  • the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
  • the computer readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above.
  • the computer readable storage media include: electrical connection with one or more wires, portable computer disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • the computer readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device.
  • the computer readable signal medium may include a data signal propagating in the baseband or as part of the carrier wave, in which the computer readable program code is carried, this transmitted data signal can take many forms, including but not limited to, electromagnetic signal, optical signal or any suitable combination of the above.
  • the computer readable signal medium may also be any computer readable media other than a computer readable storage medium.
  • the computer readable medium may transmit, propagate, or transmit a program for use by or in combination with an instruction execution system or device.
  • the program code contained on the computer readable medium can be transmitted to or from any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
  • Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof.
  • the programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as “C” language or similar programming languages.
  • the program code can be executed entirely on the user's computer, partially on the user's computer as a separate software package, partially on the user's computer, partially on the remote computer, or entirely on the remote computer or server.
  • the remote computer may be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (such as using an Internet service provider to connect through the Internet).
  • LAN local area network
  • WAN wide area network
  • Internet service provider such as using an Internet service provider to connect through the Internet.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Automation & Control Theory (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Soil Sciences (AREA)
  • Environmental Sciences (AREA)
  • Electromagnetism (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Guiding Agricultural Machines (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Navigation (AREA)
US17/739,137 2021-05-14 2022-05-08 Method and system for driving view-based agricultural machinery and device for agricultural machinery applying method Pending US20220361392A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110529974.9 2021-05-14
CN202110529974.9A CN114115214B (zh) 2021-05-14 2021-05-14 一种基于视觉的农机驾驶方法、系统、设备及存储介质

Publications (1)

Publication Number Publication Date
US20220361392A1 true US20220361392A1 (en) 2022-11-17

Family

ID=80359225

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/739,137 Pending US20220361392A1 (en) 2021-05-14 2022-05-08 Method and system for driving view-based agricultural machinery and device for agricultural machinery applying method

Country Status (6)

Country Link
US (1) US20220361392A1 (ja)
EP (1) EP4094558A1 (ja)
JP (1) JP7390428B2 (ja)
CN (1) CN114115214B (ja)
AU (1) AU2022203152B2 (ja)
CA (1) CA3158030A1 (ja)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024142668A1 (ja) * 2022-12-26 2024-07-04 株式会社クボタ 管理装置、管理方法、コンピュータプログラム、及び管理システム

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118062041B (zh) * 2024-02-23 2024-07-23 盐城工业职业技术学院 一种农用拖拉机交互控制系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9141109B1 (en) * 2014-05-06 2015-09-22 Toyota Motor Engineering & Manufacturing North America, Inc. Automated driving safety system
CN111596652A (zh) * 2019-12-30 2020-08-28 山东省科学院自动化研究所 一种农药喷洒智能农机寻路导航控制方法及系统
US20200393842A1 (en) * 2019-06-17 2020-12-17 Toyota Research Institute, Inc. Systems and methods for training a vehicle to autonomously drive a route

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102012221572A1 (de) * 2012-11-26 2014-05-28 Robert Bosch Gmbh Autonomes Fortbewegungsgerät
CN111016926B (zh) * 2014-12-12 2023-06-13 索尼公司 自动驾驶控制设备以及自动驾驶控制方法和程序
JP6514019B2 (ja) * 2015-04-22 2019-05-15 三菱マヒンドラ農機株式会社 支持装置
EP3427562B1 (en) * 2016-03-09 2020-09-02 Yanmar Co., Ltd. Travel region specification device
JP6605364B2 (ja) 2016-03-09 2019-11-13 ヤンマー株式会社 作業車両の経路生成方法
CN205594450U (zh) * 2016-04-22 2016-09-21 徐州斯塬农科网络科技有限公司 北斗农机自动导航驾驶控制系统
JP6631567B2 (ja) * 2017-03-10 2020-01-15 オムロン株式会社 自動運転支援装置、方法及びプログラム
CN108860142B (zh) * 2018-04-27 2020-11-13 榛硕（武汉）智能科技有限公司 车辆用半自动驾驶系统的驾驶方法
JP6959893B2 (ja) * 2018-06-06 2021-11-05 本田技研工業株式会社 車両制御システム
JP7357444B2 (ja) * 2018-08-01 2023-10-06 株式会社クボタ 収穫機
CN109358625A (zh) * 2018-10-25 2019-02-19 丰疆智慧农业股份有限公司 自动驾驶农机及其动力系统
CN109588107A (zh) * 2018-12-29 2019-04-09 丰疆智慧农业股份有限公司 收割机及其自动驾驶方法
CN109631903A (zh) * 2019-01-04 2019-04-16 丰疆智慧农业股份有限公司 谷物处理自动驾驶系统及其自动驾驶方法和路径规划方法
JP7132191B2 (ja) * 2019-08-08 2022-09-06 ヤンマーパワーテクノロジー株式会社 作業車両用の自動走行システム
JP2021040603A (ja) * 2019-09-13 2021-03-18 株式会社 神崎高級工機製作所 自動走行機能付き芝刈車両
CN112462749B (zh) * 2020-05-12 2023-06-06 丰疆智能科技股份有限公司 农机自动导航方法、农机自动导航系统以及农机

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9141109B1 (en) * 2014-05-06 2015-09-22 Toyota Motor Engineering & Manufacturing North America, Inc. Automated driving safety system
US20200393842A1 (en) * 2019-06-17 2020-12-17 Toyota Research Institute, Inc. Systems and methods for training a vehicle to autonomously drive a route
CN111596652A (zh) * 2019-12-30 2020-08-28 山东省科学院自动化研究所 一种农药喷洒智能农机寻路导航控制方法及系统

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
"Reina. G, etc., 'Ambient Awareness for Agricultural Robotic Vehicles', 2016" (Year: 2016) *
Machine Translation of CN109631903A (Year: 2019) *
Machine Translation of CN111596652A (Year: 2020) *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024142668A1 (ja) * 2022-12-26 2024-07-04 株式会社クボタ 管理装置、管理方法、コンピュータプログラム、及び管理システム

Also Published As

Publication number Publication date
JP2022176132A (ja) 2022-11-25
JP7390428B2 (ja) 2023-12-01
EP4094558A1 (en) 2022-11-30
CN114115214B (zh) 2024-04-05
CA3158030A1 (en) 2022-11-14
AU2022203152A1 (en) 2022-12-01
CN114115214A (zh) 2022-03-01
AU2022203152B2 (en) 2023-11-09

Similar Documents

Publication Publication Date Title
US20220361392A1 (en) Method and system for driving view-based agricultural machinery and device for agricultural machinery applying method
US11002840B2 (en) Multi-sensor calibration method, multi-sensor calibration device, computer device, medium and vehicle
US10696301B2 (en) Vehicle control device
CN108609014B (zh) 智能车辆的横向控制方法及装置
US11402842B2 (en) Method to define safe drivable area for automated driving system
US10378896B2 (en) Method and system for planning the path of an agricultural vehicle
CN110606071A (zh) 一种泊车方法、装置、车辆和存储介质
US20200231142A1 (en) Fail-operational architecture with functional safety monitors for automated driving system
US20070083299A1 (en) Method and system for determining the path of a mobile machine
CN105700533A (zh) 基于北斗导航的农业机械自动驾驶控制系统及方法
US20200233418A1 (en) Method to dynamically determine vehicle effective sensor coverage for autonomous driving application
CN109318899B (zh) 自动驾驶车辆的弯道行驶方法、装置、设备和存储介质
US20210276433A1 (en) System for automated charging of autonomous vehicles
CN107728646B (zh) 对自动驾驶车辆的摄像头进行自动控制的方法和系统
CN112834249B (zh) 转向参数检测方法、装置、设备及存储介质
CN110221330A (zh) 自移动设备及其方向确定方法、自动工作系统
US20220279700A1 (en) Method, apparatus, and computer program for defining geo-fencing data, and respective utility vehicle
WO2022141240A1 (en) Determining vehicle positions for autonomous driving based on monocular vision and semantic map
US20230320263A1 (en) Method for determining information, remote terminal, and mower
CN116576859A (zh) 路径导航方法、作业控制方法及相关装置
CN112665506A (zh) 定位装置的安装偏差检测方法、装置、设备及存储介质
US20230210050A1 (en) Autonomous mobile device and method for controlling same
EP3806044A1 (en) Data processing method, apparatus and device, and mobile platform
CN115891984A (zh) 一种智能驾驶系统、方法、设备及介质
CN116736865A (zh) 信息确定方法、远程终端、设备、割草机及存储介质

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED