NL1043784B1

NL1043784B1 - Harvest robot system

Info

Publication number: NL1043784B1
Application number: NL1043784A
Authority: NL
Inventors: Moriyama Toshiki; Toshima Ryou; Kanata Yoshio
Original assignee: Panasonic Ip Man Co Ltd
Priority date: 2019-09-13
Filing date: 2020-09-09
Publication date: 2023-03-14
Also published as: NL1043784A; JP2021040589A; JP7223977B2

Abstract

Provided is a harvest robot system including a harvest robot that harvests a target object in a farm, and a server configured to communicate with the harvest robot. The server inciudes an input/output part that receives an input of a given harvest condition including farm map information of the farm and harvest determination reference information for determining a harvest time of the target object, a determiner that determines a harvest target area based on an image of the target object imaged for each unit harvest area in the farm by the harvest robot and the given harvest condition, and a first transceiver that transmits harvest instruction information including position information of the harvest target area to the harvest robot.

Description

ref: P 2020 NL 012

TITLE: HARVEST ROBOT SYSTEM

BACKGROUND

1. Technical Field

[0001] The present disclosure relates to a harvest robot system that automatically harvests crops including fruits such as apples, fruits and vegetables such as tomatoes, and fruit-like vegetables such as strawberries based on cultivation management information. 2. Description of the Related Art

[0002] As a future social issue in the Japanese society, there is concern that the aging population will reduce the production population. In particular, regarding the primary industry, agriculture, the environment surrounding Japan is particularly remarkable for aging, and the situation is extremely severe in terms of labor shortage. According to the Ministry of Agriculture, Forestry and Fisheries basic data, is there were 1.68 million key farmers in 2014, a decrease of about 230,000 in five years. Further, the average age is 66.5, this figure highlights that the number of new farmers is also decreasing and the labor shortage due to aging. As a result, abandoned cultivated land reaches 400,000 hectares, which has a bad influence on the local farming environment and living environment. Especially in rural areas where there are many cultivated lands, depopulation due to declining birthrate and aging is progressing, and this problem is becoming more apparent because there is no farmer.

[0003] Under such circumstances, there is an increasing expectation that the automation of agriculture will address the labor shortage. According to the

Ministry of Economy, Trade and Industry's “2012 Robot Industry Market Trends”, the domestic market for agricultural-related robots is expected to grow significantly between 2018 and 2024, and is expected to reach about 220 billion yen. In fact, developments that lead to automation of agriculture, such as management using drones, automatic driving of tractors, and systems navigating cultivation, are also becoming active. Further, the Ministry of Agriculture, Forestry and Fisheries has enhanced the subsidy system to support the automation of agriculture, and the expectations of this field from the government are very high.

[0004] In the meantime, research on the technology for automation of harvesting has also been advanced in companies and universities in the related art,

and many harvest robots that automatically harvest various fruits, fruit-like vegetables, and fruits and vegetables have been proposed.

[0005] Among them, as a system for gathering and managing cultivation status information in the related art, there is a system in which a plurality of monitoring devices are installed in a farm and one server is installed in a remote place, and the devices and server are connected via the internet (for example, see

Japanese Patent No. 4586182). In an independent operation control system in the related art that gathers the cultivation status information in the farm described in

Japanese Patent No. 4586182, the monitoring device has an imaging part and captures an image of a predetermined crop. The server is connected to the plurality of monitoring devices via the Internet, and an operator can view images of crops in remote places through a terminal,

SUMMARY

[0006] A harvest robot system according to the present disclosure to achieve the above object, includes a harvest robot that harvests a target object in a farm, and a server configured to communicate with the harvest robot, in which the server includes an input/output part that receives a given harvest condition including farm map information of the farm and harvest determination reference information for determining a harvest time of the target object, a determiner that determines a harvest target area based on an image of the target object imaged for each unit harvest area in the farm by the harvest robot and the given harvest condition, and a first transceiver that transmits harvest instruction information including position information of the harvest target area to the harvest robot.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 is an overall system configuration diagram of a harvest robot system according to an exemplary embodiment of the present disclosure;

FIG. 2 is a block diagram of a server according to the exemplary embodiment of the present disclosure;

FIG. 3 is a configuration diagram of a harvest robot according to the exemplary embodiment of the present disclosure;

FIG. 4 is a configuration diagram of a collection robot according to the exemplary embodiment of the present disclosure;

FIG. 5 is a block diagram of a collection station according to the exemplary embodiment of the present disclosure;

E 3

FIG. 6 is a block diagram of a home station according to the exemplary embodiment of the present disclosure;

FIG. 7 is a schematic layout diagram of an entire farm according to the exemplary embodiment of the present disclosure;

FIG. 8 is a flow chart of a basic operation of harvesting, collecting, and supplying a storage basket according to the exemplary embodiment of the present disclosure;

FIG. 9 is a flow chart of a tool change operation according to the exemplary embodiment of the present disclosure;

FIG. 10 is a flow chart of a charging operation according to the exemplary embodiment of the present disclosure;

FIG. 11 is a flow chart of a harvesting target area determination according to the exemplary embodiment of the present disclosure;

FIG. 12 is a flow chart of an imaging rotation operation according to the exemplary embodiment of the present disclosure;

FIG. 13 is a flow chart of a degree of maturity determination with learning function applied according to the exemplary embodiment of the present disclosure;

FIG. 14 is a flow chart for a manual programming according to the exemplary embodiment of the present disclosure;

FIG. 15 is a flow chart of a harvest prediction management process according to the exemplary embodiment of the present disclosure;

FIG. 16 is a flow chart of a real-time feedback process according to the exemplary embodiment of the present disclosure;

FIG. 17 is a flow chart of a cultivation management process according to the exemplary embodiment of the present disclosure; and

FIG. 18 is a flow chart of another farm cooperation system operation according to the exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

[0008] In the related art, there are few proposals for a harvest robot system in which a cultivation management system, which performs how to automatically obtain the cultivation status information in a farm and how the harvest robot determines an area to be harvested, and a harvest robot are linked. For example, in the configuration of Japanese Patent No. 4586182 in the related art described above, there are problems that an imaging place is fixed due to a fixed installation of the monitoring device, the entire farm cannot be covered due to a limitation of the number of monitoring devices, thereby maturity status information of the crop becomes fragmentary, and the optimum harvest schedule cannot be established for the entire farm based on limited information. Further, when a wide range can be imaged, the details of the crops cannot be seen due to the limitation of the resolution of an imaging part. In particular, there is a problem that the goal cannot be achieved because the degree of maturity of small individual fruits cannot be determined with the crops in which the fruits that are connected to one main stem or one tuft are gradually matured and harvested at different times. Furthermore, the configuration of the above-mentioned Japanese Patent No. 4586182 in the related art has only a monitoring function, and no cooperation with the harvest robot is made. That is, itis difficult to operate the harvest robot efficiently.

[0009] The present disclosure is to solve the above-mentioned problems in the related art, and an object of the present disclosure is to provide a harvest robot system in which a harvest robot and a cultivation management system where a wide farm is covered all over and an optimal harvest schedule is established by obtaining detailed maturity status information of individual crops, are combined, and the harvest robot is efficiently operated.

[0010] Hereinafter, an exemplary embodiment of the present disclosure will be described with reference to the drawings.

EXEMPLARY EMBODIMENT

[0011] FIG. 1 is an overall system configuration diagram of a harvest robot system according to an exemplary embodiment of the present disclosure.

[0012] In FIG. 1, server 101 is connected to a plurality of harvest robots 102, one or more collection robots 103, collection station 104, home station 105 on which harvest robot 102 or collection robot 103 stands by, input/output unit 107 through which operator 106 inputs and outputs, and further Internet 109 that can be connected to server 108 of another farm in a wired or wireless manner, and is organically related to constitute the harvest robot system.

[0013] FIG. 2 is a block diagram of server 101.

[0014] As illustrated in FIG. 2, server 101 includes first transceiver 201, input/output part 202, determiner 203, and mass storage device 204, First transceiver 201 transmits and receives information to and from harvest robot 102 and collection robot 103 via a wireless LAN or the like. Inputfoutput part 202 inputs or outputs information with respect to collection station 104, home station 105, input/output unit 107, and server 108 of another farm in a wired or wireless manner.

Determiner 203 determines a harvest target area, determines a collection order and a collection route of the stored storage basket, and determines an empty storage basket supply dispatch plan, based on image data with a map address and a given harvest condition. Mass storage device 204 can store a large amount of information including image data with a map address.

[0015] FIG. 3 is a configuration diagram of harvest robot 102 according to the exemplary embodiment of the present disclosure.

[0016] As illustrated in FIG, 3, harvest robot 102 includes imaging part 301, first self-position detector 302, second transceiver 303, harvester 304, first lifter mechanism 305, third lifter mechanism 309, and self-propelled carriage part 308.

Imaging part 301 images target object 308 and a storage basket with a camera. First self-position detector 302 specifies a self-position by GPS, a laser radar, a white line marker, landscape recognition collation, or the like. Second transceiver 303 sends information including image data with a map address to a server via a wireless LAN i0 or the like, and receives instruction information from the server. Harvester 304 collects target object 306 with a manipulator and stores the target object 306 in storage basket 307. First lifter mechanism 305 grips and lifts lowermost storage basket 307 of stacked storage baskets 307 placed on the ground. Third lifter mechanism 309 sequentially grips, lifts, and separates empty storage baskets 307 above the second level from the bottom of stacked storage baskets 307. Self- propelled carriage part 308 is equipped with all of the above-described apparatuses and can automatically travel along ridge road 705 and main passage 703.

[0017] Imaging part 301 is desirably a stereo camera. A normal two- dimensional area camera may be used when it is used only to acquire image data with a map address for the harvest target area determination, but in the exemplary embodiment of the present disclosure, imaging part 301 is also used during the harvest work or picking up storage basket 307. Information on a distance to target object 306 is required for harvesting, and information on a distance to storage basket 307 is required to pick up storage basket 307. When a normal two-dimensional area camera is used, a distance measuring device such as a laser sensor is required separately.

[0018] The form of harvester 304 differs depending on the type of target object 306 to be harvested. In the related art, various harvest robots, harvesting manipulators, and harvesting hands have been proposed, and a detailed description thereof will be omitted. FIG. 3 shows an example in which a double-armed joint type manipulator is attached on a rotatable body part of harvester 304, and a harvesting hand is mounted on a tip of the manipulator. For the harvesting hand, the most suitable harvesting hand is selected according to the type, shape, size, and the like of target object 306, and is appropriately mounted. In FIG. 3, the manipulator for collecting target object 306 is configured to store target object 308 in storage basket 307 as itis, but it may be divided into the manipulator for collecting target object 306 and a storage mechanism for storing target object 306 in storage basket 307. In that case, harvester 304 is formed by combining the collecting manipulator and the storage mechanism.

[00181 Storage basket 307 is, for example, a box of a so-called container type, which is made of plastic and can be stacked in a state in which target objects are stored.

[0020] Further, self-propelled carriage part 308 is generally called an automatic guided vehicle (AGV), and a detailed description thereof will be omitted.

Although FIG. 3 illustrates an example in which the vehicle travels on wheels, a crawler, or the like may be employed corresponding to the characteristics of the farm.

[0021] FIG. 4 is a configuration diagram of collection robot 103 according to the exemplary embodiment of the present disclosure.

[0022] As illustrated in FIG. 4, collection robot 103 includes second lifter mechanism 401 having the same function as first lifter mechanism 305, second self- position detector 402 having the same function as first self-position detector 302, third transceiver 403 having the same function as second transceiver 303, and self- propelled carriage part 404 equipped with all of the above devices and capable of automatically traveling in main passage 703.

[0023] FIG. 5 is a block diagram of collection station 104 according to the exemplary embodiment of the present disclosure.

[0024] As illustrated in FIG. 5, collection station 104 includes harvested storage basket stock part 501 and empty storage basket stock part 502. Harvested storage basket stock part 501 receives a plurality of harvested storage baskets 307 from collection robot 103 and can stock the plurality of harvested storage baskets 307, for example, on a roller conveyor. Empty storage basket stock part 502 can stock a plurality of empty storage baskets 307 on, for example, a roller conveyor, and can deliver empty storage basket 307 to collection robot 103 at a request from server 101 or collection robot 103.

[9025] Collection station 104 further includes a fourth transceiver (not illustrated) that transmits the empty space information of harvested storage basket stock part 501 to server 101, and transmits the stock information including a stock amount of the empty storage basket existing in empty storage basket stock part 502 to server 101.

[0026] FIG. 6 is a block diagram of home station 105 according to the exemplary embodiment of the present disclosure.

[0027] Home station 105 is a location in which harvest robot 102 or collection robot 103 is moored when the harvest work or the collection work is not performed and is generally called a dock, and a plurality of parking spaces are prepared.

[0028] As illustrated in FIG, 6, home station 105 includes tool change part 601, charging part 602, and a fifth transceiver (not illustrated). Tool change part 801 replaces a tool at a tip of the manipulator for harvesting in harvester 304 of harvest robot 102, with a tool used for harvesting another type of a target object or for other work. Charging part 602 charges a driving rechargeable battery of harvest robot 102 or collection robot 103. The fifth transceiver transmits/receives data to/from server 101.

[00298] FIG. 7 ís a schematic layout diagram of the entire representative farm according to the exemplary embodiment of the present disclosure. In FIG. 7, the same components as those in FIGS. 1 and 5 are denoted by the same reference numerals, and the description thereof will be omitted.

[0030] As illustrated in FIG, 7, the farm includes cultivation yard 701 for cultivating crops and sorting shipping yard 702 for sorting and bagging the harvested crops for shipment, Cultivation yard 701 may be an open field or a greenhouse such as a vinyl house. Further, cultivation yard 701 and sorting shipping yard 702 may be directly linked or may be linked by a passage.

[0031] In a center of cultivation yard 701, there is a relatively wide main passage 703 through which harvest robot 102 and collection robot 103 travel. Main passage 703 is directly linked to sorting shipping yard 702. On both sides of main passage 703, several ridges 704 for cultivating crops such as tomatoes and strawberries are vertically disposed, respectively. Ridge 704 may be a ridge with literally raised soll, a cultivation box for elevated cultivation found in Dutch Venlo-type facility cultivation, or a row of standing fruit frees. The area in which target object 306 to be harvested and cultivated on ridge 704 is present is the harvest area. There is ridge road 705 between ridge 704 and ridge 704, so that harvest robot 102 can pass through. Harvest robot 102 can harvest target object 306 from ridges 704 on both sides. Ridge road 705 may be on the soil surface or, in the case of a greenhouse, a hot water pipe may be laid. When a hot water pipe is laid, harvest robot 102 can use the hot water pipe as a track for traveling on a ridge road. As illustrated in FIG, 7, an end of ridge road 705 opposite to main passage 703 may be a dead end. After harvesting, harvest robot 102 turns back ridge road 705, brings down harvested storage basket 307 once exiting main passage 703, and heads for next ridge road 705. Al this time, the efficiency of harvest robot 102 is high when, for example, the outward path is for the harvest of right ridge 704 and the return path is for the harvest of left ridge 704. Note that many farms have a dead end on a side opposite to the ridge road as illustrated in FIG. 7. This is to increase the area productivity of cultivation yard 701.

[0032] Storage basket 307 storing harvested target object 306 or empty storage basket 307 is temporarily placed at a position at which it does not interfere with other work of main passage 703, for example, at a ridge end on the side of main passage 703.

. &

[0033] Harvested storage basket 307 temporarily placed by harvest robot 102 beside main passage 703 is collected by collection robot 103, transported to harvested storage basket stock part 501 of collection station 104, and stocked.

Harvested storage basket stock part 501 stocks storage basket 307 and transports storage basket 307 to the vicinity of sorting bagging worker 708 by, for example, a roller conveyor. There is de-stacking unit 708 on the way, and stacked storage baskets 307 are separated one by one and transported to the place of sorting bagging worker 7086.

[0034] Sorting bagging worker 708 takes out target objects 306 from storage basket 307, sorts target objects 306, stores target objects 306 in a bag according to a predetermined procedure, and place the bag on shipping conveyor 707. The bag containing target objects 306 is transported to the next processing on a conveyor, after which the bag is packed in a box and shipped.

[0035] On the other hand, empty storage basket 307 is placed on a conveyor linked to empty storage basket stock part 502 by sorting bagging worker 706. Empty storage basket 307 is transported to empty storage basket stock part 502 by the conveyor. There is stacking unit 709 on the way, and here, empty storage baskets 307 are stacked in a predetermined number of trays and then transported to empty storage basket stock part 502 to be stocked. Thereafter, in response to a request from server 101, empty storage basket 307 is transported to cultivation yard 701 by collection robot 103, and is placed at a specified location beside main passage 703. Harvest robot 102 picks up placed empty storage basket 307 and heads for harvesting. In this way, the circulation distribution system of storage basket 307 functions.

[0036] Server 101 and input/output unit 107 are installed at an appropriate location in the farm.

[0037] Home station 105 is installed in a space close to main passage 703 and can moor a plurality of harvest robots 102 or collection robots 103. The tool is replaced, and the rechargeable battery is charged at the moored location.

[0038] Since a plurality of harvest robots 102 and collection robots 103 move back and forth in main passage 703, a basic traveling route and a rule in main passage 703 may be determined. For example, main passage 703 is divided into two lanes, which limit the advancing directions, respectively. That is, the outward path and the return path are completely separated so that passing each other can always be done, if is desirable that the basic traveling route of collection robot 103 is turned back at an end of main passage 703 and in sorting shipping yard 702, the robots travel in the order of main passage 703, harvested storage basket stock part 501, empty storage basket stock part 502, and main passage 703, and a closed circuit is constructed. The plurality of collection robots 103 circulate in one direction along the closed circuit. The traveling route may be indicated by a white line, may be defined by the farm map information, or a magnetic tape may be attached.

[0039] Collection robot 103, which has completed the installation of empty storage basket 307 and the collection of harvested storage basket 307 in a location near sorting shipping yard 702, does not trace the closed circuit to the end and makes a shortcut to head toward collection station 104. This route instruction is also issued from server 101.

[0040] The form of the farm is not limited to the form illustrated in FIG. 7. A form may be such that there is main passage 703 in which collection robot 103 can goto collection station 104 and harvest robot 102 can come out of the ridge road after finishing harvest at the ridge to main passage 703, as long as there is a contact point that can be shared by harvest robot 102 and collection robot 103 on main passage 703.

[0041] Further, the form of harvested storage basket stock part 501 and empty storage basket stock part 502 of collection station 104 is a roller conveyor, but is not limited to the form. For example, the form may be a simple space, in which only storage basket 307 is placed, divided into each. In that case, sorting bagging worker 706 goes to the space to take the storage basket, and after the work is completed, returns to the empty storage basket stock space. Collection robot 103 detects a position of an empty storage basket in a predetermined space by using, for example, an imaging device, and picks up the storage basket with second lifter mechanism 401.

[0042] Further, in the exemplary embodiment of the present disclosure in

FIG. 7, the sorting and shipping work is performed by the worker, but may be an automatic sorting bagging apparatus. In addition, for some fruits, the sorting and shipping work has been automated. In addition to the present exemplary embodiment, those automatic sorting bagging apparatus and automatic boxing apparatus may be further introduced and connected to server 101 as a shipping system, and server 101 may manage a shipping plan in addition to a harvest plan and a collection plan, thereby an automatic harvesting and shipping system can be built, and the unmanned series of harvest works will be further advanced.

[0043] Further, when sorting bagging worker 708 works only in the daytime and on the other hand, harvest robot 102 and collection robot 103 operate day and night, a shift difference occurs, so that harvested storage basket stock part 501 and empty storage basket stock part 502 require a conveyor length that can stock the number of storage baskets that can absorb the shift difference. Further, a large number of storage baskets 307 are required. As described above, when all the work including the sorting shipping are automated, the shift difference is eliminated, and

E 10 the operation can be performed with a shorter conveyor length and the small number of storage baskets.

[0044] An operation example of the harvest robot system configured as described above will be described with reference to FIG. 8.

[0048] FIG. 8 is a basic operation flow chart of harvesting, collecting, and supplying an empty storage basket according to the exemplary embodiment of the present disclosure.

[0046] As illustrated in FIG. 8, first, server 101 receives an input of a given harvest condition ($1). The given harvest condition is input to input/output part 202 by operator 106 operating input/output unit 107.

[0047] The “given harvest condition” input by operator 108 is information that should be given in advance for determiner 203 to determine the harvest target area, and includes farm map information and harvest determination reference information for determining the harvest time of the target object. The harvest determination reference information is information related to a threshold value for determining whether target object 306 should be harvested or should not be harvested by using the image of target object 306 captured by harvest robot 102.

Further, the farm map information is, for example, information related to cultivation yard 701 in the farm, the travelable area, the position of collection station 104, the position of home station 105, and the like.

[0048] In the farm map information, cultivation yard 701 is divided into a plurality of unit harvest areas, and address information is added to each unit harvest area and stored. In the harvest robot system according to the present exemplary embodiment, the harvest target area is determined for each unit harvest area.

[0048] Next, determiner 203 determines a route to be imaged by harvest robot 102 based on the farm map information or the like (82). Next, server 101 outputs imaging instruction information including the imaging route to harvest robot 102 through first transceiver 201.

[0080] Harvest robot 102, which has received the imaging instruction information through second transceiver 303, images target object 308 in the farm using imaging part 301 while moving on self-propelled carriage part 308 according fo the instruction (R1). The self-position of harvest robot 102 at the time of imaging is acquired by first self-position detector 302. The self-position may be farm coordinate data or an address code defined in a farm map. When imaging part 301 is nota fixed camera but a camera that can be freely turned, image data with a map address includes the self-position in the farm and the posture information of the camera. That is, it is necessary to specify the absolute position of target object 308 reflected in the image data within cultivation yard 701. Harvest robot 102 transmits the acquired image data with a map address to server 101 via second transceiver 303.

[0051] The “image data with a map address” is an image of target object 3086 existing in the unit harvest area imaged by harvest robot 102 for each unit harvest area in the farm. The "image data with a map address” is typically imaged so that the entire unit harvest area is reflected such that the maturity of target object 306 existing in the unit harvest area can be grasped from the image (see FIG. 12 below).

However, the maturity of target object 306 existing in the entire unit harvest area may be grasped by the plurality of images.

[0052] The "image data with a map address” stores, for example, information related to a position (for example, the self-position of harvest robot 102) in which the image is captured and a direction (for example, the posture of harvest robot 102) in which the image is captured, in association with the image. The position information stored in association with the image may be position information related to the position of the imaged target specified based on the position where the image is captured and the direction in which the image is captured.

[0053] Determiner 203 of server 101, which has received the image data with a map address in the farm from harvest robot 102 in first transceiver 201, determines an area including a predetermined amount or more of mature target objects 306 as a harvest! target area based on the given harvest condition input in advance (83). When a plurality of harvest target areas are determined in 53, the plurality of harvest target areas may be ranked in descending order of the number of mature target objects 306. Specifically, the number of ridge road 705 facing the determined harvest target area is allocated to each of harvest robots 102 in consideration of the harvest ability of harvest robot 102. Basically, one harvest robot 102 is assigned to one ridge road 705. itis a principle that only one harvest robot 102 is allowed to enter one ridge road 705 at a certain moment. Harvest robot 102 may be assigned to a plurality of ridge road 705, All ridges 704 on both sides of ridge road 705 may be the harvest target area, or only one part of ridge 704 on one side may be the harvest target area. These are the harvest instruction information.

[0054] First transceiver 201 of server 101 transmits the harvest instruction information including the harvest target area to harvest robot 102.

[0055] In the present exemplary embodiment, as described above, the harvest target area to be harvested by harvest robot 102 is determined by determiner 203 of server 101 based on the image data with a map address of the harvested object and the given harvest condition, but operator 106 may visually check the growth state of the harvest target object in the farm and determine the harvest target area based on the experience in the related art, thereafter the number and the harvest tract of ridge road 705 to be harvested by each harvest robot 102 may be directly input through input/output unit 107 to serve as harvest Instruction information.

[0056] Harvest robot 102 enters specified ridge road 705 by self-propelled carriage part 308 based on the harvest instruction information received by second transceiver 303, makes full use of harvester 304 to harvest target object 306 in the harvest target area, and store it in storage basket 307 (R2). it is assumed that harvest robot 102 has first lifter mechanism 305 gripping lowermost storage basket 307 of empty storage baskets 307 stacked in advance. This inherits a state of RS described later. When harvest robot 102 receives the harvest instruction information, third lifter mechanism 309 grips and lifts storage basket 307 at the second level from the bottom in the state. Then, space is formed above lowermost storage basket 307.

Using the space, harvester 304 stores target object 306 in lowermost storage basket 307. When lowermost storage basket 307 is full, third lifter mechanism 309 descends, releases the grip of storage basket 307 at the second level from the bottom after all storage baskets 307 are once stacked, ascends only by one height of storage basket 307, and grips storage basket 307 at the third level from the bottom and ascends. As a result, harvester 304 resumes the harvest operation using the space formed above storage basket 307 at the second level from the bottom, and contains target object 306 in storage basket 307 at the second level from the bottom.

FIG. 3 illustrates the state at this ime. Hereinafter, this operation is repeated, and harvest robot 102, in which all placed storage baskets 307 are full with target objects 306 or the harvesting of the planned harvest target area is completed, transmits harvest status information including a harvest completion signal and current position information from second transceiver 303 to server 101. [00571 Server 101, which has received the harvest status information including the harvest completion signal from first transceiver 201, sets new harvest instruction information to harvest robot 102 to head to the next harvest target area when there is still empty space in placed storage basket 307. When all placed storage baskets 307 are full with target objects 306, information for collecting the storage basket is collected (84). The given collection condition necessary for establishing the collection plan of harvested storage basket 307 has been input to inputioutput part 202 by operator 108 through input/output unit 107 in advance, The given collection condition is farm map information including a position of collection station 104, and is a collection robot specification including the number, travel speeds, or the like of collection robots 103 being operated. Empty space information of harvested storage basket stock part 501 is obtained from collection station 104 via inputfoutput part 202, and it is checked whether collection station 104 has room to accept harvested storage basket 307. However, harvested storage basket stock part 501 is designed so that there is always enough space, and at the worst case, when collection robot 103 can wait for loading and unloading of harvested storage basket 307 in front of collection station 104, the empty space information is not necessary.

[0058] Next, server 101 transmits a request signal to collection robot 103 from first transceiver 201 to understand the status of collection robot 103.

[6059] Collection robot 103, which has received the request signal at third transceiver 403, detects a self-pasition by second self-position detector 402, and transmits its self-position data, whether a storage basket is currently loaded, currently-performed work content, for example, whether charging is done at home station 105, and loading status information such as whether harvested or empty, or which storage basket 307 is on the way to somewhere for example, from third transceiver 403 to server 101 (K1).

[0060] In this way, server 101, which has obtained all the information necessary for the collection work of harvested storage basket 307, allows determiner 203, based on the information, to determine which position harvest robot 102 that has completed the harvest work brings down harvested storage basket 307 at, and which collection robot 103 determines which route to go to collect storage basket 307 placed by harvest robot 102 and which route to go to collection station 104 (85).

These are collection instruction information, and first transceiver 201 outputs the information to harvest robot 102.

[0081] Harvest robot 102, which has received the collection instruction information from server 101 at second transceiver 303, moves, by self-propelled carriage part 308, to the position instructed by the collection instruction information, and upon arrival, lowers first lifter mechanism 305 to bring down harvested storage basket 307 to the ground (R3}). When harvest robot 102 transmits the fact that storage basket 307 has been brought down from second transceiver 303 to server 101 {harvest status information including a completion signal and current position information), server 101 sends the collection instruction information from first transceiver 201 to collection robot 103 (86). Based on the collection instruction information received by third transceiver 403, collection robot 103 heads to the instructed location by self-propelled carriage part 404, and picks up harvested storage basket 307 placed on the ground by raising second lifter mechanism 401 (K2).

[0062] The collection instruction information transmitted to the harvest robot and the collection instruction information transmitted to the collection robot are the same information in here, but the collection instruction information transmitted to the harvest robot may include information on the placement position at which the full storage basket is placed separately from itself, and the collection instruction information transmitted to the collection robot may include information on a collection route for collecting the storage basket placed by the harvest robot and transporting the basket to the collection station.

E 14

[0063] Thereafter, collection robot 103 moves to collection station 104 through the route instructed by server 101 by self-propelled carriage part 404 (K3), and brings down storage basket 307 in harvested storage basket stock part 501 of collection station 104 by lowering second lifter mechanism 401 (K4).

[0064] Next, third transceiver 403 of collection robot 103 transmits the loading status information including the completion signal and the current position information to server 101, Server 101, which has received the loading status information including the completion signal, or the like of collection robot 103 by first transceiver 201, obtains stock information of empty storage basket stock part 502 of collection station 104 (87). At this time, when there is a sufficient storage basket distribution amount and a sufficient empty storage basket stock capacity and there is no stock at the worst case, obtaining the stock information can be omitted when it is acceptable to allow collection robot 103 to be in a standby state in front of collection station 104.

[00658] When there is a stock of empty storage baskets, determiner 203 determines an empty storage basket supply dispatch plan for loading empty storage basket 307 and directing where collection robot 103 heads to (88). These are the storage basket supply instruction information, and server 101 transmits the storage basket supply instruction information from first transcelver 201 to collection robot 103. Collection robot 103, which has received the storage basket supply instruction information at third transceiver 403, goes to empty storage basket stock part 502 of collection station 104 by self-propelled carriage part 404, and lifts and places empty storage basket 307 by second lifter mechanism 401 (K5).

[0068] Thereafter, collection robot 103 moves to the location instructed by server 101 by self-propelled carriage part 404, and upon arrival, second lifter mechanism 401 descends, and empty storage basket 307 is brought down (K8).

Collection robot 103 transmits the loading status information including the completion signal and the current position information from third transceiver 403 to server 101.

[0067] Server 101, which has received the loading status information including the completion signal and the current position information at first transceiver 201, transmits the storage basket supply instruction information to harvest robot 102 by first transceiver 201 {39}.

[0068] Harvest robot 102, which has received the storage basket supply instruction information at second transceiver 303, moves to the specified location by self-propelled carriage part 308, and lifts and holds empty storage basket 307 placed on the ground by first lifter mechanism 305 (R4). Harvest robot 102 waits until receiving the next instruction from server 101 (R58). This state is a state at the time of imaging the harvest area of R1 or immediately before shifting to the harvesting/storage operation of R2. Thereafter, this is repeated.

~ 15

[9069] The above operation flow is a basic pattern, and in practice, a plurality of harvest robots 102 and a plurality of collection robots 103 operate in a complex manner according to a harvest status, and are operated to run efficiently as a whole.

[0070] For example, in the operation flow chart of FIG. 8, the operations of one harvest robot 102 and one collection robot 103 are separated into a harvest basic flow, a collection basic flow, and an empty storage basket supply basic flow for easy understanding, but actually, the operations may be operated as one unit. That is, by constantly understanding states of the plurality of harvest robots 102 and collection robots 103 one by one, server 101 determines a harvest target area, determines a collection order and a collection route, and performs an empty storage basket supply dispatch plan, simultaneously. For example, when a point at which the harvested storage basket 307 is to be collected at any point in time is relatively close to a point at which empty storage basket 307 is to be supplied, collection robot 103 supplies empty storage basket 307 on an outward path from collection station 104 and collects harvested storage basket 307 on a return path to collection station 104, so that collection robot 103 hardly has a section in which it travels with an empty load, and a wasteful waiting time does not occur for collection robot 103. Similarly, after bringing down harvested storage basket 307, when harvest robot 102 immediately goes to a nearby empty storage basket placement point, picks up storage basket 307, enters into ridge road 705 of the next harvest target area instructed by server 101, and starts the harvest work, unnecessary waiting time is not generated also in harvest robot 102,

[0071] There is no pairing relationship between harvest robot 102 and collection robot 103, and since storage basket 307 is once placed on the ground, harvest robot 102 and collection robot 103 do not have to synchronize the delivery operation of storage basket 307. Furthermore, there may be a considerable time lag from the placement of harvested storage basket 307 of harvest robot 102 to the pick- up by collection robot 103, so that server 101 can develop such efficient operation programming. As a result, a minimum number of collection robots 103 can be operated with respect to the number of harvest robots 102. Basically, a harvest span of harvest robot 102 from the start of harvest to the completion of the harvest is long, and on the other hand, a collection supply span of collection robot 103 from the collection of harvested storage basket 307 to the installation of empty storage basket 307 is short.

[0072] Inthe present harvest robot system, a harvest plan, a collection plan, and the storage basket supply plan are optimally programmed by server 101 so that harvest robot 102 and collection robot 103 do not wait. A game theory, a queuing theory, or the like may be applied to the process.

[0073] Further, a harvest schedule may be programmed by applying Al or the like and making some predictions in advance. However, unlike industrial products, craps do not grow uniformly, and vary greatly depending on the weather and other factors. A prediction may be made at first, but as the prediction is wrong, a feedback mechanism that constantly changes the program on the fly with the latest information is necessary.

[0074] Further, when harvest robot 102 has third lifter mechanism 309 as illustrated in FIG, 3, a plurality of storage baskets 307 can be placed on harvest robot 102, and more harvested target objects 306 can be stored at one time. That is, the collection lot size of collection robot 103 increases, and the collection frequency of storage basket 307 of collection robot 103 decreases. That is, the number of necessary collection robots 103 decreases. Desirably, the stock amount is larger than the harvest amount per one ridge road 705.

[0075] In this case, a position at which storage basket 307 stacked to iS correspond to ridge road 705 one by ane is to be placed, can be determined in advance. Further, instead of placing storage basket 307 directly on the ground, a dedicated storage basket placement table that is fixedly installed may be provided.

The storage basket placement table that is fixedly installed can reliably and easily deliver storage basket 307 to and from each robot. In either case, when it is determined that empty storage basket 307 exists in that location in a steady state, # is easy to manage and more efficient operation can be performed. That is, as soon as the harvest target area, that is, ridge 704 is determined, the harvest robot 102 picks up empty storage basket 307 located at a position corresponding to ridge 704, enters ridge road 705, performs the harvesting, and finishes the harvesting, and after placing harvested storage basket 307 at the same position that is the original position and sending a harvest completion signal to server 101, a new harvest instruction is received from server 101 without delay and heading for the next harvest target area and the place where empty storage basket 307 is paired therewith, Collection robot 103 places empty storage basket 307 at a nearby fixed position at which harvested storage basket 307 has been collected and becomes available, and receives a collection instruction from server 101 on its return path and picks up harvested storage basket 307. After issuing a collection signal to server 101, it goes to collection station 104. Thereafter, this is repeated.

[0076] As a mechanism in which harvest robot 102 is placed with a plurality of storage baskets 307 and sequentially each storage baskets 307 is stored, a structure is exemplified such that as illustrated in FIG. 3, first lifter mechanism 305 and third lifter mechanism 309 are vertically disposed in the same axis, and third lifter mechanism 309 sequentially stores the harvest target object using the gap generated by lifting remaining empty storage basket 307, but the present exemplary

2 17 _ embodiment is not limited to this. In short, any structure may be used as long as a plurality of storage baskets 307 can be placed, target object 308 can be stored, and storage baskets 307 stacked from the ground or a fixed table can be loaded or unloaded.

[9077] Further, in the extreme case, storage baskets 307 may not be stacked and may be only one, in which case third lifter mechanism 308 is unnecessary. However, since the harvest span is shortened, the collection efficiency is deteriorated, and the number of necessary collection robots 103 is increased. Of course, when the length of the ridge is short and target object 306 to be harvested is a small fruit such as sweet cherry, one storage basket can be used for one ridge road, so this is not the case.

[0078] Next, the option group in the present exemplary embodiment will be sequentially described with reference to FIGS. 9 to 18. These option groups improve, reinforce, or supplement the functions of the harvest robot system more effectively based on the basic operation flow of harvesting, collecting, and supplying a storage basket in FIG. 8 described above. The option groups may be appropriately combined with the exemplary embodiment as long as there is no contradiction.

[0078] FIG. 9 is a flow chart of the tool change operation according to the exemplary embodiment of the present disclosure.

[00801] When a plurality of types of crops are being cultivated in cultivation yard 701, harvest robot 102 is generally equipped with a dedicated harvesting hand according to the characteristics of each of the crops on the tip of the manipulator of harvester 304 to perform harvest work, Various harvesting hands are part of a group of working tools, and the replacement of the working tools is generally called a tool change. When the present option is applied, the manipulator of harvester 304 of harvest robot 102 and the tip of the harvesting hand have a structure capable of automatically replacing the harvesting hand by a mechanism called an auto tool! changer of an industrial robot, for example. A detailed description of the auto fool changer is omitted.

[0081] In the tool change operation flow, as illustrated in FIG. 9, server 101 receives an input of a given tool change condition in advance (S101). The given tool change condition is input to input/output part 202 by operator 106 operating input/output unit 107. The given tool change condition input by operator 108 is information that should be given in advance for determiner 203 to determine the tool replacement, and is, for example, planting information and tool code information.

The planting information is linked with the farm map information to specify the position of the harvest area for each variety of target object 306 being cultivated in cultivation yard 701, that is, the ridge number, and to identify the type of harvesting hand to be applied to each variety is defined by a tool code. The tool code

2 18 information is a code number assigned to each type of harvesting hand, and server 101 configures the tool change information indicating which harvesting hand harvest should be placed on harvest robot 102 using the tool code.

[0082] In the state where the given tool change condition is input, determiner 203 of server 101 to which the image data with a map address is input from step R1 in FIG. 8 determines the harvest target area. At the same time, the tool code of the harvesting hand corresponding to the type of target object 306 that is determined as the harvest target area is determined (8102). On the other hand, server 101 obtains the robot state information including the tool state from harvest robot 102 at the same time as the image data with a map address. When the harvasting hand which is mounted on current harvest robot 102 and the tool code of the determined harvesting hand match {Y in $103), harvest instruction information is output, and harvest robot 102 performs the harvesting and storing (R2) according to the basic operation flow of the harvesting, collecting, and supplying a storage basket in FIG. 8. When the tool codes do not match (N in $103), server 101 outputs a request signal to home station 105, and inquires home station 105 about the available status of the standby position and the stock status of tools including the harvesting hand. Home station 105 returns empty position information, that is, which dock is currently empty, and tool stock information, that is, how many tools having what code, are left (H101). Determiner 203 of server 101, which has checked the emptiness of the standby position of home station 105 and the stock of necessary tools, determines the replacement of the tools (5104). Server 101 outputs the home station destined instruction information that includes the tool change information and indicates which dock to head to, to harvest robot 102. Harvest robot 102 that received the instruction information, moves to home station 105 and enters the designated dock (R101). After arriving at home station 105, harvest robot 102 transmits an arrival signal to server 101. Server 101, which received the arrival signal, transmits the tool change information to home station 105 (8105). Home station 105, which received the tool change information, is linked to harvest robot 102 and replaces the designated tool (H102). When home station 105 notifies server 101 of the completion of the replacement as a tool change completion signal, server 101 fransmits the harvest instruction information to harvest robot 102 (S108). After that, the procedure returns to the basic operation flow of harvesting, collecting, and supplying a storage basket illustrated in FIG. 8.

[0083] The tool change H102 in home station 105 may be based on the tool change information from harvest robot 102 instead of the tool change information from server 101. However, home station 105 needs a transceiver capable of directly transmitting or receiving information to or from harvest robot 102 via a wireless LAN or the like, or a structure capable of exchanging information at a contact point.

Similarly, the tool change completion signal may be transmitted from harvest robot 102 to server 101,

[0084] FIG, 10 is a flow chart of the charging operation according to the exemplary embodiment of the present disclosure. Harvest robot 102 and collection robot 103 have self-propelled carriage parts 308 and 404, and a rechargeable battery such as a lead storage battery or a lithium ion secondary battery is mounted as a power source. These are used not only as power sources for self-propelling, but also as other power sources such as harvesting manipulators, and need to be charged regularly. An automatic charging device that automatically performs charging is generally used in an automatic guided vehicle (AGV), and a description thereof will be omitted.

[0085] In FIG. 10, when the battery level of the rechargeable battery reaches a predetermined warning level, harvest robot 102 or collection robot 103 transmits the robot state information including the charging state to that effect to server 101 (R151). For the warning level, the minimum required remaining capacity of the rechargeable battery is more than the capacity that can be reached to home station 105 by interrupting work anywhere in the farm. Server 101, which has received the warning that the battery is dead in the robot state information, transmits a request signal to home station 105 to check the availability of the dock equipped with the automatic charging device (8151). When there is an empty dock, home station 105 returns the empty dock number to server 101 as empty position information (H151}. Determiner 203 of server 101, which has checked the emptiness of the dock, determines to charge (8152). Next, server 101 transmits to harvest robot 102 or collection robot 103, home station destined instruction information including charging instruction information indicating which dock to go to. Harvest robot 102 or collection robot 103 that has received the home station destined instruction information moves to the designated dock at home station 105 (R152).

Harvest robot 102 or collection robot 103 arrives at home station 105 and transmits an arrival signal to server 101, thereafter server 101 transmits charging instruction 39 information to home station 105 (8153). Home station 105, which has received the charging instruction information starts charging (H152). After charging is completed, home station 105 transmits a charging completion signal to server 101. Server 101 which has received the charging completion signal transmits the standby instruction information for instructing the standby when harvest robot 102 or collection robot 103 has no work (8154). Harvest robot 102 or collection robot 103 which has received the standby instruction information stands by at the designated position (R153).

Determiner 203 of server 101 assigns a work when harvest robot 102 or collection robot 103 has the work, and gives the harvest instruction information and other instruction information to harvest robot 102 or collection robot 103 (5155).

Cc 20

[0086] The charging instruction information from above-described server 101 to home station 105 may be transmitted from harvest robot 102 or collection robot 103. However, home station 105 is required to have a transceiver device, or a structure capable of exchanging information a contact point. Similarly, the charging completion signal may be transmitted from harvest robot 102 or collection robot 103 to server 101.

[0087] This charging operation may be performed systematically at the same time as the tool change operation in FIG. 9 described above.

[0088] Further, although the charging operation is started when the battery level reaches the warning level, server 101 may periodically monitor the states of the rechargeable batteries of harvest robot 102 and collection robot 103, and may systematically allow harvest robot 102 and collection robot 103 to enter the dock ahead of time when work can be easily separated to avoid starting the charging operation during work.

[0088] FIG. 11 is a flow chart for determining a harvest target area according to the exemplary embodiment of the present disclosure.

[0080] There are various procedures for the harvest target area determination (53) in the basic operation flow chart for harvesting, collecting, and supplying a storage basket in FIG. 8. An example of a procedure for objectively and efficiently determining the harvest target area is illustrated in FIG. 11.

[0091] As illustrated in FIG. 11, when the image data with a map address is input to server 101 from harvest robot 102 that has imaged the harvest area and specified the seif-position (R1), determiner 203 first removes unnecessary noises such as backgrounds or leaves from the image data with a map address, and extracts only fruits (5201). Since various image processing techniques have been proposed in the related art for this method, a detailed description thereof will be omitted. Next, determiner 203 ranks the degree of maturity of each extracted fruit one by one based on the degree of maturity rank reference (N201) input by operator 106 through input/output unit 107 in advance (8202). The degree of maturity rank reference is that, for example, most fruits continuously change from a first color that is immature to a second color that is fully ripe, so when it is divided into 10 stages during that period, it will be ranked 10 ranks. That is, the degree of maturity rank reference is, for example, information in which the color of fruit and the degree of maturity of fruit are associated with each other for each type of fruit. For example, determiner 203 specifies the color of one or a plurality of fruits extracted from the image data with a map address. Determiner 203 specifies each degree of maturity (rank) corresponding to the color of the specified fruit for each extracted fruit with reference to the degree of maturity rank reference. The first color is, for example, green, and the second color is, for example, red. The combinations of the first color u 21 and the second color differ depending on the fruit. Further, depending on the fruit, during maturity, one fruit may be divided into the first color and the second color depending on the location, or the color between the first color and the second color may continuously change. Further, the size of the fruit generally increases as the fruit matures. The degree of maturity rank reference is obtained by comprehensively performing a pattern classification of the features according to the characteristics of target object 306 by degree of maturity. Determiner 203 compares these ranking reference patterns with target object 306, and ranks the rank of the closest reference pattern as the rank of target object 306.

[0082] Next, since the rank that can be harvested or higher is input as an advance threshold value in the rank (N202), determiner 203 extracts the fruit having the rank equal to or higher than the threshold value from the ranked fruits, and estimates the harvest amount for each unit harvest area (8203). Basically, based on the estimated harvest amount, the ridge road with the largest harvest amount may be set as the target harvest area, and harvest robot 102 may be distributed in order, but some correction may be performed in here (5204). For example, instead of uniformly counting the number of fruits having a rank equal to or higher than the threshold value, weighting is performed for the fruits having a higher degree of maturity rank (N203). This is because fruits having a high degree of maturity rank need to be harvested promptly before the ripeness progresses and the product quality deteriorates, The weighting coefficient for each rank is input to server 101 from input/output unit 107 in advance.

[0083] On the other hand, a certain variety may be prioritized according to the harvest schedule table that is determined in advance and is input to server 101 from input/output unit 107 (N204). This is based on the determination that it is better to prioritize the varieties that can be sold, as the harvest schedule table reflects the needs from the market. In that case, the weighting coefficient of the variety that can be sold is set to a large value. Further, in a case where the shipping plan for each variety is predetermined based on the contract with the large-scale customer of crops, when it is cut at a predetermined threshold value as usual, the quantity of the variety may be insufficient. In that case, some unripe fruit may be mixed, but it is necessary to lower the rank of the threshold value to satisfy the predetermined quantity. On the contrary, when it is cut at the predetermined threshold value as usual, the quantity of the variety may exceed the plan, In that case, it is necessary to raise the rank of the threshold value to reduce the planned harvest quantity.

Abandonment of crops should be avoided for a farm. Determiner 203 corrects the evaluation value by reflecting these circumstances. Based on the result, determiner 203 finally determines the harvest target area (5205). Thereafter, the process

~ 22 proceeds as illustrated in the basic operation flow chart of harvesting, collecting, and supplying a storage basket in FIG. 8.

[0094] Although the above-described method for determining the degree of maturity of fruits is performed based on the image data imaged by harvest robot 102, it may be determined based on other information. For example, instead of the harvesting hand of harvester 304 of harvest robot 102, a probe of a saccharimeter may be mounted, the sugar content of the fruit may be measured, and the harvest target area may be determined based on the sugar content.

[0095] Further, the operation flow illustrated in FIG. 11 has been described with reference to the example of the fruit that is easily colored, but the same method can be applied to other crops. However, depending on the type of crop, not only the color but also the shape and size may be used for the determination. The point is that depending on the characteristics of the crop, a determination reference is selected that is easy to determine the degree of maturity.

[0086] FIG, 12 is a flow chart of an imaging rotation operation according to the exemplary embodiment of the present disclosure.

[0097] Inthe imaging processing of the image data with a map address according to the flow chart in FIG, 8, harvest robot 102 first performs an imaging based on imaging instruction information from server 101 (R1). This routine is mainly used for crops that are harvested only once each year in the same harvest area for a fimited time. On the other hand, the harvest work continues all year long or for several months at some high-cultivation facilities for greenhouses such as cherry tomatoes and strawberries. In these crops, innumerable fruits grow in sequence from one seedling and stem during the harvesting period, and thus the same ridges are used for harvesting many times during the harvesting period. That is, fruits that have not been harvested due to immaturity during a certain harvest work reach a ripening period, for example, one week later. By utilizing this characteristic, it is possible to omit the scanning traveling of harvest robot 102 for the purpose of only imaging. The operation will be described below with reference to FIG. 12, 39 [0088] In FIG, 12, harvest robot 102 that has received the harvest instruction information from server 101 performs harvesting and storing (R2}. More specifically, in the harvesting and storing by harvest robot 102, first, target object 306 is recognized in imaging part 301, harvest availability determination is performed and a position of harvestable target object 306 is specified (R251), thereafter the harvesting is performed in harvester 304 (R252) and target object 306 is stored in storage basket 307 (R253). When all the placed storage baskets 307 are full of target objects 306 or when the harvesting in a ridge that is instructed by server 101 is completed, harvest robot 102 transmits a harvest completion signal to server 101 (R254). In the above flow, in the present option, the image of target object 306

27 23 imaged for harvesting target object 308 is transmitted to server 101 as the image data with a map address each time (R251). All of the unit harvest areas is imaged at _ first in order for harvest robot 102 to select target object 306 to be harvested in the unit harvest area at the time of harvesting, information on all fruits including immaturity is obtained.

[09998] Since the image dala obtained by server 101 includes the crop to be harvested by harvest robot 102, determiner 203 first deletes the harvested crop from the image data (5251). Harvest success information on which the crops are harvested or failed is obtained from harvest robot 102. For example, harvest robot 102 determines that the harvesting has succeeded when the crop to be harvested is not detected from the image data, and determines that the harvesting has failed when the crop to be harvested is detected.

[0100] Server 101 accumulates the processed image data in the image data transmitted from harvest robot 102 (8252). When a certain amount is accumulated, determiner 203 creates a maturity distribution map of crops (8253). A certain amount is, for example, when the harvest work of the day is completed or when the harvesting of one ridge road is completed. The maturity distribution map is data in which the degree of maturity rank for all fruits is performed, and is data indicating in which region fruits of which level of degree of maturity are distributed. From this maturity distribution map, determiner 203 estimates the total amount of crops that can be harvested after a certain period of time has passed in a certain ridge, for example (5254). At this time, the threshold value of the harvest availability is input separately. Based on this estimated amount, determiner 203 determines the harvest target area after a certain period of time has passed (8258). From these, determiner 203 performs a determination of a harvest schedule after the certain period of time has passed (5258), and server 101 transmits the harvest instruction information to harvest robot 102 at any time based on the harvest schedule. By repeating this as rotation in a cycle of a certain time, the image data with a map address is sequentially obtained at the same time as the harvesting, so that harvest robot 102 does not need {o bother to scan inside of the farm only for imaging. Therefore, the work efficiency of harvest robot 102 is improved, and the number of harvest robots 102 can be reduced accordingly.

[0101] In FIG. 12, the maturity distribution map is created after the image data is accumulated, but the distribution map may be created sequentially after the image data is obtained.

[0102] The rotation span is appropriately set to a span with high operational efficiency and high prediction accuracy based on the characteristics of crops and farms, and past data.

CC 24

[0103] FIG. 13 is a flow chart of a degree of maturity determination with learning function applied according to the exemplary embodiment of the present disclosure.

[0104] In the basic operation flow chart for harvesting, collecting, and supplying a storage basket in FIG. 8, the determination of the harvest target area by determiner 203 is performed based on the given harvest condition including the harvest determination reference input in advance. More specifically, as illustrated in the flow chart of a harvest target area determination according to the exemplary embodiment of the present disclosure in FIG. 11, the degree of maturity rank reference and the harvestable threshold value are given as the given harvest conditions. In the present option, instead of people set the degree of maturity rank reference and threshold value based on human experience and intuition, determiner 203 sets each of the degree of maturity rank reference and threshold value based on the accumulated image data with a map address (5300). The operation will be described below with reference to FIG. 13.

[01058] In FIG. 13, server 101 accumulates the image data with a map address imaged by harvest robot 102 (8301). An image of a single crop that is randomly extracted from the accumulated image data regardless of the degree of maturity is displayed on input/output unit 107 (N301). Operator 108, who checked input/output unit 107, determines the harvest availability of the crop (P301), attaches a label with two options, and inputs the label through input/output unit 107 (N302).

Server 101 associates the image data of the crop with a label corresponding to the image data (8302), and accumulates the associated data (S303). When sufficient accumulation is obtained, determiner 203 of server 101 uses the predetermined determination factors, for example, hue, brightness, saturation, degree of color mixture {variation}, size, shape, or the like of the crop, and generates a degree of maturity evaluation function based on the evaluation indexes thereof. Next, the evaluation function that most clearly divides the rank of harvest availability is extracted, and the boundary is determined as a threshold value (5304). These are newly set as given harvest conditions and applied to the image data with a map address from harvest robot 102, and determiner 203 determines the harvest target area (83) and transmits the harvest instruction information to harvest robot 102.

[0106] As the evaluation function, typically, a classifier utilizing a machine learning such as a neural network, a Bayes classifier, or a support vector machine (SVM) is used. The classifier uses the teacher data in which the image of the target object and the degree of maturity of the target object are associated with each other, the learning process is applied, and the amount of target objects that can be harvested for each unit harvest area is estimated using the learned classifier.

EB 25

[0107] In the process of generating the evaluation function and the threshold value, operator 106 may intervene at any time to make a correction.

[0108] FIG, 14 is a flow chart for a manual programming according to the exemplary embodiment of the present disclosure. In the basic operation flow chart of the harvesting, collecting, and supplying a storage basket according to the exemplary embodiment of the present disclosure in FIG, 8, although the harvest target area to be harvested by harvest robot 102 is determined by determiner 203 of server 101 based on the image data with a map address of target object 306 and the given harvest condition as described above, it is the present option that, in some cases, operator 106 is allowed to make a determination. In an actual farm, the inconvenience may occur when the harvest target area is automatically determined.

When the inconvenience is not caused by a single occurrence, the factor is added to the correction term in S204.

[0108] The operation flow will be described below with reference to FIG. 14.

[0110] As illustrated in FIG. 14, in the present option, a determination factor of "is the determination of the harvest target area automatic?” (8351) is newly added.

YES or NO is a parameter, which operator 106 inputs in advance. In the case of

YES, the harvest work proceeds according to the flow in FIG. 8 as in the related art,

When the parameter is NO, the image of the image data with a map address and the area information are displayed on input/output unit 107 (N351). After checking, operator 106 determines in which harvest area target object 306 is to be harvested (P351), and inputs the result to input/output unit 107 (N352). Server 101 which has received the determination, determines the input area as the harvest target area {83}, and transmits the harvest instruction information to harvest robot 102.

[0111] FIG. 15 is a flow chart of a harvest prediction management process according to the exemplary embodiment of the present disclosure,

[0112] In the harvest robot system according to the exemplary embodiment of the present disclosure, a harvest plan, a collection plan, and the storage basket supply plan are optimally programmed by server 101 so that harvest robot 102 and collection robot 103 do not wait, A game theory, a queuing theory, or the like may be applied to the process. Further, a harvest schedule may be programmed by making some predictions in advance based on the vast amount of past data. One example thereof will be described with reference to FIG. 15.

[0413] In FIG, 15, all image data with a map address imaged by harvest robot 102 are accumulated in mass storage device 204 (S401). The imaging date and time is also recorded in the image data as additional information. Determiner 203 systematically analyzes the image data in space and time based on the accumulated images and the information on the time and place associated with the images (8402). For example, when focusing on a specific crop at a certain place,

E 26 changes over time can be extracted, and a maturity curve on the time axis of the crop can be obtained. Further, it is understood that when the crop is harvested. That is, from this data, the future degree of maturity of a certain crop can be predicted. Next, determiner 203 predicts a future degree of maturity progress for each unit harvest area and creates a time-series degree of maturity rank predicted distribution map (S403). Determiner 203 estimates the daily harvest amount in the near future, for example, the next one week based on the predicted distribution map (8404). Next, determiner 203 determines an optimized harvest schedule in the near future based on the above estimation (S405), and transmits harvest instruction information to 19 harvest robot 102. The optimization means, for example, adding a correction to the harvest schedule so that the daily harvest amount for the next one week will be leveled. On the farm side, it will be easier to make a shipping plan because the harvest amount in the near future can be predicted. This is because, for the farm management, a shortage of shipping volume leads to a loss of sales opportunities, and an excess of shipping volume leads to a loss of abandonment.

[0114] The harvest schedule near future does not have to be a weekly schedule, and an optimal planning span is set as appropriate according to the characteristics of the crop, market needs, or the like, such as a monthly schedule.

[9115] Furthermore, when harvest robot 102 is equipped with various sensors in addition to imaging part 301 and various information other than the image data is collected at the same time, the maturity prediction accuracy is further enhanced. The various sensors are a thermometer, a hygrometer, an illuminance meter, a gas concentration meter, and a soil sensor. Harvest robot 102 is equipped with any one of the above sensors or a combination of a plurality of sensors according to the purpose. All of the items detected by these sensors have a great influence on the growth of crops. For example, generally, the higher the temperature and the higher the carbon dioxide concentration, the faster the growth, or the like.

Harvest robot 102 attaches the information obtained by these sensors to the image data with a map address and transmits the image data to server 101. Determiner 203 creates the degree of maturity rank predicted distribution map by adding these pieces of information to the above-mentioned image data spatiotemporal analysis and future degree of maturity progress prediction. The following is based on the flow described with reference to FIG. 15.

[0116] FIG, 16 is a flow chart of a real-time feedback process according to the exemplary embodiment of the present disclosure.

[0117] With the option illustrated in FIG. 15, a harvest prediction for the near future, for example, the next one week can be established, and thus the weekly harvest schedule can be established. As a result, the efficiency of work on the entire farm including harvesting is improved. However, unlike industrial products, crops do

E 27 not grow uniformly, and vary greatly depending on the weather and other factors. A prediction may be made at first, but as the prediction is wrong, a feedback mechanism thal constantly changes the program on the fly with the latest information is necessary. One example thereof will be described with reference to FIG. 16.

[0118] In FIG. 16, steps S401 to 8405 are the same flow as those in FIG. 15. In the present option, harvest robot 102 images the harvest target area before actually harvesting based on the harvest schedule. Harvest robot 102 transmits the latest image data with a map address to server 101. Determiner 203 of server 101 compares the obtained image data with a map address with the degree of maturity prediction map at the time of creating the harvest schedule plan in the near future {5451} and directly executes the initial harvest plan (5453) when the deviation with the prediction is less than or equal to the allowable value. When the deviation exceeds the allowable value, the modification of the harvest target area is performed (8452).

[0118] The modification may be directly performed by operator 106 via input/output unit 107.

[0120] FIG, 17 is a flow chart of a cultivation management process according to the exemplary embodiment of the present disclosure.

[0121] Up to now, the target of the present harvest robot system implementation work has been limited to the harvest work, but with the present option, the target of the implementation work is expanded to agricultural work other than harvesting. The agricultural work other than harvesting includes, for example, pruning, chemical spraying, fruit picking, and leaf picking. Many harvesting hands are equipped with scissors for harvesting in the related art, and the present harvest robot 102 can be used for pruning, fruit picking, leaf picking, or the like as they are or with a slight arrangement. When the picked leaves and branches, the thinned immature fruits, or the like can be stored in storage basket 307 as they are, the work of cleaning up the residue can be reduced. Chemical application is possible by replacing the harvesting hand with a spray gun and by equipping a chemical tank instead of a storage basket. In this way, the present harvest robot system can be applied to agricultural work other than harvesting by simply replacing tools, and has a wide range of applications. Tools, in this case include not only various harvesting hands, but also pruning scissors, spray guns, and the like.

[0122] With reference to FIG. 17, a cultivation management flow including harvesting focusing on the tool replacement will be described.

[0123] In FIG, 17, the same components as in FIG. 9 are denoted by the same reference numerals, and the description thereof will be omitted. Information necessary for the tool replacement is given in advance as a given condition (S501).

The agricultural work information other than the harvesting is added from the given iE 28 condition in 5101 of FIG. 8. The tool code information is also extended to tools used for agricultural work other than the harvesting hand. The agricultural work information other than the harvesting includes, for example, state pattern data of trees to be pruned, fruit picked, and leaf picked, and a determination reference. The contents are in accordance with the given harvest condition. The chemical application is not determined based on the state of the imaged tree, and since itis performed a plurality of times at a fixed time every year, the time is input as agricultural work information. The chemical application period is set to have a wide range, and determiner 203 of server 101 determines execution of the chemical application when harvest robot 102 has an extra capacity to avoid a busy period of other agricultural work such as harvesting.

[0124] Determiner 203 of server 101 that has received the image data with a map address from harvest robot 102 determines the agricultural work to be performed and the target area based on the given condition (8502). At the same time, the robot state information including the tool stats is received from harvest robot 102, and it is determined whether the agricultural work determined with the tool can be performed (5503). When it is OK, server 101 transmits the agricultural work instruction information including the harvest work to harvest robot 102. When it is

NG, the request signal is transmitted to home station 105. The subsequent steps follow the tool change operation flow in FIG. 9. However, the target range of the tool replacement determination is extended to agricultural work tools other than the harvesting hand (8504).

[0125] FIG. 18 is a flow chart of another farm cooperation system operation according to the exemplary embodiment of the present disclosure.

[9126] Up to now, the target of implementation of the present harvest robot system has been described by limiting the scope to one farm managed by server 101, but with the present option, the target of information management is extended to another farm managed by other servers, and a system cooperating with servers 108 of the other farms is constructed via Internet 109 (see FIG. 1). Except for some high- cultivation facilities for greenhouses such as cherry tomatoes or the like, usually, the harvesting period for one crop is limited to a certain period of the year, but the harvest time differs for different types of crops. For example, the harvest time of tomatoes and cucumbers is generally from October to July, while the harvest time of eggplants and bell peppers is from June to October, thereby the peak of harvest deviates. Similarly, the harvest time of strawberries is from December to May and the harvest time of blueberries is from June to September. Further, the harvest time of apples is from August to November, and the harvest time of sweet cherry is from

May to July.

E 29

[0127] Since harvest robot 102 can cope with the harvesting of many crops by replacing tools, that is, harvesting hands, harvest robot 102 can be time-shared between the above crops. Time-sharing increases the annual operation rate of the expensive harvest robot 102, and a more cost-effective harvest robot system can be assembled.

[0128] An operation example of the cooperation system with another farm will be described with reference to FIG. 18. First, an annual harvest schedule table is created (S551). The annual harvest schedule table is input to server 101 through input/output unit 107 by operator 108 in consideration of the variable factors of this year based on the past results. Next, in order to achieve the plan, determiner 203 calculates the number of harvest robots 102 and collection robots 103, which are particularly necessary during the busy season of harvesting and determines whether it is possible to handle with the own resources (8552). When it is sufficient, determiner 203 determines a robot distribution plan with the robot on hand (8553) and sets it as a given harvest condition. When it is not sufficient, a request signal is transmitted to server 108 of another farm to obtain information on the other farm including the robot operation plan of the other farm (T7551). Next, determiner 203 incorporates the surplus resources of the other farm, determines a robot distribution plan ($554), and displays the plan on input/output unit 107 (N551). After checking, operator 106 obtains a robot rental approval of another farm and inputs the confirmation (N552). After receiving the input, determiner 203 finalizes the robot distribution plan (S558), transmits the information on the own farm including the operation plan of the robot to server 108 of another farm, and sets the information as a given harvest condition (8556). After receiving the input of the information (T8552), server 108 of another farm utilizes the information in the other farm.

[0128] Inthe above description, server 101 determines the robot distribution plan, but operator 106 may browse the status data of another farm and determines the plan. For the present system, there is no problem with either server 101 or operator 106. It is important to have a mechanism for exchanging information in a network between servers and sharing information, that is, a cooperation system.

[0130] Although the communication network between the servers is the internet, a dedicated LAN line may be used.

[0131] Further, although the flow assuming an annual plan is explained in the above, in the case where the distance between the farms is relatively close to each other, detailed time sharing on a weekly or daily basis is possible, thereby the effect of adapting flexibly by networking is high. Further, when the cooperation between the farms is expanded and developed on the premise that harvest robot 102 is also utilized for other agricultural work other than harvesting, the operation effect of the farm will be even higher.

2 30

Effect

[0132] As described above, the harvest robot system according to the present exemplary embodiment includes harvest robot 102 that harvests a target object in a farm, and server 101 configured to communicate with harvest robot 102, in which server 101 includes input/output part 202 that receives an input of a given harvest condition including farm map information of the farm and harvest determination reference information for determining a harvest time of the target object, determiner 203 that determines a harvest target area based on an image of the target object imaged for each unit harvest area in the farm by harvest robot 102 and the given harvest condition, and first transceiver 201 that transmits harvest instruction information including position information of the harvest target area to harvest robot 102.

[0133] As described above, in the harvest robot system according to the present exemplary embodiment, a plurality of harvest robots 102 are centrally managed hy server 101, and harvest robot 102 which is capable of traveling, images the harvest area in the farm all over by using imaging device 301 before harvesting, thereby the precise maturity status information of crops in the entire farm is obtained, which cannot be obtained with a monitoring system using a fixed-point camera, and server 101 can always determine an optimal harvest schedule based on the information. Based on the harvest schedule from server 101, it is possible to realize a flow in which harvest robot 102 that has received the harvest instruction performs harvesting. Thereby, the harvest work efficiency of the harvest robot can be greatly improved.

[0134] Further, the various options described above can be added to the present system, and as a result, it is possible to further improve the operation efficiency of the harvest robot, thereby the cost effectiveness is further improved.

[0135] As described above, according to the harvest robot system of the present disclosure, the server can establish an optimal harvest schedule based on necessary and sufficient information on the cultivation status of the crops, and the harvest robot that has received the instruction can perform the harvest work most efficiently.

[0136] The harvest robot system of the present disclosure has a cultivation management harvest system that collects cultivation status information in a wide range and precisely, establishes an optimal work plan based on the information, and executes the work plan, and a circulation-type distribution system that efficiently collects wastes such as harvested objects and weeds, and can also be applied not only to the automation of harvesting work at facility-type fruit farms, but also to the automation of the harvesting work of almost any crop including open-field cultivation

E 31 and standing trees, as well as automation of a variety of other farming applications besides harvesting.

Claims

a 32 CONCLUSIONS 1, A harvesting robot system comprising: a harvesting robot that harvests a target object on a farm; and a server configured to communicate with the harvesting robot, the server comprising an input/output portion that receives a given harvest condition, the given harvest condition including farm map information of the farm and harvest determination reference information for determining a harvest time of the target object , gene determination entity that determines a harvest target area based on an image of the target object and the given harvest condition, the image of the target object for each unit of harvest area on the farm being mapped by the harvesting robot, and a first transceiver that transmits harvest instruction information including position sends information from the target area to the cogst robot,

The harvesting robot system according to claim 1, wherein the harvesting robot comprises an imaging part that images the target object to generate the image, a first self-position detector that detects a current position of the harvesting robot, a second transceiver that transmits the image to the server in conjunction with information relating to an imaging position, and a harvesting unit that harvests the target object based on the harvest instruction information received from the server.

The harvesting robot system according to claim 1 or 2, further comprising: a home station where the harvesting robot can stand by on the farm, the home station comprising a tool changer assembly that replaces an arm of the harvesting unit of the cogst robot from a first tool to a second tool on based on tool change information from the server, a tool changer that charges a rechargeable battery mounted on the harvesting robot, and a third transceiver that transmits empty position information and tool stock information to the server, the empty position information including dock availability information for linking the

Co 33 indicates pogstrobot, where the germ stock information includes a type and the number of germs the home station currently holds. 4. The harvesting robot system according to claim 3, wherein the determining unit determines whether to perform charging or a tool change for the harvesting robot on the basis of the empty position information, tool stock information, a given germ change condition, and robot state information, the empty position information and tool stock information is received from the home station, where the given tool change condition is used to determine a tool attached to an arm used in the harvesting robot, the robot condition information includes a battery level of the vogst robot and a tool condition of the arm and transmits it from the harvesting robot are received, and wherein the first transceiver transmits instruction information destined for the home station including load instruction information and the tool change information to the harvesting robot,

The harvesting robot system according to claim 1, wherein the image is one or more images captured such that the dosl object present in the entire unit crop area appears.

The harvesting robot system of claim 1, wherein the determination unit estimates an amount of the dosl object that can be harvested for each unit of harvest area. based on a color of the target object that appears in the image, and determines the harvesting area.

The cogsi robot system according to claim 2, wherein the harvesting robot performs imaging of the harvest area unit in accordance with the harvest target area in the imaging part when the target object in the harvest target area is harvested as instructed by the server.

The harvesting robo system according to claim 1, wherein the determining unit extracts the target object existing in the entire unit pogsi area from the image, creates a degree of maturity ranking distribution map of the target object in the unit of harvest area, and a harvestable amount of the unit harvest area estimates based on the degree-of-ripeness ranking distribution map.

‚© 34

9. The harvesting robot system according to claim 1, wherein the determining unit comprises a classifier in which a learning process is performed using teacher data with the image of the target object and a degree of maturity of the target object being associated with each other, estimating an amount of the target object that for each unit of harvest area can be harvested using the classifier, and determines the harvest target area,

710. The harvesting robot system of claim 1, wherein the input/output portion is configured to receive an input from the harvest target area by an operator.

The harvesting robot system of claim 1, wherein the determination unit creates a harvest schedule that defines a time to harvest the target object in the unit harvest area based on the time series images.

The harvesting robot system of claim 1, wherein the determining unit creates a harvesting schedule that defines a time to harvest the target area in the harvesting area based on sensor data from at least one of a thermometer, a hygrometer, a light meter, a gas concentration meter, and a soil sensor in the harvesting robot.

The harvesting robot system according to claim 1, further comprising: a collecting robot that collects the target object harvested by the harvesting robot and transports the target object to a collecting station on the farm.

The harvesting robot system according to claim 1, wherein the determining unit determines the harvesting target area based on a position and an operating state of the cogst robot.

The harvesting robot system according to claim 3, wherein the determination unit determines a time required for agricultural work other than harvesting in the harvest area unit based on the image, and the instruction information destined for the home station and the tool change information at the time are sent to the harvesting robot sends. 16. The harvesting robot system according to claim 1,

© 35 wherein the input/output part obtains information about another farm including a business plan from a harvesting robot of the other farm, and wherein the determination unit creates a business plan of the harvesting robot from the farm based on the information at the other farm.