JP2021051767A - System and method for supporting operation using drone - Google Patents

Abstract

To facilitate utilization of a drone for a desired operation.SOLUTION: A system for supporting an operation using a drone comprises: first operation result storage means for receiving a result of a first operation performed to a first object and storing it; operation report preparation means for preparing an operation report representing the result of the first operation; operation report provision means for providing at least one of users with the operation report; second object specification tool provision means for providing at least one of the users with a second object specification tool constituted so that at least one of the users can specify a second object according to the result of the first operation displayed in the operation report; second object storage means for receiving specification of the second object from at least one of the users to store a position of the second object; and second object information processing means for performing information processing regarding a second operation regarding the second object according to the position of the second object to provide at least one of the users or at least one of drones with a result of the information processing.SELECTED DRAWING: Figure 2

Description

The present invention relates to systems and methods for supporting operations such as investigation, inspection, maintenance, processing, destruction or construction of man-made or natural objects such as facilities, structures, lands, fields and forests, and particularly with respect to drones. Concerning systems and methods suitable for assisting the work that was being done. In the present specification, the term "drone" is used to refer to a mobile body that can be operated without a crew member (for example, an aircraft, a traveling vehicle, a surface ship, or a submersible that can be remotely controlled wirelessly or autonomously. Etc.).

Technologies for using drones for various purposes are known. For example, the management system disclosed in Patent Document 1 uses an unmanned aerial vehicle to identify the position of an object existing in a building (for example, a parked vehicle in an indoor parking lot) and acquire characteristic information.

According to this management system, a flight path for locating a vehicle in a building is calculated based on a map in the building. Then, an unmanned aerial vehicle flies along the flight path. During the flight, the position-related information acquisition unit mounted on the unmanned aerial vehicle acquires information related to the position of the vehicle. Then, the position of the vehicle in the building is specified based on the acquired information, and the flight path for specifying the characteristics of each vehicle is calculated based on the result of the position identification. After that, the unmanned aerial vehicle flies along the latter flight path. During the flight, the feature information acquisition unit mounted on the unmanned aerial vehicle acquires the feature information of each vehicle. Then, by associating the acquired feature information with the position of the specified vehicle, the feature information of each vehicle is specified.

Patent Document 2 discloses a power transmission line inspection system using an unmanned aerial vehicle. According to this system, an unmanned aircraft photographs the inspection points of the transmission line (steel tower, the site directly under the transmission line, trees near the transmission line, etc.), and a three-dimensional image of the inspection location is generated from the photographed images. Anomalies at the inspection points are found based on the three-dimensional image. For example, the degree of corrosion of plating is determined from the color, brightness, saturation, etc. of the steel tower, or new changes are grasped by comparison with past three-dimensional images.

Patent Document 3 discloses a water level management system that grasps the water level of a paddy field from an image obtained by photographing the paddy field with an unmanned air vehicle and transmits an opening / closing instruction of a floodgate.

Patent Document 4 describes an inspection system that estimates deterioration of structures such as buildings and bridges, creates an inspection plan, and determines whether repair is necessary. The inspection system automatically estimates the deterioration of the structure based on the design data and status data of the structure (usage history, recent inspection results, repair results, etc.), and based on the estimated results, the unmanned aerial vehicle The first inspection plan for carrying out a simple inspection using photography etc. is automatically created. Then, the inspection system automatically creates a second inspection plan for carrying out a detailed inspection by a person based on the result of the simple inspection carried out, and based on the result of the carried out detailed inspection, the structure. Automatically determines the necessity of repair.

Japanese Unexamined Patent Publication No. 2015-131713 Japanese Unexamined Patent Publication No. 2005-265699 Japanese Unexamined Patent Publication No. 2016-220681 JP-A-2017-071972

In order to use a drone for a certain task, it is necessary to prepare a movement plan suitable for the task (for example, a flight plan for a flying drone), or to manually steer the drone so that the task can be performed properly. is there. However, this is not so easy for many workers. For example, the availability of drones for agricultural work such as pesticide spraying would improve agricultural productivity, but it is not easy for the average farmer to do so.

It is hoped that the work will be properly planned and carried out to achieve the expected effects. However, determining what kind of work is appropriate in various real-life situations is not easy for both individual workers and the designers of systems that support the work. Furthermore, for example, in the case of spraying pesticides in the field, the same pesticides do not always have the expected effect on the same disease of the same crop. If you continue to use the same pesticide for the same disease repeatedly, the effect will decrease, so it may be better to change the pesticide. Such circumstances make it more difficult to design a work plan to obtain the expected work effect.

One of the objects of the present invention is to further facilitate the user's use of the mobile body for desired work in a system that supports work using a moving body such as a drone.

Another object of the present invention is to support the design of a work plan to obtain the expected work effect.

A drone work support system configured to be able to communicate with one or more drones or to one or more users using one or more drones according to one embodiment of the present invention is a line for the first object. The first work result storage means for receiving and storing the result of the first work, the work report creation means for creating the work report representing the stored result of the first work, and the created work report. A second target configured to provide a work report providing means to at least one of the users and to allow at least one of the users to identify a second target according to the result of the first work displayed in the work report. A second target identification tool providing means for providing the target identification tool to at least one of the users, and a second target for storing the position of the second target after receiving the identification of the second target from at least one of the users. Information processing related to the second work related to the second object according to the storage means and the position of the second object stored, and providing the result of the information processing to at least one of the users or at least one of the drones. (Ii) Provided with a target information processing means.

According to this drone work support system, in a system that supports work using a moving body such as a drone, it is possible to further facilitate the user's use of the moving body for a desired work.

The block diagram which shows the whole structure of the system which supports the work using the drone which concerns on one Embodiment of this invention. A flowchart showing the flow of an overall control example of the system. The flowchart which shows the flow of the control example of the registration of the area position in the system. Explanatory drawing which shows the example of the designation method of the area position and the data structure example of the area position. A flowchart showing the flow of a control example of creating and setting a survey flight plan in the system. Explanatory drawing showing an example of the relationship between the area and the survey flight plan. Explanatory drawing which shows the data structure example of the investigation flight plan. A flowchart showing the flow of a control example of photography using a survey drone in the same system. A flowchart showing a flow of control examples of creating and registering a regional image from a captured image in the system. Explanatory drawing which shows the data structure example of the area image. A flowchart showing the flow of a control example of registration of a work location and an actual work flight plan in the system. Explanatory drawing which shows the data structure example of a work place. Explanatory drawing which shows an example of the relationship between a work place and an actual work flight plan. Explanatory drawing which shows the data structure example of the actual work flight plan. The flowchart which shows the flow of the control example of the setting of the actual work flight plan in the system. The flowchart which shows the flow of the control example of the actual work using the actual work drone in the same system. The block diagram which shows the whole structure of the work support system which concerns on 2nd Embodiment of this invention. The flowchart which shows the flow of the whole control example of the system which concerns on 2nd Embodiment. A block diagram showing a configuration example of the symptomatology analysis unit. A block diagram showing a configuration example of the work analysis unit. The figure which shows the example of the graphic interface when the area of a work place is selected on the display screen of an operation terminal. The figure which shows the example of the graphic interface when the symptom of a work place is identified on the display screen of an operation terminal. The figure which shows the graphic interface when deciding the actual work to be performed on the work place on the display screen of the operation terminal. A block diagram showing another configuration example of the analysis unit. A block diagram showing another configuration example of the analysis unit. A plan view showing a plan design of an example of an image correction scale that can be used for photography with a drone. FIG. 6 is a diagram showing an example of dimensional conditions satisfied by each reference color panel in an example of the image correction scale shown in FIG. 26. A block diagram showing a configuration example of a system for correcting a regional image using an image correction scale. The flowchart which shows an example of the control performed by the correction system shown in FIG. 28.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

In the following explanation, an example is the use of a drone configured as an unmanned aerial vehicle for inspection and maintenance of a field (for example, identification of a location where a disease or physiological disorder has occurred and spraying pesticides or fertilizers on that location). Hereinafter, embodiments of the present invention will be described. However, this use is merely an example for illustration purposes and is not intended to limit the application of the embodiment to other uses.

FIG. 1 shows the overall configuration of a system that supports work using a drone according to an embodiment of the present invention.

The work support system 1 shown in FIG. 1 is typically available to a plurality of different users. However, in the following, the explanation will be focused on the usage scene of one user among a plurality of users.

As shown in FIG. 1, when using the work support system 1, the user can use different types of drones suitable for different tasks, for example, two types of drones 3, 5. In the present embodiment, one of them is the survey drone 3, which serves to survey the state of the geographical area, that is, the area (for example, the field owned or used by the user) desired by the user, that is, to collect information on the area. Have. In the present embodiment, as a method of the area survey, a photograph of the area is taken from the sky. However, photography is an example, and other survey methods, such as radio radar, sound wave radar, or information gathering methods using various sensors, may be adopted. Moreover, you may combine different investigation methods.

In this embodiment, another type of drone is the actual work drone 5, which is a type of work that should be performed in the area (hereinafter referred to as actual work), for example, a disease has occurred in a vast field. It has the role of selectively spraying pesticides on the spots.

The same aircraft drone may be used as the survey drone 3 and the actual work drone 5, but it is often desirable to use different aircraft drones in order to optimally fulfill their respective roles. It is also possible to use a plurality of survey drones 3 and / or a plurality of actual work drones 5 for one area. However, in the following, the description will proceed by taking as an example the case where one user uses one survey drone 3 and another actual work drone 5.

The work support system 1 also has a data server 7, which is a computer system for managing and processing various data necessary for using the drones 3 and 5.

Further, when using the work support system 1, the user uses the operation terminal 9. The operation terminal 9 has a function of performing data communication with the data server 7 through a communication network such as the Internet. The operation terminal 9 also has a function of transferring data to and from the drones 3 and 5 (for example, data can be exchanged with the drones 3 and 5 by a wired or wireless communication method or via a portable data storage. ).

A general-purpose portable information processing terminal having such a function (for example, a so-called mobile phone, a smartphone, a tablet terminal, a mobile personal computer, or a notebook personal computer), or another type of computer (for example, a desktop). A type personal computer) can be adopted as the operation terminal 9. In that case, a computer program for using the work support system 1 (for example, an application dedicated to this system or a general-purpose web browser that can access a website that is an external interface of the data server 7) is the general-purpose. By being installed in the information processing terminal of the above and executing the computer program on the terminal, the general-purpose terminal functions as an operation terminal 9 for the work support system 1. Alternatively, as the operation terminal 9, dedicated hardware may be prepared for the work support system 1.

When using the work support system 1, each user may always use only the same one operation terminal 9, or appropriately select one of a plurality of operation terminals 9 depending on the time and the case. You can use it. Further, a plurality of different users can use their respective operation terminals 9 to refer to information about the same area (for example, the same field) and input desired information. In the following, the description will proceed by taking the case where one user uses one operation terminal 9 as an example.

The survey drone 3 includes a flight mechanism 11 for flying it, a GPS receiver 13 for measuring its three-dimensional position (latitude, longitude, altitude), a camera 15 for taking a picture, and their devices 11, 13 , 15 has a controller 17 for controlling. Further, the survey drone 3 is attached with a radio control device 19 (so-called radio control) for the user to remotely control it wirelessly. The radio-controlled model 19 can wirelessly communicate with the controller 17, and transmits various control commands to the controller 17 in response to various user operations on the radio-controlled model 19.

The actual work drone 5 includes a flight mechanism 21 for flying it, a GPS receiver 23 for measuring its three-dimensional position (latitude, longitude, altitude), an actual work device 25 for performing actual work, and those devices. It has a controller 27 for controlling 21, 23, 25. When the actual work is pesticide spraying in the field, the actual work machine 25 is naturally a pesticide spraying device, but an auxiliary device such as a camera for photographing the actual work situation is additionally equipped. May be good. Further, the actual work drone 5 is attached with a radio control device 29 (so-called radio control) for the user to remotely control it wirelessly. The radio-controlled model 29 can wirelessly communicate with the controller 27, and transmits various control commands to the controller 27 in response to various user operations on the radio-controlled model 29.

The operation terminal 9 has the area registration unit 31, the flight plan input unit 33, the flight plan output unit 35, the photographed image input unit 37, the photographed image output unit 39, and the photographed image in the order of the normal steps in which the user uses the operation terminal 9. It has information processing function elements such as a position input unit 41, a shooting position output unit 43, an image display unit 45, a work location registration unit 47, a work result input unit 48, and a work result output unit 49.

In response to user input, the area registration unit 31 responds to the position of the geographical area, that is, the area (for example, the field owned by the user) desired by the user (for example, the latitude and longitude of the apex of the contour of the area) (hereinafter, Register the area location) in the data server 7.

The flight plan input unit 33 downloads the flight plan desired by the user from the data server 7 in response to the user input. The flight plan output unit 35 sends the downloaded flight plan to the drone 3 or 5 desired by the user in response to the user input, and sends the flight plan to the controller 17 or 27 of the drone 3 or 5. Install it. Here, the flight plan is a kind of movement plan that defines the geographical route that the drone will fly.

In response to the user input, the captured image input unit 37 receives an image captured by the survey drone 3 (a large number of images are captured in one area) (hereinafter, referred to as a captured image). The captured image output unit 39 transmits a large number of received captured images to the data server 7 in response to the user input.

The shooting position input unit 41 responds to the user input and responds to the position and angle of the survey drone 3 (or camera 15) at the time of shooting each shot image measured by the survey drone 3 (hereinafter, the shooting position and angle are collectively referred to). And receive the shooting position). The shooting position output unit 43 transmits the received shooting position to the data server 7 in response to the user input.

In response to user input, the image display unit 45 downloads an entire image of the area already registered by the user (hereinafter referred to as an area image) from the data server 7, and displays the display screen of the operation terminal 9 (not shown). ). Here, each area image is created by the data server 7 to combine a large number of captured images of the area.

In response to user input, the work location registration unit 47 identifies one or more geographical small areas, that is, locations (hereinafter referred to as work locations) in which the actual work in the region is to be performed on the displayed region image. Then, the work place is registered in the data server 7. For example, when the actual work is spraying pesticides in a field, the work place is the place where the user observes the color and shape of crops and leaves and recognizes that a disease has occurred from the area image of a certain field. Can be. When registering this work location, the user also specifies the content of the work to be performed on the work location (for example, spraying pesticide A and pesticide B), and associates it with the work location on the data server 7. register.

The work result input unit 48 receives the result of the actual work performed by the actual work drone 5 (for example, the spraying position and the spraying amount of the pesticide) in response to the user input. The work result output unit 49 responds to the user input and transmits the received work result to the data server 7.

The data server 7 includes databases such as a regional location database 51, a three-dimensional map database 53, a flight planning database 55, a captured image database 57, a captured position database 59, a regional image database 61, a work location database 63, and a work result database 65. Have. Further, the data server 7 has information processing functional elements such as a flight plan creation unit 71, a photographed image input unit 73, a photographed position input unit 75, a photographed image combining unit 77, and an analysis unit 79.

The area location database 51 registers and manages the area position of the area designated by the user (for example, the field of the user) in response to the area registration unit 31 of the operation terminal 9.

The three-dimensional map database 53 stores and manages a three-dimensional map that defines the latitude, longitude, and altitude of each position. The three-dimensional map database 53 is used by the flight planning unit 71 to create flight plans for drones 3, 5. The data server 7 may use an external three-dimensional map on the Internet or the like without having the three-dimensional map database 53 in the server 7.

The flight plan database 55 stores and manages flight plans for drones 3 and 5. The flight plan database 55 receives a request from the flight plan input unit 33 of the operation terminal 9 and sends the requested flight plan to the flight plan input unit 33. The flight plans managed in the flight plan database 55 are roughly classified into two types, for example, a survey flight plan and an actual work flight plan. The survey flight plan is a flight plan for surveying (for example, taking a picture) each registered area with the survey drone 3. The actual work flight plan is a flight plan for performing actual work (for example, spraying pesticides) with the actual work drone 5 at one or more work locations in each registered area.

The captured image database 57 stores and manages captured images received from the captured image output unit 39 of the operation terminal 9. The shooting position database 59 stores and manages the shooting position received from the shooting position output unit 43 of the operation terminal 9.

The regional image database 61 stores and manages regional images showing the entire picture of each region created by combining captured images of each region. The area image database 61 receives a request from the image display unit 45 of the operation terminal 9 and sends the requested area image to the image display unit 45.

The work location database registers and manages the work location specified by the user in response to the work location registration unit 47 of the operation terminal 9.

The work result database 65 stores and manages the work results received from the work result output unit 49 of the operation terminal 9.

The photographed image input unit 73 and the photographed position input unit 75 receive the photographed image and the photographed position passed from the survey drone 3 to the operation terminal 9 from the operation terminal 9, and register them in the photographed image database 57 and the photographed position database 59, respectively. ..

The flight plan creation unit 71 creates a survey flight plan for each region and an actual work flight plan for one or more work locations in each region, and registers the created flight plan in the flight plan database 55. The flight plan creation unit 71 may be a fully automatic tool that can create a flight plan fully automatically, or a semi-automatic tool that allows a person to operate it to create a flight plan. ..

The survey flight plan for each region includes the regional location of each region in the regional location database 51 (for example, the latitude and longitude of the apex of the contour of the region) and the three-dimensional position of various points in each region in the three-dimensional map database 53. Created using (eg latitude, longitude and altitude). The actual work flight plan for one or more work locations in each region includes the location of the work location in the work location database 63 (eg, the latitude and longitude of the apex of the contour of the work location) and in the 3D map database 53. It is created using the three-dimensional positions (eg latitude, longitude and altitude) of various points within each work location.

The photographed image input unit 73 receives a photographed image of each area from the photographed image output unit 39 of the operation terminal 9, and registers the received photographed image in the photographed image database 57. The shooting position input unit 75 receives the shooting position of each area from the shooting position output unit 43 of the operation terminal 9, and registers the received shooting position in the shooting position database 59.

The photographed image combining unit 77 combines the photographed images of each area in the photographed image database 57 based on each area photographed position in the photographed position database 59 to create a regional image showing the whole of each area, and creates a regional image thereof. The position image is registered in the regional image database 61. The area image of each area serves as a survey report for informing the user of the result of the survey (for example, photography) of the area. By referring to the area image of each area, the user can determine which part in the area the actual work (for example, pesticide spraying) should be applied and specify the part.

The analysis unit 79 analyzes the work results stored in the work result database 65. The analysis result of the work result can be utilized for the improvement of the work method and other uses later.

FIG. 2 shows the flow of an overall control example of the work support system 1.

As shown in FIG. 2, in the operation terminal 9, the user performs an operation for registering a desired area (for example, a field owned by the user) (step S1). The registration operation is, for example, an operation of displaying a map as provided on the Internet on the screen of the operation terminal 9 and designating the area position of the area on the map. In response to this registration operation, the data server 7 registers the area by recording the area position of the area (step S2). After that, the data server 7 creates a survey flight plan for the area based on the location of the area (step S3).

After that, the user operates the operation terminal 9, downloads the survey flight plan of the area from the data server 7 to the operation terminal 9, and sends the survey flight plan from the operation terminal 9 to the survey drone 3 (step S4). .. Then, the survey flight plan is installed in the survey drone 3 (step S5).

The user takes the exploration drone 3 to the area (eg, the field owned by the user) and operates the radio control 19 to send the exploration drone 3 takeoff and other ancillary maneuvering instructions (eg, the user's own field). Step S6). The survey drone 3 automatically and autonomously flies over the area according to the survey flight plan, and repeatedly takes photographs of the surface area at various positions on the flight path, and many captured images and Record the shooting position (step S7). The control of this flight is basically performed automatically and autonomously according to the survey flight plan, and the flight instructions from the radio control device 19 are auxiliary such as the start of takeoff and slight modification of the flight position or speed. Used for.

A large number of captured images and captured positions in the area are passed from the survey drone 3 to the operating terminal 9 and then sent from the operating terminal 9 to the data server 7 (step S8). The data server 7 combines these captured images according to their respective captured positions to create a regional image of the entire region (step S9).

After that, the user operates the operation terminal 9, receives the area image of the area from the data server 7, displays it on the screen, and works on the displayed area image (for example, a disease in the field). The location where is visually recognized as occurring) is specified, and the server 7 is requested to register the work location (step S10). The data server 7 registers the work site and creates an actual work flight plan for the work site (for example, a flight plan for spraying pesticides at the work site) (step S11).

After that, the user operates the operation terminal 9, downloads the actual work flight plan from the data server 7 to the operation terminal 9, and sends the actual work flight plan from the operation terminal 9 to the actual work drone 5 (step S12). Then, the actual work flight plan is installed in the actual work drone 5 (step S13).

The user takes the working drone 5 to the area (eg, the field owned by the user) and operates the radio control 29 to send the working drone takeoff and other auxiliary maneuvering instructions. (Step S14). The actual work drone 5 basically carries out actual work (for example, spraying pesticides on diseased parts in the field) at the work place while flying over the work place according to the actual work flight plan, and records the work result. (Step S15). The control of this flight is basically performed automatically and autonomously according to the actual flight plan, and the flight instruction from the radio control device 19 assists such as the start of takeoff and slight correction of the flight position or speed. Used for

The work result is passed from the actual work drone 5 to the operation terminal 9, and is sent from the operation terminal 9 to the data server 7 (step S16). The data server 7 saves and analyzes the work result (step S17).

As can be seen from the above control flow, the user can relatively easily perform the desired work in the desired area by using the drones 3 and 5 using the operation terminal 9.

In the following, the above-mentioned control flow is divided into a plurality of parts, and each part will be described in more detail.

FIG. 3 shows a flow of a control example of registration of the area position in the work support system 1.

As shown in FIG. 3, in the operation terminal 9, the user inputs his / her own user authentication information and requests login to the data server 7 (step S21). The data server 7 authenticates the user and allows the user to log in (step S22).

Then, on the operation terminal 9, the user displays a map (step S23) and specifies a desired area (for example, his / her own field) on the map (step S24). Further, the user inputs the name of the area on the operation terminal 9 (step S25).

The map used here may be, for example, a map provided from the Internet or a map provided from the data server 7. As a method for designating the area, for example, a method in which the user specifies the apex of the outline of the desired area on the map can be adopted. Alternatively, a method of inputting a region name (for example, a place name or a lot number) whose outline is defined can be adopted. When the vertices of the region are specified, the positions of those vertices (for example, latitude and longitude) are specified by the operation terminal 9.

Then, the operation terminal 9 sends the designated area name and area position to the data server 7 (step S26). The data server 7 associates the area name, the area location, and the area ID (identification name) with the user's ID (identification name), and registers the area location database 51 (step S27).

FIG. 4 shows an example of the above-mentioned method for designating the area position and an example of the data configuration of the registered area position.

As shown in the left side portion of FIG. 4, as a method of designating a certain area 81, the user can specify the vertices A1 to A5 of the contour 83 of the area 81 on the map. In addition to this, the user may also specify the position of the departure / arrival point P1 at which the drone can arrive and depart within the area 81 or near the periphery of the area 81. When these points are specified on the map, the positions of those specified points A1 to A5 and P1 (for example, latitude and longitude (Xa1)) are based on the position coordinate system of the map (for example, the coordinate system of latitude and longitude). , Ya1) to (Xa5, Ya5), (Xp1, Yp1)) are automatically specified by the operation terminal 9.

Then, in the data server 7, as shown in the right part of FIG. 4, the area ID 87, the area name 88, the area position (for example, a set of latitude and longitude of all vertices) 89, and the departure / arrival point position (for example, the latitude of the departure / arrival point) And longitude) 91 are stored in the regional location database 51 as regional data 93 in association with the user ID 85 of the user who specified them.

FIG. 5 shows the flow of a control example of creating and setting a survey flight plan.

As shown in FIG. 5, the data server 7 creates a survey flight plan based on the regional position and the departure / arrival point position of the region registered by a user in the regional location database 51 (step S31). The research flight plan is for the shooting drone 3 to take off the research drone 3 from the departure / arrival point, take a picture of all areas of the area while flying over the area, and land at the departure / arrival point. This is an operation control instruction. When creating this survey flight plan, the data server 7 takes into account the 3D positions (eg, latitude, longitude and altitude) of the points in the area provided by the 3D map database 53, and determines the flight path. decide.

The data server 7 registers the survey flight plan in the flight plan database 55 in association with the user ID of the user and the area name of the area (step S32).

After that, the user logs in to the data server 7 from the operation terminal 9 (steps S33 to S34). In response to the user request, the data server 7 selects the survey flight plan associated with the user ID of the user from the flight plan database 55, and sends a list of the selected survey flight plans to the operation terminal (step). S35). The operation terminal 9 displays the list (step S36).

The user selects a desired survey flight plan from the displayed list and requests the data server 7, and the data server 7 sends the survey flight plan to the operation terminal, whereby the survey flight plan is sent to the operation terminal 9. It will be downloaded (steps S37-S38).

After that, the user supplies the survey flight plan from the operation terminal 9 to the survey drone 3 (step S39). The survey drone 3 installs the received survey flight plan on its own controller 17 (S40).

FIG. 6 shows an example of the relationship between the registered area and the survey flight plan.

The survey flight plan includes the flight path, flight speed, and shooting position. As shown in the left portion of FIG. 6, flight path 113 takes off from landing point P1 and flies over area 81 so that all photographs of the plurality of areas covering the entire area 81 can be taken. The three-dimensional route of the survey drone 3 until landing at the departure / arrival point P1 is defined by using, for example, the three-dimensional position (for example, latitude, longitude, and altitude) of the flight direction or velocity change point on the route.

The flight speed defines a control target value of the flight speed in various sections on the flight path 101.

The shooting position defines all the positions (indicated by dots) on the flight path 101 to be photographed, as shown in the right side portion of FIG. 6 (enlarged view of the portion surrounded by the alternate long and short dash line) in FIG. ..

At each of the shooting positions, the photographic image taken there (indicated by a solid line or a broken line rectangle in FIG. 6) and the photographic image taken at the next shooting position overlap at each edge portion. Be placed. For example, in FIG. 6, the photographic image 103A taken at a certain shooting position 103 overlaps with the photographic image 105A taken at the next shooting position 105 at the upper edge portion and the lower edge portion, respectively, and is adjacent to the left side thereof. The photographic image 107A taken at the shooting position 107 of the above overlaps with the left edge portion and the right edge of each. As a result, by superimposing and combining the photographic images taken at all the shooting positions at the overlapping portions, it is possible to synthesize the photographic images of the entire area 81 (that is, the position image of the area 81). ..

FIG. 7 shows an example of the data structure of the survey flight plan.

As shown in FIG. 7, the survey flight plan 109 includes the flight plan ID (identification name) 111 and the flight path (eg, latitude Xr, longitude Yr, and altitude Zr, respectively, of all changes on that path. ) And flight speed (target speed V from each change to the next change) and shooting position (eg, latitude Xs and longitude Ys and altitude Zs and shooting angle As of each shooting position) 115 Is included. The survey flight plan 109 is associated with the corresponding user ID 85 and area ID 87 and managed in the flight plan database 55.

FIG. 8 shows a flow of a control example of photography using the survey drone 3.

After placing the survey drone 3 at the departure / arrival point in the target area, the user operates the radio controller 19 for the survey drone 3 to first instruct the survey drone 3 to start work (step S41). In response to the instruction, the survey drone 3 takes off (step S42), and each shooting position while flying along the flight path of the flight plan by autopilot control according to the survey flight plan by the controller 17. Take a photo with (step S43). Then, the survey drone 3 records the photographed image and the photographing position (for example, latitude, longitude, and altitude) at the time of photographing (step S44).

While observing the flight of the survey drone 3, the user has a situation in which the flight state (for example, flight position and flight speed) should be corrected (for example, the flight state of the survey drone 3 is a flight plan due to the influence of wind, etc.). When it is out of the way, it is necessary to change the flight state to avoid collision with other objects or danger, etc.), operate the radio controller 19 and give auxiliary maneuvering instructions to make necessary corrections. Send to Survey Drone 3 (step S45). The investigation drone 3 corrects the flight state (position or speed, etc.) according to the correction instruction by processing the auxiliary instruction as, for example, an interrupt to the autopilot control (step S46).

After the above modification process is completed, the research drone 3 returns to the autopilot control (step S47) and continues the flight and photography according to the research flight plan (S43). After that, after flying all routes and taking photographs at all shooting positions, the survey drone 3 lands at the landing point specified in the flight plan (step S48).

The captured image 117 and the captured position 119 recorded on the survey drone 3 are sent to the operation terminal 7 and recorded in the operation terminal 7 (steps S49 to S50). This data transfer may be performed by wireless communication during flight, by wireless communication or wired communication after landing, or by taking out the data storage recording the captured image 117 and the captured position 119 from the survey drone 3 and operating the operation terminal. It may be done by attaching to 7.

FIG. 9 shows a flow of a control example of creating and registering a regional image from a captured image.

The user operates the operation terminal 9 to log in to the data server 7 (steps S51 to S52). After that, the data server 7 selects the area name associated with the user ID from the area location database 51 in response to the user request, and sends the list of the area names to the operation terminal 9 (step S53). On the operation terminal 9, the area name list is displayed, the user selects a desired area name from the list (step S54), and the photographed image 121 and the imaged position 123 acquired from the survey drone 3 are selected from the list. It is associated with the area ID and sent to the data server 7 (step S55).

The data server 7 registers the received photographed image and the photographed position in the photographed image database 57 and the photographed position database 59 in association with the corresponding user ID and area ID (step S56). After that, the data server 7 reads the photographed image and the photographed position of the area from the photographed image database 57 and the photographed position database 59, and combines the photographed images according to the photographed position to cover the entire area of the area. Combine the images (step S57). The area image is associated with the corresponding user ID and area ID, and is registered in the area image database 61 (step S58).

FIG. 10 shows an example of data composition of a regional image.

As shown in FIG. 10, the area image 125 includes, for example, an ID (identification name) 127 of the area image, a shooting date and time 129, and image data 131 covering the entire area. The area image 125 is associated with the corresponding user ID 85 and area ID 87, and is managed in the area image database 61.

FIG. 11 shows a flow of a control example of registration of a work location and an actual work flight plan.

As shown in FIG. 11, the user logs in to the data server 7 from the operation terminal 9 (steps S61 to S62). After that, the data server 7 selects a regional image associated with the user ID from the regional image database 61 in response to the user request, and sends a list of the status images to the operation terminal 9 (step S63).

On the operation terminal 9, a list of the area images is displayed, and the user selects a desired position image from the list (step S64). In response to the selection, the data server 7 reads the selected regional image from the regional image database 61 and sends it to the operation terminal 9 (step S65). The operation terminal displays the area image (step S66).

The user looks at the area image (for example, the image of the field) displayed on the operation terminal 9, finds the place where he / she wants to work (for example, the place where he / she wants to spray pesticides in the field, that is, a small area), and finds the place. Specify the work location on the area image (step S67). The work location can be specified, for example, by designating the apex of the contour of the work location. In this step S67, the user also specifies the content of the work to be performed on the work location (for example, "work type is pesticide spraying, pesticides used are pesticide A and pesticide B").

In response to this designation, the data server 7 determines the position of the work location (for example, the outline of the work location) based on the location of the work location on the regional image and the location of the region recorded in the regional location database 51. (Latitude and longitude of the apex of) are specified, and the work location is registered in the work location database 63 (step S68). In step S68, the database server 7 also registers the work content specified by the user in the work location database 63 in association with the work location.

After that, the data server 7 creates an actual work flight plan based on the position of the registered work place and the three-dimensional position of the area and various points of the work place recorded in the three-dimensional map database 53. (Step S69). The actual work flight plan is to take off the actual work drone from the departure and arrival point in the area, fly over one or more work points in the area, and perform the actual work (for example, spraying pesticides) at each work point. This is an operation control instruction for the actual work drone 5 for returning to the departure / arrival point and landing.

The data server 7 registers the actual work flight plan in the flight plan database 55 in association with the corresponding user ID, area ID, and work location ID (step S70).

FIG. 12 shows an example of data configuration at the work location.

As shown in FIG. 12, the work location 133 includes its ID (distinguishing name), the date and time it was specified by the user, and the position of the vertices of the contour of the area (eg, latitude Xa and longitude Ya of each vertex). 139 and the like are included. This work place 133 is associated with the corresponding user ID 85, area ID 87, and area image ID 127, and is managed in the work place database 63. Further, the work content 140 is associated with the work location 133 and managed in the work location database 63. By managing the work content 63, it is useful to evaluate the work content later (for example, after performing the actual work, perform a survey flight with a drone to check the state of the work site, and the survey result and the work content. It can be evaluated by analyzing and).

FIG. 13 shows an example of the relationship between the work location and the actual work flight plan.

In the example shown in FIG. 13, three work locations 141, 145, and 147 are designated in the area 81, and one actual work flight plan 143 is prepared for one of the work locations 141. , Another actual work flight plan 149 is prepared for the two work sites 145 and 147. If there are multiple work locations, create one actual work flight plan that covers all of those work locations, or create two or more actual work flight plans that share different work locations as shown in the example shown. May be appropriately selected according to the user's request or the convenience of flight time.

When a plurality of actual work flight plans are prepared for one area 81, work is carried out with a plurality of actual work drones 5 based on each flight plan, or according to different flight plans at different dates and times. You can carry out other tasks.

Each actual work flight plan includes the flight path and speed of the actual work drone 5. For example, the illustrated actual work flight meter 149 takes off from the landing point P1 and goes to the first work point 145, goes around the work place 145 so that the actual work (for example, pesticide spraying) can be applied to the entire area, and then It includes flight paths such as going to the next work site 147, going around it in the same way, and then returning to the departure / arrival point P1 to land.

Each actual work flight plan further includes control instructions for driving and controlling the actual work device 25 during the flight of the actual work drone 5. The control instructions include, for example, the position where the actual work (for example, pesticide spraying) should be executed (for example, the work start position and the end position) and other control values (for example, the amount of pesticide sprayed per unit time). Can be included.

FIG. 14 shows an example of data configuration of an actual work flight plan.

As shown in FIG. 14, the actual work flight plan 151 includes, for example, its ID (identification name) 153, the flight path (for example, the latitude Xr and the longitude Yr of the change that should change the flight direction or speed on the path. Altitude Zr) and flight speed (eg, speed Vr from each change to the next change), and work position (eg latitude Xs and longitude Ys and altitude Zs at the start of the work, as well as the end position of the work. Latitude Xe, Longitude Ye, and Altitude Ze) and control values (for example, target value C1 for the first control amount, target value C2 for the second control amount, and target value C3 for the third control amount from each start position to the end position. , Etc.) 157 is included.

In addition, when a camera that shows the work situation is mounted on the actual work drone 5, the position 159 for taking a picture with the camera (for example, latitude Xs, longitude Ys, altitude Zs, and shooting angle As of the shooting position) is also determined. , May be included in the actual work flight plan 151.

The actual work item plan 151 having such a configuration is associated with the corresponding user ID 85, area ID 87, and work location ID 135, and is managed in the flight plan database 55.

FIG. 15 shows a flow of a control example of setting an actual work flight plan.

As shown in FIG. 15, the user logs in to the data server 7 from the operation terminal 9 (steps S71 to S72). In response to the user request, the data server 7 selects the actual work flight plan associated with the user ID from the flight plan database 55, and sends a list of the selected actual work flight plans to the operation terminal (step S72). ). The operation terminal 9 displays the list (step S73).

The user selects a desired actual work flight plan from the displayed list and requests the data server 7, and the data server 7 sends the actual work flight plan to the operation terminal, whereby the actual work flight plan is sent to the operation terminal 9. Is downloaded (steps S74-S75).

After that, the user sends the actual work flight plan from the operation terminal 9 to the actual work drone 5 (step S76). The actual work drone 5 installs the received actual work flight plan on its own controller 27 (S77).

FIG. 16 shows a flow of a control example of actual work using an actual work drone.

After placing the actual work drone 5 at the departure / arrival point in the target area, the user operates the radio controller 29 for the actual work drone 5 to first instruct the actual work drone 5 to start the work (step S81). In response to the instruction, the actual work drone 5 takes off (step S82), and each of them flies along the flight path of the flight plan by autopilot control according to the actual work flight plan by the controller 27. The actual work (for example, pesticide spraying) is executed at the work position (for example, the section from each work start position to the end position) (step S83). Then, the actual work drone 5 records the result of the performed work (for example, the start position, the end position, the time, and the amount of spraying of the pesticide) (step S84).

While observing the flight of the actual work drone 5, the user has a situation in which the flight state (for example, the flight position and the flight speed) should be corrected (for example, the flight position of the actual work drone 5 or the flight position due to the influence of wind, etc.). When the speed deviates from the flight plan, it becomes necessary to change the flight state to avoid collision with other objects or danger, etc.), the radio control 29 is operated to assist in making necessary corrections. Send a flight instruction to the actual work drone 3 (step S85). The actual work drone 5 corrects the flight state (position or speed, etc.) according to the correction instruction by processing the auxiliary instruction as, for example, an interrupt to the autopilot control (step S86).

When the above modification process is completed, the actual work drone 5 returns to the autopilot control (step S87) and continues the flight and the actual work according to the actual work flight plan (S83). After that, after flying all routes and performing actual work at all work positions, the actual work drone 5 lands at the landing point specified in the flight plan (step S88).

The work result 161 recorded in the actual work drone 5 is sent to the operation terminal 7 and recorded in the operation terminal 7 (steps S89 to S90). This data transfer may be performed by wireless communication during flight, by wireless communication or wired communication after landing, or by removing the data storage recording the work result 161 from the actual work drone 3 and attaching it to the operation terminal 7. It may be done by doing so.

After that, as already described with reference to FIG. 2, the work result 163 recorded in the operation terminal 7 is sent to the data server 7 (step S16), and the data server 7 sends the work result later. Analyze for effective use (step S17).

FIG. 17 shows the overall configuration of the work support system according to the second embodiment of the present invention.

The work support system 100 according to this embodiment facilitates the setting of a work plan for obtaining the expected work effect, in addition to the components of the work support system 1 according to the first embodiment shown in FIG. It has some additional components to make it. In the following description and reference drawings of the work support system 100, the same reference numbers are given to the elements common to the system 1 of the first embodiment, and redundant description is omitted.

As shown in FIG. 17, in the work support system 100 according to the second embodiment, the data server 101 and the operation terminal 111 have the data server 7 and the operation terminal 9 of the work support system 1 according to the first embodiment, respectively. In addition to the components, it has several additional components.

That is, the data server 101 has a work plan database 103, a work plan proposal database 105, and an analysis unit 107 as additional components. The operation terminal 111 has a peculiar part detection unit 113, a work plan input unit 115, a proposal presentation unit 117, and a work selection unit 119 as additional components.

The peculiar part detection unit 113 of the operation terminal 111 automatically displays the survey result of the area, that is, the area image (for example, the image of the registered specific field) displayed by the image display unit 45 on the display screen of the operation terminal 111. In the area image, a peculiar part (for example, a region where a disease or the like in the field is presumed to occur) is detected. The singularity detection unit 113 displays the detected singularity region (for example, a frame line showing the outline of the region) in the area image on the display screen.

The singular location detection unit 113 may be provided in the data server 101 instead of being provided in the operation terminal 111. For example, the analysis unit 107 of the data server 101 may have a singular location detection unit 113. The singular part detection unit 113 is configured by using a deep neural network like the symptom analysis unit 108 or the work analysis unit 109 in the analysis unit 107 described later, and is an inference method for detecting a singular part from a regional image by deep learning. May be machine-learned.

In the work plan input unit 115 of the operation terminal 111, the image display unit 45 displays an area image (for example, an image of a registered specific field) on the display screen of the operation terminal 111, and the user can register the work location. When a work location is specified in the area image using 47, the user can input a work plan for the specified work location. That is, the work plan input unit 115 displays on the display screen a work plan input tool for the user to input an arbitrary work plan for each work location on the display screen. By operating the work plan input tool, the user can input an arbitrary work plan to the system 100 for each work location. When the input of the work plan is completed (for example, when the user requests the registration of the work plan on the display screen), the work plan input unit 115 sends the input work plan to the data server 101, and the work plan responds. It is registered in the work plan database 103 of the data server 101 in association with the work place to be performed.

Here, the work plan is data that defines the actual work to be performed for each work location. In this embodiment, the work plan for each work place is, for example, the name of the symptom of the work place (for example, the name of a disease or a physiological disorder) and the name of the actual work to be performed according to the symptom (for example, the name of the actual work). Includes pesticides, fertilizers and other maintenance work to be applied to the work site). Alternatively, the work plan may not include the symptom name but only the work name, or in addition to the symptom name and the work name, additional information (eg, information or images identifying the work location, or the work location). Agricultural chemicals, fertilizer application amount, etc.) may be included.

The flight plan creation unit 71 of the data server 101 determines the work plan (for example, the symptom name and the actual work name for each peculiar part in each field) of each work place in each area registered in the work plan database 103 in each area. It can be used as follows when creating a flight plan. That is, for example, when different work plans are registered for different work places in the same area, the flight plan creation unit 71 classifies the work places given the same work plan (same work name) into the same group. Then create one flight plan for each group, that is, different flight plans for different groups. For example, in one field, if a work plan for applying pesticide C is registered in work sites A and B, and a work plan for applying another pesticide F is registered in different work sites D and E, a flight plan is registered. The preparation unit 71 creates a flight plan for applying pesticide C to the groups of work sites A and B, and creates a flight plan for spraying another pesticide F to the groups of different work sites D and E. To do.

When the user inputs a work plan for each work location using the work plan input unit 115, the proposal presentation unit 117 of the operation terminal 111 proposes a work plan for each work location from the work plan proposal database 105 of the data server 101. Is read, and each work plan proposal is displayed in association with each work location on the display screen of the operation terminal 111.

Here, the work plan proposal is a work plan proposal that is generated by inference by the analysis unit 107 of the data server 101 and is recommended for each work location. The work plan proposal of each work location generated by the analysis unit 107 is associated with each work location and stored in the work plan proposal database 105. A more specific configuration and function of the analysis unit 107 will be described later.

The work plan proposal displayed on the display screen of the operation terminal 111 serves as a reference for the user when the user inputs the work plan using the work plan input unit 115. Work plan proposals are helpful, especially for users who are inexperienced in determining work plans, in order to determine more appropriate work plans. The higher the inference ability of the analysis unit 107, the higher the performance of the work plan proposal to help the user. In order to enhance the inference ability of the analysis unit 107, the analysis unit 107 has a configuration described later.

The work selection unit 119 of the operation terminal 111 specifies that the actual work drone 5 wants to execute this time from the flight plans for a specific area (for example, a specific field) read from the flight plan database 55 by the flight plan input unit 33. Let the user choose a flight plan for the work. For example, when a flight plan for spraying pesticide C and a flight plan for spraying another pesticide F are read by the flight plan input unit 33 for the specific field, the work selection unit 119 makes those flights. Display the plan on the display screen and let the user select the desired flight plan. The work selection unit 119 provides the selected flight plan to the flight plan output unit 35. The selected flight plan is supplied from the flight plan output unit 35 to the controller 27 of the actual work drone 5.

The analysis unit 107 of the data server 101 is an image of each area (for example, each field) registered from the area image database 61, the work location database 63, the area location database 51, the work plan database 103, and the work result database 65. Location of each work location in each region, work plan for each work location (for example, symptom name and actual work name), image of each region (each work location) after actual work based on each work plan, after work implementation Read the data such as the work plan (especially the symptom name) re-entered by the user based on the image. The analysis unit 107 creates (that is, learns) an inference method for automatically generating a work plan proposal for each work location by performing machine learning using the read data (once created). In other words, it also includes improving the learned inference method). In addition, the analysis unit 107 creates a work plan proposal corresponding to the image from the image of each work location by using the inference method created by the machine learning. The created work plan proposal for each work location is associated with each work location and stored in the work plan proposal database 105.

Here, the proposal of the work plan for each work place is, for example, the proposal of the presumed symptomatism (for example, the name of the disease) for the work place and the actual work recommended for the work place according to the symptomatism (for example, the actual work recommended for the work place). For example, it includes proposals for pesticide names, fertilizer names, etc.).

FIG. 18 shows the flow of an overall control example of the work support system 100.

After the area image of the area desired by the user is registered in the data server 101 (step S9), the user can display the area image on the display screen of the operation terminal 111 (step S20). In step S20, the operation terminal 111 automatically analyzes the area image, automatically detects the peculiar part, and displays the detected peculiar part in the area image on the display screen. Then, the user visually identifies the work portion with reference to the displayed peculiar portion, and requests the operation terminal 111 to register the identified work portion. Then, the work location is notified from the operation terminal 111 to the data server 101 and registered in the data server 101 (step S21).

When the work location is registered in the data server 101 in step S21, the analysis unit 107 of the data server 101 executes inference for the image of the registered work location and automatically proposes a work plan for the work location. (Step S22). The operation terminal 111 receives the work plan proposal from the data server 111, and displays the work plan proposal on the display screen in association with the work location displayed on the display screen (step S23).

The user determines a work plan (for example, a symptomatological name and an actual work name) for the work place by referring to the work plan proposal for the work place displayed on the operation terminal 111, inputs the work plan to the operation terminal 111, and inputs the work plan to the operation terminal 111. Request registration from the operation terminal 111 (step S23). Then, the input work plan is associated with the work location, notified to the data server 101, and registered there (step S25).

After the actual work is carried out on a certain area, a photographic image of the area (where the effect of the actual work is shown) is taken again to obtain the area image, and further, based on the area image. The user re-identified the work location (the same location as the previous work location, in other words, it may contain a different location, or there may be no work location), and the user for each work location. When the newly determined symptom name (the same symptom as before or a symptom different from the previous one may be included) is input and registered, the inference unit 107 of the data server 101 sets the work plan of the actual work performed. , The image showing the effect of the work and the work location and symptom name re-identified based on the image are received as teacher data, machine learning is performed, and the inference method of the work plan is automatically created, that is, learned. (Step S26). The analysis unit 107 can apply the learned inference method to the inference in step S22 to be executed later.

With the above control, the inference method for generating the work plan proposal possessed by the data server 101 will be improved to a higher performance as the photography and the actual work of many areas are repeated. , It will be possible to provide users with more appropriate work plan proposals. This makes it easier for the user to design a work plan to obtain the expected effects.

See FIG. 17 again. The analysis unit 107 of the data server 101 includes a symptomatology analysis unit 108 and a work analysis unit 109. The symptom analysis unit 108 analyzes the image of each work place (the part corresponding to each work place in the area image), estimates the symptom of the work place, and estimates the symptom (for example, disease name and physiology). (Failure name, etc.) is stored in the work plan proposal database 105 as a symptom proposal in association with the work location. The work analysis unit 109 estimates the actual work (for example, pesticide name, fertilizer name, etc.) recommended to be applied to the work place from the symptom of each work place, and uses the estimated actual work as a work proposal. , Store in the work plan proposal database 105 in association with the work location. Symptom proposals and work proposals for each work location constitute a work plan proposal for that work location.

The symptomatology analysis unit 108 and the work analysis unit 109 can be configured to perform machine learning and inference suitable for their respective purposes by using, for example, a neural network.

19 and 20 show configuration examples of the symptomatology analysis unit 108 and the work analysis unit 109, respectively.

As shown in FIG. 19, the symptomatology analysis unit 108 has the following two types of deep neural networks (hereinafter, abbreviated as DNN). One is symptomatology learning DNN121 and the other is symptomatology inference DNN123.

The symptom learning DNN121 inputs a large amount of images and symptoms of a large number of work locations, images and symptoms from the past to the present, and a history of actual work (transition history) as teacher data 125, and performs machine learning, for example, deep. Learn, and thereby learn inference methods for inferring symptoms from images (ie, build inference neural networks).

The symptom inference DNN123 has an inference method (that is, an inference neural network) created at a certain point in the past by machine learning by the symptom learning DNN 121, and the inference method (inference neural) uses the image and history data of each work location. The network) is input to perform inference, and the symptom proposal data 129 for the work location is output.

The symptomatology learning DNN 121 and the symptomatology inference DNN 123 may be configured as different hardware or different computer software. In that case, by replicating the inference method created by performing a certain amount of learning in a certain period of time to the symptom inference DNN123, the symptom inference DNN123 can execute the inference method thereafter. By repeating such replication at an appropriate period, the inference performance of the symptom inference DNN123 improves with the passage of time.

Alternatively, the symptomatology learning DNN 121 and the symptomatology inference DNN 123 may be configured as the same hardware or the same computer software. In that case, the same hardware or computer software can act as a symptomatology learning DNN121 at one time and as a symptomatology inference DNN123 at another time. By alternately repeating learning and inference in different time zones in this way, the learning results of the previous time zone are repeatedly used for inference in the next time zone, so that symptom inference DNN123 with the passage of time. The inference performance of is improved. Alternatively, the same hardware or computer software may be configured to perform learning and inference in parallel (ie, act as symptomatology learning DNN121 and symptomatology inference DNN123 in parallel).

As shown in FIG. 20, the work analysis unit 109 also has two types of DNNs like the symptomatology analysis unit 108 described above. They are work learning DNN131 and work inference DNN133.

The work learning DNN131 captures the symptoms of a large number of work locations, the actual work applied to them, the images of the work locations taken again after the actual work is performed, and the symptoms judged again based on the images. Input a large amount of teacher data 135 to perform machine learning, such as deep learning, thereby learning inference methods for inferring the actual work to apply to it from the symptoms (ie, inference neural networks). To make up).

The work inference DNN 133 has an inference method (that is, an inference neural network) created at a certain point in the past by machine learning by the work learning DNN 131, and inputs the symptom of each work location into the inference method (inference neural network). Then, inference is performed, and work proposal data 139 for the work location is output.

The work learning DNN 131 and the work inference DNN 133 may be configured as different hardware or different computer software, similarly to the symptomatology learning DNN 121 and the symptomatology inference DNN 123 described above. In that case, by duplicating the inference method created by performing a certain amount of learning in a certain period of time to the work inference DNN 13, the work inference DNN 133 can execute the inference method thereafter. By repeating such duplication at an appropriate period, the inference performance of the work inference DNN133 is improved with the passage of time.

Alternatively, the work learning DNN 131 and the work inference DNN 133 may be configured as the same hardware or the same computer software. In that case, the same hardware or computer software can operate as a work learning DNN131 at one time zone and as a work inference DNN133 at another time zone. By alternately repeating learning and inference in different time zones in this way, the learning results of the previous time zone are repeatedly used for inference in the next time zone, so that work inference DNN133 with the passage of time. The inference performance of is improved. Alternatively, the same hardware or computer software may be configured to perform learning and inference in parallel (ie, act as work learning DNN 131 and work inference DNN 133 in parallel).

21 to 23 show an example of a graphic interface for a user to identify a work location and a work plan on the display screen of the operation terminal 111.

FIG. 21 shows an example of an image of the interface 200 when specifying a work location. An image of an area (for example, a field) desired by the user is displayed in the area image window 201 (in this example, a part of the field is enlarged and displayed so that the state of each crop can be understood). When the user operates the area selection button 203, the area image is analyzed by the singular part detection unit 113, the singular part in the area is automatically detected, and the detected singular part is the area selection tool. Proposed by 213. The user can refer to this proposal when visually locating the work site. The user can operate the area selection tool 213 as necessary to identify the work location. The user can input the name of the work place by using the work place name input tool 205. By operating the registration button 211, the user can register the work location (the position of the region specified by the region selection tool 211) and its name in the data server 101.

FIG. 22 shows an image example of the graphic interface when identifying the symptom of the work place. Once the work locations 221 and 223 are registered, the data server 101 estimates the symptom from the images of the work locations 221, 223 and generates a symptom proposal. The symptomatological proposals 225 and 227 are displayed on the area image window 201 in association with each work location 221 and 223. When identifying the symptom of each work place 221 and 223, the user can refer to the symptom proposal. The user can operate the symptom input tool 207 to specify the symptom name of each work location 221 and 223. By operating the registration button 211, the user can register the symptom names of the work locations 221 and 223 in the data server 101.

FIG. 23 shows an image example of the graphic interface when determining the actual work for each work location. Once the symptoms of the work locations 221 and 223 are registered, the data server 101 estimates the recommended actual work from the symptoms of the work locations 221 and 223 and generates a work proposal. The work proposals 235 and 237 are displayed in association with each work location 221 and 223 on the area image window. The user can refer to the work proposal when identifying the actual work of each work location 221 and 223. The user can operate the actual work input tool 209 to determine the actual work name for each work location 221 and 223. By operating the registration button 211, the user can register the actual work name for each work location 221 and 223 in the data server 101.

FIG. 24 shows another configuration example of the analysis unit 107 of the data server 101.

In the configuration example of FIG. 24, the analysis unit 107 can output the work plan proposal of each work location, that is, the symptom proposal and the actual work proposal together without separating them. That is, the analysis unit 107 has a work plan learning DNN 141 and a work plan inference DNN 143.

Work plan learning DNN141 includes images and work plans (symptom names and actual work names) of a large number of work locations, images and symptoms of the same work locations after actual work is performed based on the work plan, and from the past to the present. A large amount of images and symptoms of the work location and the history of actual work (transition history) are input as teacher data 145 to execute machine learning, for example, deep learning, thereby inferring a work plan from the images. Learn the inference method for (that is, build an inference neural network).

The work plan inference DNN143 has an inference method (that is, an inference neural network) created at a certain point in the past by machine learning by the work plan symptom learning DNN141, and infers the image of each work part and the data of the above history. Input to the method (inference neural network) to perform inference, and output the work plan proposal 149 for the work location.
The work plan learning DNN 141 and the work plan inference DNN 143 may be configured as different hardware or different computer software, or may be configured as the same hardware or the same computer software.

FIG. 25 shows another configuration example of the analysis unit 107.

In the configuration example of FIG. 25, the analysis unit 107 proposes a work plan for each region, not for each work location in the region (there are the positions of one or more work locations in the region and the symptoms of each work location). Name and actual work name are included) can be created and output. That is, the analysis unit 107 has a work plan learning DNN 151 and a work plan inference DNN 153.

The work plan learning DNN 151 includes an overall image of a large number of areas, the position of each work location in the area, the symptom name and the actual work name, and the overall image, the work location and the symptom after the actual work is performed in those areas, and the symptom. , The whole image of those areas from the past to the present, the work place, the symptom, and the history of the actual work (transition history) are input in large quantities as teacher data 155, and machine learning, for example, deep learning, is executed, thereby performing machine learning, for example, deep learning. , Learn the inference method for inferring the work plan (the position of the work part in the area, the symptom name and the actual work name of each work part) from the whole image of the area (that is, create an inference neural network).

The work plan inference DNN153 has an inference method (that is, an inference neural network) created at a certain point in the past by machine learning by the work plan symptom learning DNN151, and images of work locations in each region and the above history data are obtained. Input to the inference method (inference neural network) to perform inference, and output work plan proposal 159 for the area.

The work plan learning DNN 151 and the work plan inference DNN 153 may be configured as different hardware or different computer software, or may be configured as the same hardware or the same computer software.

See FIG. 17 again. The data stored in the regional image database 61, the work location database 63, the regional location database 51, the flight plan database 55, the work plan database, the work result database, the work plan proposal database 105 of the data server 101, and the analysis unit 107 are machines. The inference method created by learning, that is, the inference neural network, can be used for various useful purposes other than the use of displaying on the operation terminal 111 to help the user. Therefore, the data stored in those databases and all or any part of the inference neural network can be output from the data server 101 to the outside.

The work support systems according to the two embodiments of the present invention have been described above, but these work support systems are based on the data collected in the survey by the drone (for example, the image of the area such as the field obtained by photography). A correction system for performing data correction for removing included noise (for example, an error from the actual state such as color tone and brightness of a photographic image caused by an environmental condition such as the weather and time zone at the time of shooting) may be provided. For example, the work support systems 1 and 100 according to the two embodiments described above may each include such a correction system in, for example, data servers 7, 101 or, for example, operating terminals 9, 111.

Hereinafter, an example of such a correction system will be described. The correction system illustrated below is for correcting a regional image obtained by taking a picture with a drone, and can be provided in the work support systems 1 and 100 according to the above two embodiments.

FIG. 26 shows an example planar design of an image correction scale used to utilize the correction system.

The image correction scale 161 shown in FIG. 26 is, for example, a rectangular flat plate, and the surface shown in FIG. 26 faces upward on the ground surface in or around the area to be photographed when taking a picture with a drone. Be placed.

The image correction scale 161 has a plurality of reference color scales 163C1 to 163C7, 163G (8 in this example, but other numbers may be used). Among them, some of the reference color scales 163C1 to 163C7 (7 in this example, but other numbers may be used) are colored in different predetermined chromatic colors, and these are collectively referred to as "colors" below. The scale is called "163C". Further, the remaining reference color scale 163G (one in this example, but other numbers may be used) is colored in a predetermined achromatic color (for example, pure white or a predetermined gray color). This is hereinafter referred to as "gray scale".

The different chromatic colors painted on the plurality of reference color scales 163C1 to 163C7 of the color scale 163C are predetermined in order to judge the state of the target area (that is, the tone values of the element colors such as RGB are predetermined. It is a different standard color. For example, when the application area of the image correction scale 161 is a rice field, the reference colors thereof are different greenish colors predetermined for determining the state of the rice from the leaf color of the rice. An identification code is defined in advance for each reference color, and each reference color scale 163C1 to 163C7 is accompanied by a display of the identification code. The reference color of the grayscale 163C is also an achromatic color having a predetermined element color tone value, and the display of the predetermined identification code is attached to the grayscale 163C.

All reference color scales 163C1 to 163C7, 163G are, for example, perfectly circular. When the drone takes a picture of an area, it takes not only the area but also the surroundings in the vicinity of the area, and therefore also the image correction scale 161. The area image obtained by the photography includes not only the image of the area but also the image of the vicinity of the area, and also includes the image of the image correction scale 161. Regardless of the direction in which the image of the image correction scale 161 is oriented in the regional image, if each reference color scale 163C1 to 163C7, 163G is a perfect circle, each reference color is selected from the regional images obtained by shooting. It is easy to identify and extract images of scales 163C1 to 163C7 and 163G by image processing.

FIG. 27 shows an example of the dimensional conditions satisfied by the panel on which each reference color scale in the example of the image correction scale shown in FIG. 26 is displayed.

As shown in FIG. 27, each reference color scale 167 in the image correction scale 161 is a perfect circle as described above. Each reference color scale 167 is displayed on the surface of the unit panel 165, and the unit panel 1 is, for example, a square. The reference color identification code 169 of the reference color scale 167 is displayed in the background region (which has a predetermined color different from the reference color) outside the reference color scale 167 on the surface of the unit panel 165. By integrating a predetermined number of unit panels 165 on which reference color scales 167 of different colors are drawn, a flat plate-shaped image correction scale 161 as shown in FIG. 26 is configured.

The size of the unit panel 165 is chosen to include the circumscribed square of the reference color scale 167 (shown by the alternate long and short dash line) (ie, the circumscribed square does not extend outside the unit panel 165). Further, the display position of the identification code 169 on the unit panel 165 exists outside the circumscribed square regardless of the direction of the circumscribed square, as shown by the auxiliary line of the broken line (that is, the circumscribed square). It is chosen not to go inside).

As a result, in the image processing for identifying the reference color scale 167 from the images taken by the drone, the background color and the reference color are within the circumscribed square regardless of which direction the unit panel 165 is facing in the image. Since there are only two color regions, it is easy to accurately identify and extract the reference color scale 167.

FIG. 28 shows a configuration example of a correction system for correcting a regional image using the image of the image correction scale 161 shown in FIG. 26.

As shown in FIG. 28, the correction system 171 includes an image input unit 173, a color scale search unit 177, a color scale pixel value storage unit 179, a grayscale search unit 181 and a grayscale pixel value storage unit 183, and a reference pixel value storage unit. It has a regional polygon data storage unit 187, an image separation unit 189, a first color tone adjustment unit 191 and a second color tone adjustment unit 193, an image composition unit 195, and a color analysis unit 197.

The image input unit 173 inputs a regional image of the target area obtained by taking a photograph. The regional image of the target area can be input from the regional image database 61 in the work support systems 1 and 100 according to the above two embodiments (see FIGS. 1 and 17).

The color scale search unit 177 identifies the images of the reference color scales 163C1 to 163C7 of the color scale 163C from the input area image by image processing, and all the pixels in the identified reference color scale image. Extract a set of values for (eg, a set of RGB values). The color scale pixel value storage unit 179 stores all pixel value sets in the extracted reference color scale image.

The grayscale search unit 181 identifies an image of grayscale 163G from the input area image by image processing, and sets the values of all pixels in the identified grayscale image (for example, RGB values). Set) is extracted. The grayscale pixel value storage unit 183 stores the entire pixel value set of the extracted grayscale image.

The reference pixel value storage unit 185 stores a predetermined pixel value set (for example, a set of RGB values) (hereinafter, referred to as a reference pixel value set) of all the reference color scales 163C1 to 163C7, 163G of the image correction scale 161. To do.

The regional polygon data storage unit 187 stores data (hereinafter, referred to as regional polygon data) indicating the shape and position of the target area (for example, the geographical coordinate value of the outline of the target area). The regional polygon data of the target area exists in the regional location database 51 in the work support systems 1 and 100 (see FIGS. 1 and 17) according to the above two embodiments.

The image separation unit 189 uses the area polygon data 187 to combine the area image (for example, an image obtained by taking a photograph of the field) input from the image input unit 173 with the part of the target area (for example, the image of the field). , Separate into parts other than the target area (for example, an image of the peripheral area in the vicinity of the field taken together with the field).

The first color tone adjusting unit 191 performs a process of correcting the color of the image of the target area portion from the image separating unit 189. In this correction process, the pixel value sets of the reference color scales 163C1 to 163C7 of the color scales 163C from the color scale pixel storage unit 179 and the reference color scales 163C1 to 163C7 of the color scales 163C from the reference pixel value storage unit 185. A reference pixel value set is used.

The second color tone adjustment unit 193 performs a process of correcting the color of the image of the portion other than the area from the image separation unit 189. In this correction process, a grayscale 163G pixel value set from the grayscale pixel storage unit 183 and a grayscale 163G reference pixel value set from the reference pixel value storage unit 185 are used.

The image synthesizing unit 195 combines the corrected image of the target area portion and the image of the portion other than the corrected target area to synthesize the corrected regional image.

The color analysis unit 197 analyzes the pixel values of each part of the corrected area image to generate information for later work, such as detection of a singular region. For example, the image of the target area portion in the regional image is divided into small subdivisions, and statistical values such as the average value and standard deviation of the pixel value set of all the pixels in each subdivision are calculated. Since these statistical values are useful for finding, for example, a peculiar region where a disease or a physiological disorder of agricultural products in the field is occurring, for example, the work support system 100 according to the second embodiment described above (see FIG. 17). It can be used in the analysis unit 107 of the above, the singular place detection unit 113, or the like.

FIG. 29 shows an example flow of control performed by the correction system 171 shown in FIG. 28.

As shown in FIG. 29, the area image obtained by photography is input, and the image of the color scale 163C (that is, the image of the reference color scales 163C1 to 163C7) is searched from the input area image (step S101). ), The pixel value of the image of the found color scale 163C (that is, the pixel value set of all the pixels of the image of the reference color scales 163C1 to 163C7) is extracted and saved (step S102). Then, based on the stored pixel value of the color scale 163C and the corresponding reference pixel value determined in advance (that is, the reference pixel value set of the reference color scales 163C1 to 163C7 from the reference pixel value storage unit 185). The first correction function for correcting the former color tone to match the latter color tone is calculated (step S103).

Further, the grayscale 163G image is searched from the input area image (step S104), and the found pixel value of the grayscale 163G (that is, the pixel value set of all the pixels of the grayscale 163G image) is saved (that is, the pixel value set of all the pixels of the grayscale 163G image). Step S105). Then, based on the saved pixel value of the gray scale 163G and the predetermined reference pixel value of the gray scale 163G (that is, the reference pixel value set of the gray scale 163G from the reference pixel value storage unit 185), the former A second correction function is calculated to correct the color tone of the above to match the latter color tone (step S106).

Then, the image of the target area portion is separated from the input area image, and the color tone of the image is corrected by using the first correction function (step S107). Further, the image of the portion other than the target area is separated from the input area image, and the color tone of the image is corrected by using the second correction function (step S108). As a result, even if the color tone of the regional image obtained by photography deviates from the actual regional image, the image of the regional portion (for example, the image of the rice field) uses the actual state of the region as a reference of the color scale 163C. It is corrected so that it can be evaluated more appropriately based on the image, and the image of the part other than the target area is corrected so that the color tone looks more natural.

Then, the corrected regional portion and the image of the other portion are combined, and the corrected regional image is combined (step S109).

Then, the corrected regional image is analyzed, and a predetermined statistical value (for example, an average pixel value set of a large number of subdivisions of the regional image and its standard deviation) is calculated (step S110). The calculated statistics can be used for later work, eg, to find singular areas of interest.

Although some embodiments of the present invention have been described above, those embodiments are merely examples for explanation, and the scope of the present invention is not limited to those embodiments. The present invention can be implemented in various forms different from the above-described embodiment.

For example, the present invention is not limited to spraying pesticides in the field, but can also be applied to work support systems for various other uses. For example, the present invention can be applied to a wide range of applications such as transfer of an object using a drone in a material storage area, monitoring and maintenance by a drone of a power transmission line or a steel tower, or taking a photograph or a moving image of a district or place desired by a user.

Depending on the application, a process including only the actual work flight can be performed instead of the process including the two-stage flight of the survey flight and the actual work flight as in the above embodiment. For example, if you want to shoot a video of a certain location, the user specifies the location on the operating terminal, the server creates a flight plan for shooting the location, and the user downloads the flight plan and installs it on the drone. , And you can do a simpler process, such as taking a drone shoot flight.

Also, depending on the application, a process involving more multi-step flight can be performed. For example, in the case of pesticide spraying in a field, after a certain period of time has passed from the day when the pesticide was sprayed, a survey flight may be conducted again (in this case, a survey flight for the entire area may be conducted) in order to investigate the effect of the pesticide. More complex processes, such as conducting survey flights that focus on the work area), repeating survey flights and pesticide spraying on a regular basis, or flying for sowing and fertilization. This can be done by applying the present invention.

Further, a part or all of the above-mentioned data server may be implemented in the operation terminal used by the user. For example, a software tool for creating a flight plan is installed on the operating terminal 7, and the user can create a flight plan by himself using the tool, or the tool automatically creates a flight plan. It may be like this.
The present invention may also be applied when a drone capable of moving methods other than flight, such as ground traveling, water navigation, and underwater diving, is used depending on the type or situation of the work object.

1 Work support system 3 Survey drone 5 Actual work drone 7 Data server 9 Operation terminals 17, 27 Controllers 19, 29 Radio controller 31 Area registration unit 33 Flight plan input unit 35 Flight plan output unit 37 Photographed image input unit 39 Photographed image Output unit 41 Shooting position input unit 43 Shooting position output unit 45 Image display unit 47 Work location registration unit 48 Work result input unit 49 Work result output unit 51 Area position database 53 Three-dimensional map database 55 Flight plan database 57 Photographed image database 59 Shooting Location database 61 Area image database 63 Work location database 65 Work result database 71 Flight plan creation unit 73 Photographed image input unit 75 Photographed position input unit 77 Photographed image combination unit 79 Analysis unit 100 Work support system 101 Data server 111 Operation terminal 103 Work plan Database 105 Work plan proposal database 107 Analysis unit 108 Symptom analysis unit 109 Work analysis unit 113 Singularity detection unit 115 Work plan input unit 117 Proposal presentation unit 119 Work selection unit 121 Symptom learning deep neural network 123 Symptom inference deep neural network 131 Work learning Deep Neural Network 133 Work Inference Deep Neural Network 141, 151 Work Plan Learning Deep Neural Network 143, 153 Work Plan Inference Deep Neural Network 161 Image Correction Scale 163C Color Scale 163G Gray Scale 165 Unit Panel 167 Reference Color Scale 169 Identification Code 171 Correction System

Claims (7)

In a drone work support system configured to be able to communicate with one or more drones, or to communicate with one or more users using the one or more drones.
The first work result storage means for receiving and storing the result of the first work performed on the first object,
A work report creation means for creating a work report representing the stored result of the first work, and
A work report providing means for providing the created work report to at least one of the users, and
A second targeting tool configured to allow at least one of the users to identify a second target according to the result of the first work displayed in the work report is provided at least by the user. A second means of providing targeting tools to one person,
A second object storage means that receives the identification of the second object from at least one of the users and stores the position of the second object, and
Information processing related to the second work related to the second object is performed according to the stored position of the second object, and the result of the information processing is provided to at least one of the users or at least one of the drones. A drone work support system equipped with a second target information processing means. A symptom identification tool configured to allow at least one of the users to identify the symptom of the second target according to the result of the first work displayed in the work report is provided by at least one of the users. Means for providing symptom identification tools to one person,
A symptom memory means for receiving the identification of the symptom from at least one of the users and memorizing the symptom,
The drone work support system according to claim 1, further comprising. In response to the result of the first operation, a singular part is automatically detected from the first target, and the detected singular part is sent to at least one of the users as a proposal of the second target. The drone work support system according to any one of claims 1 to 2, further comprising a second target proposal means to be provided. A claim further comprising a work content proposing means that automatically generates a work content proposal for the second work in response to the result of the first work and provides the work content proposal to at least one of the users. The drone work support system according to any one of items 1 to 3. Claim 1 further comprising a symptom suggestion means that automatically generates a symptom suggestion for the second target in response to the result of the first work and provides the symptom suggestion to at least one of the users. The drone work support system according to any one of 4 to 4. In a drone work support method performed by a computer system configured to be able to communicate with one or more drones or to be able to communicate with one or more users using the one or more drones.
A step to receive and memorize the results of the first work performed on the first object,
Steps to create a work report representing the results of the first work stored,
A step of providing the created work report to at least one of the users,
A second targeting tool configured to allow at least one of the users to identify a second target according to the result of the first work displayed in the work report is provided at least by the user. Steps to offer to one person,
A step of receiving the identification of the second target from at least one of the users and storing the position of the second target, and
Information processing related to the second work in the second object is performed according to the stored position of the second object, and the result of the information processing is provided to at least one of the users or at least one of the drones. Drone work support method with steps. In a machine-readable computer program for causing a computer system configured to communicate with one or more drones, or with one or more users using the one or more drones, to execute a drone work support method. The drone work support method
A step to receive and memorize the results of the first work performed on the first object,
Steps to create a work report representing the results of the first work stored,
A step of providing the created work report to at least one of the users,
A second targeting tool configured to allow at least one of the users to identify a second target according to the result of the first work displayed in the work report is provided at least by the user. Steps to offer to one person,
A step of receiving the identification of the second target from at least one of the users and storing the position of the second target, and
Information processing related to the second work in the second object is performed according to the stored position of the second object, and the result of the information processing is provided to at least one of the users or at least one of the drones. A computer program with steps.

Images