WO2023233832A1

WO2023233832A1 - Method, program, and information processing device

Info

Publication number: WO2023233832A1
Application number: PCT/JP2023/014996
Authority: WO
Inventors: 隆洋中村
Original assignee: 株式会社ポーラスター・スペース
Priority date: 2022-05-31
Filing date: 2023-04-13
Publication date: 2023-12-07
Also published as: JP2023177352A; JP2023176173A

Abstract

The present invention causes a processor to execute: a first step of storing, in a memory, a trained model 126 for detecting that a crop under observation is in a specific state, the trained model 126 being used for comparison with a spectrum image 127 captured by a multispectral camera; a second step of generating the spectrum image 127 by capturing an image of a region under observation by the multispectral camera at a wavelength for detecting that a crop is in the specific state; a third step of identifying a region including the crop in the specific state in the region under observation on the basis of the captured spectrum image 127 and the trained model 126 for detecting that the crop is in the specific state; and a fourth step of outputting information on the identified region.

Description

Method, program and information processing device

The present disclosure relates to a method, a program, and an information processing device.

For the purpose of understanding the growth of agricultural crops (hereinafter generally referred to as "crops" in this specification), pests, and soil conditions, the land where crops are grown is imaged using a spectral camera, A technique for acquiring a spectral image of this land is known (see, for example, Patent Document 1).

International Publication No. 2017/179378

For agricultural business operators, when cultivating and harvesting crops, it is thought that the breeder knows where the crops are planted. On the other hand, when harvesting and shipping crops, it may not be possible to judge whether the crops are growing well or not just by looking at whether the vegetation is thick or not.

Therefore, for businesses that grow and harvest crops, there is a need for technology that allows them to manage the growing situation more appropriately.

Therefore, the present disclosure has been made to solve the above-mentioned problems, and the purpose is to provide a method, a program, and a program that enable businesses engaged in harvesting to more appropriately manage growing conditions. An object of the present invention is to provide an information processing device.

According to one embodiment, a method for operating a computer is provided. This method involves the first step of having a computer processor store in memory information for detecting a specific state of the crop being observed, for comparison with a spectral image taken by a multispectral camera. A second step of generating a spectral image by photographing with a multispectral camera using a wavelength to detect that the crops are in a specific state in the area to be observed; and a spectral image that is the result of the photographing; a third step of identifying areas where crops are in a specific state in the observation target area based on information for detecting that the crops are in a specific state; and a third step of outputting information on the identified area. 4 steps.

According to the present disclosure, it becomes possible for businesses that conduct harvesting businesses to manage the growing situation more appropriately.

1 is a diagram showing an overview of a system according to an embodiment. FIG. 1 is a block diagram showing the configuration of a system according to an embodiment. It is a front view showing a drone concerning an embodiment. FIG. 1 is a plan view showing a drone according to an embodiment. FIG. 1 is a block diagram showing the hardware configuration of a drone according to an embodiment. FIG. 1 is a block diagram showing the functional configuration of a drone according to an embodiment. FIG. 2 is a conceptual diagram showing a state of imaging of land by a drone according to an embodiment. FIG. 1 is a block diagram showing the hardware configuration of a satellite according to an embodiment. FIG. 1 is a block diagram showing the functional configuration of a satellite according to an embodiment. FIG. 2 is a conceptual diagram showing a state of imaging of land by a satellite according to an embodiment. FIG. 1 is a block diagram showing a hardware configuration of an information processing device according to an embodiment. FIG. 1 is a diagram showing a functional configuration of an information processing device according to an embodiment. FIG. 2 is a diagram showing a data structure of a first imaging condition DB stored in a satellite according to an embodiment. It is a diagram showing a data structure of an image DB stored in a satellite according to an embodiment. It is a figure showing the data structure of the 2nd imaging end time DB stored in the drone concerning an embodiment. It is a diagram showing the data structure of a flight plan DB stored in the drone according to the embodiment. FIG. 2 is a diagram showing a data structure of a wide area image DB stored in the information processing device according to the embodiment. It is a flowchart for explaining an example of operation of a satellite concerning an embodiment. 3 is a flowchart for explaining an example of the operation of the information processing apparatus according to the embodiment. It is a flowchart for explaining an example of operation of a drone concerning an embodiment. 12 is a flowchart for explaining another example of the operation of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of the operation of wide area image generation in the information processing device according to the embodiment. FIG. 2 is a conceptual diagram showing a procedure for determining that a crop is in a specific state using a spectral image. FIG. 3 is a diagram showing an example of a screen displayed on the information processing device according to the embodiment. FIG. 7 is a diagram illustrating another example of a screen displayed on the information processing device according to the embodiment.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all the figures explaining the embodiments, common components are given the same reference numerals and repeated explanations will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are essential components of the present disclosure. Furthermore, each figure is a schematic diagram and is not necessarily strictly illustrated.

Additionally, in the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

Furthermore, at least one processor may be a processor in a broad sense such as a hardware circuit (for example, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

In addition, in the following explanation, information such as "xxx table" may be used to explain information that provides an output in response to an input, but this information may be data of any structure, and A learning model such as a generated neural network may also be used. Therefore, the "xxx table" can be called "xxx information."

In addition, in the following explanation, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. good.

In addition, in the following description, processing may be explained using the subject "program", but a program is executed by a processor to carry out a prescribed process, and to use the storage unit and/or interface unit as appropriate. Since the processing is performed while using the processor, the subject of the processing may be a processor (or a device such as a controller having the processor).

The program may be installed on a device such as a computer, or may be located on a program distribution server or a computer-readable (e.g., non-temporary) recording medium, for example. Furthermore, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

Furthermore, in the following description, identification numbers are used as identification information for various objects, but other types of identification information than identification numbers (for example, identifiers containing alphabetic characters or codes) may be employed.

In addition, in the following explanation, when the same type of elements are explained without distinguishing them, reference numerals (or common numerals among the reference numerals) are used, and when the same kind of elements are explained separately, the element An identification number (or reference number) may be used.

In addition, in the following description, control lines and information lines are shown to be necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All configurations may be interconnected.

In this specification, a "spectral image" means an image consisting only of reflected light in a specific wavelength band (including specific wavelengths) among the reflected light from an object. For example, as shown in the embodiments described below, a spectral image can be created by passing reflected light from an object (crop) through a filter (preferably with a variable transmission wavelength band) that passes only a specific wavelength band. Only reflected light in a specific wavelength band is imaged on an image sensor, and generated based on an imaging signal output from the image sensor. Alternatively, the reflected light from the object can be imaged on an image sensor without going through a filter, and the image signal output from this image sensor can be converted (filtered) into an image signal of only a specific wavelength band through signal processing. generated. Note that an image that is a collection of a plurality of spectral images of the same object made up of only reflected light in a specific wavelength band may also be referred to as a spectral image.

In addition, as used herein, the term "flying object" refers to a mobile object that flies in the air and can be operated without a crew (e.g., an aircraft that can be remotely controlled wirelessly or autonomously, a drone, etc.). , satellite, airship).

Furthermore, in this specification, the expression that a crop is in a "specific state" includes the following.
・Whether or not the crop is sick, and which disease it is infected with, and which disease it is likely to be ・Growth status of the crop (growth status includes whether it is harvestable or not)
・If the target area is a forest, the condition of the vegetation and the distribution of the vegetation

Crops include not only agricultural products but also plants. Therefore, the term "a crop is in a specific state" includes the condition of the plants on which the crop is grown (for example, if the crop is an apple, the condition of the leaves of the tree on which the apple is grown) is also included.

<Embodiment>
<Overview of embodiment>
In the system according to the embodiment, a first multispectral camera images the ground surface including an area where crops are cultivated, and an operator confirms the spectral image obtained by imaging, and identifies a specific wavelength band in a certain area. (This corresponds to the strength of reflected light in a specific wavelength band and reflected light in other wavelength bands from the same object) and the specific state of crops in that area. Operator associates. Save this association as a library. Next, the area to be observed is similarly imaged by the second multispectral camera to obtain a spectral image. Then, a spectral image taken of the area to be observed is analyzed according to the above-mentioned library, and an area where crops are in a specific state in this area is identified.

The first multispectral camera and the second multispectral camera are preferably mounted on a moving body, and acquire spectral images by moving the moving body over the ground surface and the area to be observed. Suitable examples of mobile objects include flying objects such as drones and satellites, but there are no special limitations as long as the object can acquire spectral images by moving the first and second multispectral cameras. do not have. Furthermore, the moving object does not need to be a flying object, and a moving object on which the first and second spectral cameras are mounted may move on the ground surface. For example, spectral images can be acquired by means such as an operator holding the first and second spectral cameras moving on the ground surface, or a car equipped with the first and second spectral cameras driving on the ground surface. It's okay. One example is a mode in which a multispectral camera realized by a smartphone is used as the spectroscopic terminal device disclosed in Japanese Patent No. 6,342,594.

The mobile body on which the first multispectral camera is mounted and the mobile body on which the second multispectral camera is mounted do not need to be the same. In the embodiments described below, the first multispectral camera is mounted on a satellite and the second multispectral camera is mounted on a drone, but the first multispectral camera is mounted on the drone and the second multispectral The camera may be mounted on a satellite. Furthermore, both the first and second multispectral cameras may be mounted on a satellite, or both the first and second multispectral cameras may be mounted on a drone. Furthermore, as described above, at least one of the first multispectral camera and the second multispectral camera may be mounted on a moving body that moves on the ground surface.

The first multispectral camera and the second multispectral camera both receive reflected light from the ground surface in multiple wavelength bands, and generate a spectral image based on this reflected light. Preferably, the first multispectral camera has a member capable of switching the imaging wavelength, such as a liquid crystal variable wavelength filter, and generates spectral images for a larger number of wavelengths than the second multispectral camera. I can do it. On the other hand, the second multispectral camera can generate spectral images for specific wavelengths. At this time, it is preferable that the specific wavelength cannot be switched while the second multispectral camera is imaging the area to be observed. That is, the specific wavelength is a predetermined wavelength. Naturally, there is no particular restriction on the magnitude relationship between the number of wavelength bands of spectral images that can be imaged by the first multispectral camera and the number of wavelength bands of spectral images that can be imaged by the second multispectral camera.

The term "library" herein refers to a highly versatile data set and/or program. Preferably, the library is implemented as training data in artificial intelligence technology. More preferably, in the system of this embodiment, a learning model is generated using teacher data, and based on this learning model, areas where crops are in a specific state are identified. Therefore, in the following explanation, the term "library" will be used mainly to refer to teaching data, but the term "library" may be used without making any particular distinction between teaching data and learning models. Note that in the following description, the learning model includes a model that has already been learned, that is, a trained model.

The details will be described later, but when creating a library, it is necessary to take into account the altitude of the sun at the time the spectral image was captured. This is because the wavelength distribution of the reflected light from the crop changes depending on the angle at which the sun hits the crop (particularly the leaves of the crop), so it is necessary to consider the influence of the solar altitude (sun angle) on the library. However, when creating a library based on the spectral images taken by the first multispectral camera, the variations of the spectral images taken by the first multispectral camera (particularly the spectra based on different solar angles) are required at the beginning of system operation. Image variations) may be lacking, making it difficult to create a library that takes the sun angle into consideration when creating a library.

Therefore, the spectral image captured by the first multispectral camera is taken at a constant solar angle (as an example, the spectral image at a mid-south altitude), and a library is created based on the spectral image at a constant solar angle. Next, when acquiring a spectral image with the second multispectral camera, the spectral image is acquired at the same solar angle as the one at which the spectral image was acquired with the first multispectral camera, thereby minimizing the influence of the solar angle. This can be done to identify areas where crops are in a particular state.

After this, it is possible to increase the variation of the spectral images acquired by the first and second multispectral cameras, i.e. to acquire spectral images at different solar angles, and to formulate the influence of the solar angle on the library. .

When creating a library, and furthermore, when identifying areas where crops are in a particular state, there is a need to create a library for a wide range of observation target areas and to perform area identification work. On the other hand, the imaging range of the spectral images captured by the first and second multispectral cameras varies depending on the angle of view of each multispectral camera and the altitude of the mobile body equipped with the multispectral camera and the ground surface. It becomes a given thing. In this case, it is preferable to connect multiple spectral images to generate a wide range spectral image. Techniques for connecting spectral images are well known, and one example is a technique for generating orthoimages from spectral images and connecting the orthoimages. The method of connecting spectral images may be applied to spectral images obtained from either the first multispectral camera or the second multispectral camera.

With reference to FIG. 1, an overview of a system to which a method according to an embodiment is applied will be described. In the system of this embodiment, an example will be mainly explained in which the first multispectral camera is mounted on a satellite and the second multispectral camera is mounted on a drone. The moving object and the method of moving the first and second multispectral cameras are not limited to the disclosure of the embodiments. Furthermore, the configurations of the first and second multispectral cameras (particularly the number of wavelength bands of spectral images that can be obtained) are not limited to the disclosure of the embodiments.

In the system 1 according to the embodiment, a spectral image for library creation is generated by a first multispectral camera mounted on a satellite 2, which is an example of a flying vehicle, and an area to be observed is detected by a drone 3, which is an example of a flying vehicle. A second multispectral camera mounted on the system captures images, and the resulting spectral images are used to identify areas where crops are in a particular state.

However, the flying object for capturing the spectral image for library creation and the flying object for capturing the spectral image for region identification may be the same, and there are no restrictions on their types. However, by capturing spectral images for library creation using the satellite 2, there is an advantage that spectral images from a wider range of the earth's surface can be obtained efficiently. On the other hand, by capturing spectral images for area identification with the drone 3, it is possible to easily obtain spectral images by flying the drone 3 to focus on the area where the condition should be identified, and furthermore, it is possible to image the same area multiple times. This has the advantage of lowering the actual cost.

In addition, although the details will be described later, Satellite 2 is equipped with a first multispectral camera that uses a liquid crystal tunable filter (LCTF), while Drone 3 is equipped with a first multispectral camera that uses a liquid crystal tunable filter (LCTF). A second multispectral camera capable of capturing a spectral image using a combination of multiple wavelength bands that can clearly identify that the light is in a specific state (preferably, a camera capable of capturing only reflected light in a specific wavelength band) It is equipped with multiple multispectral cameras. Naturally, there are no limitations to the configuration as long as the camera is a multispectral camera that can capture images based on reflected light in multiple wavelength bands. Further, the drone 3 may be equipped with a first multispectral camera using LTCF, and the satellite 2 may be equipped with a second multispectral camera capable of capturing spectral images using a combination of a plurality of wavelength bands.

The first multispectral camera on board Satellite 2 is capable of capturing reflected light in a specific wavelength band among the reflected light in a wide wavelength band from the visible to near-infrared ranges, due to the action of a liquid crystal wavelength tunable filter. . When capturing an image of reflected light from the ground surface, the system 1 of this embodiment scans a wide wavelength band and captures spectral images of the same area based on reflected light in multiple wavelength bands.

A spectral image captured by the first multispectral camera mounted on the satellite 2 is transmitted to the information processing device 4. The operator of the information processing device 4 compares multiple (preferably three or more wavelength bands) spectral images among the spectral images taken of the same region by the first multispectral camera mounted on the satellite 2, and compares each spectral image. A shading pattern indicating that the crop is in a specific state is identified from the shading pattern in the image, and this shading pattern is associated with information indicating the specific state. Therefore, the operator of the information processing device 4 grasps the growth state of the crops on the ground surface in the same area mentioned above, and based on this growth state, can determine in advance which area's crops are in what specific state. It is assumed that you understand it. The above-mentioned associations are stored in the memory of the information processing device 4 as a library. Preferably, this library is implemented as training data in artificial intelligence technology. More preferably, when creating the library, the angle of the sun with respect to the earth's surface is acquired or calculated from the time when the earth's surface was imaged by the satellite 2, and the library is created in association with this angle of the sun. However, as already explained, the dependence on the solar angle is not essential, including at the beginning of operation of the system 1.

Next, the drone 3 is flown based on a predetermined flight plan over the observation area where it is to be determined whether or not the crops are in a specific state, and the second multispectral camera mounted on the drone 3 The image is taken by As described above, since the imaging range by the second multispectral camera mounted on the drone 3 is relatively narrow, preferably, multiple drones 3 are flown at the same time and images are captured in parallel by the second multispectral camera. It is preferable to do so.

It is preferable that the flight plan be created in advance by the operator of the information processing device 4. At this time, it is preferable that the operator grasps the topography of the area to be observed in advance and creates a flight plan in which the drone 3 is to fly focused on areas where crops are expected to be in a specific state. At this time, it is also possible for the information processing device 4 to recommend areas where crops are expected to be in a specific state. Areas where the crop is expected to be in certain conditions are, for example, along rivers if the crop is sensitive to humidity, areas where fog is expected to occur for long periods of time, or even at certain altitudes. If the crop is expected to be in a particular state in width, then there is an area of this altitude width, and so on.

In the system 1 of this embodiment, a plurality of wavelength bands are selected in advance so that it is possible to clearly grasp whether or not a specific state is present, and a plurality of cameras (preferably 3 or more cameras, preferably 4 cameras), and these multiple cameras constitute a multispectral camera.

A spectral image obtained by imaging the area to be observed using such a second multispectral camera is input to the information processing device 4, and is compared with the library to identify areas where crops are in a specific state. In FIG. 1, the hatched areas are identified as areas where crops are in a specific state. At this time, the information processing device 4 connects the individual spectral images taken by the second multispectral camera mounted on the drone 3 to generate one spectral image, and uses this spectral image to identify the crop. It is preferable to identify an area in a state of . This is because Drone 3 flies at a lower altitude compared to Satellite 2, so if the area to be observed covers a wide area, it is difficult to identify the entire area with a single spectral image. be. The meaning of piecing together spectral images also applies to the spectral images obtained from the first multispectral camera mounted on Satellite 2, so as already explained, the spectral images obtained from the first multispectral camera can also be connected as appropriate. Preferably, they are joined together. Preferably, when identifying the area where the crop is in a particular state, the sun angle of the area at the time of spectral image capturing is obtained and calculated, and this sun angle is taken into account to identify the area where the crop is in the particular state. Identify.

<Basic configuration of system 1>
With reference to FIG. 2, the basic configuration of the system 1 to which the method of the embodiment is applied will be described.

The system 1 of this embodiment includes an information processing device 10, a drone 20, and a satellite 30 that are connected via a network N. The hardware configuration of the information processing device 10 is shown in FIG. 11, the hardware configuration of the drone 20 is shown in FIG. 5, and the hardware configuration of the satellite 30 is shown in FIG. These drones 20 and satellites 30 are equipped with information processing devices.

The information processing device is composed of a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later.

The network N is composed of various mobile communication systems constructed by the Internet, LAN, wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), a wireless network (eg, Wi-Fi (registered trademark)) that can be connected to the Internet through a predetermined access point, and the like. When connecting wirelessly, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), and the like. In the case of a wired connection, the network also includes a network that is directly connected using a USB (Universal Serial Bus) cable or the like.

The information processing device 10 receives a spectrum image captured by the satellite 30, and creates a library based on this spectrum image. Then, the information processing device 10 receives the spectral image captured by the drone 20, and identifies an area where crops are in a specific state in the observation target area based on the spectral image and the library.

The configuration and operation of each device will be explained below.

<Hardware configuration of drone 20>
The drone 20 of this embodiment is configured to be able to fly stationary, and includes a spectral camera control device 21 and a spectral camera 22 controlled by the spectral camera control device 21, as shown in FIGS. 3 and 4. It is equipped with a spectral camera control system 23. Each configuration will be explained in detail below.

The drone 20 is a flying object that has a function of flying stationary in the air, a so-called hovering function, and in this embodiment, it is a multicopter type drone having a plurality of rotary wings. Further, the drone 20 of this embodiment has a function of autonomously flying along a pre-specified flight path and a function of flying by remote control from a communication device or the like. Furthermore, although not shown in FIGS. 3 and 4, the drone 20 is equipped with a GPS (Global Positioning System) receiver for detecting the position (latitude and longitude) and altitude of the drone during flight, and an attitude sensor that detects the attitude of the user.

As shown in FIG. 5, the drone 20 is equipped with a spectral camera control system 23 that includes a spectral camera control device 21 and a spectral camera 22 that is a second multispectral camera controlled by the spectral camera control device 21. ing.

As shown in FIG. 5, the spectral camera control system 23 mainly includes a spectral camera 22, an attitude position detector 24 that detects information on the attitude and position of the spectral camera 22, and a spectral camera control system that controls the spectral camera 22. It has a device 21 and a battery 25 that supplies power to each of these devices.

As shown in FIG. 5, the spectrum camera 22 is mounted on the drone 20 so as to face vertically downward so that the ground surface becomes the imaging target while the drone 20 is flying stationary.

The spectrum camera 22 mounted on the drone 20 of this embodiment has wavelengths corresponding to each of a plurality of wavelength band combinations that can detect that crops are in a specific state as a result of library generation described later. It has an image sensor that can detect reflected light in a band. Specifically, the spectral camera 22 has an image sensor and a plurality of filters that can pass only light in a plurality of wavelength bands, and when the spectral camera 22 takes images of the ground surface, the plurality of filters are appropriately switched (replaced) to By letting only the reflected light in the wavelength band reach the image sensor and continuously switching the filter, an imaging result based on the reflected light in the wavelength band corresponding to each combination of a plurality of wavelength bands is obtained.

At this time, before the drone 20 images the area to be observed, it is determined in advance which specific state the crops are in, and a spectral camera consisting of a combination of filters determined based on the determination result. 22 is mounted on the drone 20. However, a plurality of cameras capable of imaging only light in a single wavelength band may be installed, and the spectral camera 22 may be constituted by the plurality of cameras. In this way, the spectral camera 22 generates a spectral image 2104 based on reflected light in a predetermined specific wavelength band, and the wavelength band can be switched while the drone 20 is imaging the area to be observed. This is a configuration in which this is not done. Note that the above description does not exclude that the spectrum camera 22 is configured to be capable of arbitrarily switching between a plurality of wavelength bands, similar to the spectrum camera 32 mounted on the satellite 30.

The image sensor captures a spectrum image using a snapshot method. In this embodiment, the image sensor 222 is a two-dimensional image sensor such as a CMOS image sensor or a CCD image sensor that can capture images within the field of view at the same timing. Further, the image sensor 222 is configured to perform imaging based on an imaging command signal transmitted from the spectral camera control device 21.

The attitude and position detector 24 is a device that detects the attitude and position of the spectrum camera 22. The attitude position detector 24 in this embodiment includes a GPS receiver 240 that detects position information and altitude information of the spectrum camera 22, and an attitude sensor 241 that detects attitude information of the spectrum camera 22.

The GPS receiver 240 acquires current position information and altitude information by capturing the positions of multiple artificial satellites. The GPS receiver 240 in this embodiment is configured to acquire longitude information and latitude information as position information, and altitude information as altitude information. Note that the position information and altitude information are not limited to those acquired by the GPS receiver 240, and may be acquired by other methods. For example, a reference point may be set and distance and altitude information from the reference point may be acquired using a distance measuring device that uses laser light or acoustic reflection.

The attitude sensor 241 detects attitude information such as the inclination angle, angular velocity, and acceleration of the spectrum camera 22. Although not shown, the attitude sensor 241 in this embodiment includes a gyro sensor that uses gyro characteristics and an acceleration sensor, and is configured to obtain tilt angle, angular velocity, and acceleration in three axial directions as attitude information.

Note that in this embodiment, position information and orientation information are acquired from the orientation position detector 24 provided as the spectrum camera control system 23, but the configuration is not limited to this. For example, the position information and attitude information may be acquired from a GPS receiver and an attitude sensor that the drone 20 already has.

<Functional configuration of spectral camera control device 21>
FIG. 6 shows a functional configuration realized by the spectrum camera control device 21 of the drone 20. The spectral camera control device 21 includes a storage section 210, a control section 211, and a communication section 212. The communication unit 212 is configured by an unillustrated communication IF of the spectral camera control device 21, the storage unit 210 is configured by an unillustrated main storage device and auxiliary storage device of the spectral camera control device 21, and the control unit 211 mainly includes It is constituted by an unillustrated processor of the spectral camera control device 21.

The communication unit 212 communicates with the information processing device 10 and the like via a network N (not shown).

<Configuration of storage unit 210 of spectral camera control device 21>
The storage unit 210 of the spectral camera control device 21 includes a flight plan DB (DataBase) 2101, a second imaging condition DB 2102, an image DB 2103, and a spectral image 2104.

Of these flight plan DB 2101, etc., everything except the spectrum image 2104 is a database. The database referred to here refers to a relational database, which is used to manage data sets called tabular tables, which are structurally defined by rows and columns, in relation to each other. In a database, a table is called a table, a table column is called a column, and a table row is called a record. In a relational database, you can set and associate relationships between tables.

Normally, each table has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit 211 can cause the processor to add, delete, or update records to a specific table stored in the storage unit 210 according to various programs.

The flight plan DB 2101 is a database storing a flight plan for flying the drone 20 along a predetermined route when flying the drone 20 in the air. The image DB 2103 is a database for managing spectral images 2104 obtained as a result of imaging by the spectral camera 22 according to the second imaging condition DB 2102. The image 2104 is a spectrum image obtained as a result of imaging by the spectrum camera 22 according to the second imaging condition DB 2102. Details of the flight plan DB 2101, second imaging condition DB 2102, and image DB 2103 will be described later.
<Configuration of control unit 211 of spectral camera control device 21>

The control unit 211 of the spectral camera control device 21 includes a reception control unit 2110, a transmission control unit 2111, a flight control unit 2112, an attitude position information acquisition unit 2113, and an imaging control unit 2114. The control unit 211 implements functional units such as the reception control unit 2110 by executing the application program 2100 stored in the storage unit 210.

The reception control unit 2110 controls the process by which the spectral camera control device 21 receives signals from an external device according to a communication protocol.

The transmission control unit 2111 controls the process by which the spectral camera control device 21 transmits a signal to an external device according to a communication protocol.

The flight control unit 2112 directs the drone toward the target point specified by the flight plan DB 2101 based on the geographical information of the target point specified by the flight plan DB 2101 and the attitude position information acquired by the attitude position information acquisition unit 2113. Control 20 flights.

The attitude position information acquisition unit 2113 acquires the position information, altitude information, and attitude information of the drone 20 detected by the attitude position detector 24, and provides the acquisition results to the flight control unit 2112 and the imaging control unit 2114.

The imaging control unit 2114 refers to the second imaging condition DB 2102, images the observation target area with the spectral camera 22 at the imaging position specified by the second imaging condition DB 2102, and stores the captured spectral image 2104 in the storage unit 210. At the same time, the conditions under which the spectrum image 2104 was captured are also stored in the image DB 2103.

FIG. 7 conceptually shows a state in which the drone 20 of this embodiment images an area to be observed and obtains a spectral image. The imaging area of the spectrum camera 22 at one time is, for example, a rectangular (square) area with one side of about 150 m, and the altitude of the drone 20 at this time is about 150 m.

<Hardware configuration of satellite 30>
The satellite 30 of this embodiment orbits at a substantially constant speed in an orbit set above the earth. As shown in FIG. 8, the satellite 30 is equipped with a spectral camera control system 33 that includes a spectral camera control device 31 and a spectral camera 32 that is a first multispectral camera controlled by the spectral camera control device 31. ing. The configurations of the spectrum camera control device 31, spectrum camera 32, and spectrum camera control system 33 are similar to those of the spectrum camera control device 21, spectrum camera 22, and spectrum camera control system 23 of the drone 20. Therefore, descriptions of substantially the same configurations will be omitted, and the description will focus on the main differences.

The spectral camera control system 33 mainly includes a spectral camera 32 equipped with a variable liquid crystal wavelength filter 34, an attitude position detector 35 that detects information on the attitude and position of the spectral camera 32, and a tunable liquid crystal wavelength filter of the spectral camera 32. It has a liquid crystal variable wavelength filter control circuit 36 that controls the filter 34, a spectral camera control device 31 that controls the spectral camera 32, and a battery 37 that supplies power to each of these devices.

The spectral camera 32 is for capturing a spectral image using a snapshot method, and mainly includes a lens group 320, a depolarizing plate 321 for converting polarized light into non-polarized light, and a liquid crystal that can arbitrarily select the transmission wavelength. It has a variable wavelength filter 34 and an image sensor 322 that captures a two-dimensional spectral image.

The lens group 320 uses light refraction to transmit the light from the imaging target to the liquid crystal variable wavelength filter 34, and focuses the transmitted light on the image sensor 322. The lens group 320 in this embodiment includes a light entrance lens 320a that condenses the light of the imaging target and makes it enter the liquid crystal wavelength tunable filter 34, and an image of the light having only the transmitted wavelength after passing through the liquid crystal wavelength tunable filter 34. A condensing lens 320b condenses light onto a sensor 322. Note that the type and number of lenses are not particularly limited, and may be appropriately selected depending on the performance of the spectral camera 32, etc.

The depolarizing plate 321 is for depolarizing light and making it non-polarized light. In this embodiment, the depolarizing plate 321 is provided on the light incident side of the liquid crystal variable wavelength filter 34, and depolarizes the light before passing through the liquid crystal variable wavelength filter 34, thereby reducing the polarization characteristics. There is.

The liquid crystal variable wavelength filter 34 is an optical filter that can arbitrarily select a transmission wavelength from within a predetermined wavelength range. Although not shown, the liquid crystal variable wavelength filter 34 has a structure in which a plurality of plate-shaped liquid crystal elements and plate-shaped polarizing elements are stacked alternately. The alignment state of each liquid crystal element is independently controlled by the applied voltage supplied from the liquid crystal variable wavelength filter control circuit 36. Therefore, the liquid crystal variable wavelength filter 34 is configured to transmit light of any wavelength depending on the alignment state of the liquid crystal element and the combination of the polarizing element.

In this embodiment, the width of the transmission wavelength of the liquid crystal variable wavelength filter 34 is about 20 nm or less, the transmission center wavelength can be set in steps of 1 nm, and the wavelength switching time is about 10 ms to several 100 ms.

The image sensor 322 captures a spectrum image using a snapshot method. In this embodiment, the image sensor 322 is a two-dimensional image sensor such as a CMOS image sensor or a CCD image sensor that can capture images within the field of view at the same timing. Further, the image sensor 322 is configured to perform imaging based on an imaging command signal transmitted from the spectral camera control device 31.

The liquid crystal variable wavelength filter control circuit 36 controls the liquid crystal variable wavelength filter 34. In this embodiment, upon receiving the wavelength specifying signal transmitted from the spectral camera control device 31, the liquid crystal variable wavelength filter control circuit 36 supplies an applied voltage according to the wavelength specifying signal to the liquid crystal element of the liquid crystal variable wavelength filter 34. It is supposed to be done. Further, the wavelength specifying signal includes information on the transmission wavelength to be transmitted by the liquid crystal wavelength tunable filter 34, and the liquid crystal wavelength tunable filter control circuit 36 determines which liquid crystal element to apply voltage to based on the transmission wavelength information. It is determined whether the applied voltage is supplied to the liquid crystal element, and the applied voltage is supplied to the specified liquid crystal element.

Note that the liquid crystal variable wavelength filter control circuit 36 in this embodiment is configured independently from other components such as the spectral camera control device 31, but is not limited to this. For example, the spectral camera control device 31 or the spectrum camera 32 may be provided.

<Functional configuration of spectral camera control device 31>
FIG. 9 shows a functional configuration realized by the spectrum camera control device 31 of the satellite 30. The spectral camera control device 31 includes a storage section 310, a control section 311, and a communication section 312. The communication unit 312 is configured by an unillustrated communication IF of the spectral camera control device 31, the storage unit 310 is configured by an unillustrated main storage device and auxiliary storage device of the spectral camera control device 31, and the control unit 311 mainly includes It is constituted by an unillustrated processor of the spectral camera control device 31.

The communication unit 312 communicates with the information processing device 10 and the like via a network N (not shown).

<Configuration of storage unit 310 of spectral camera control device 31>
The storage unit 310 of the spectral camera control device 31 includes a first imaging condition DB (DataBase) 3101, an image DB 3102, and a spectral image 3103.

Of these first imaging condition DB 3101, etc., all of them except for the spectral image 3103 are databases. The database referred to here refers to a relational database, which is used to manage data sets called tabular tables, which are structurally defined by rows and columns, in relation to each other. In a database, a table is called a table, a table column is called a column, and a table row is called a record. In a relational database, you can set and associate relationships between tables.

Normally, each table has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit 311 can cause the processor to add, delete, or update records to a specific table stored in the storage unit 310 according to various programs.

The first imaging condition DB 3101 is a database regarding conditions for imaging the ground surface with the spectral camera 32 mounted on the satellite 30, and the image DB 3102 is a spectrum obtained as a result of imaging by the spectral camera 32 according to the first imaging condition DB 3101. It is a database for managing images, and the spectrum image 3103 is a spectrum image obtained as a result of imaging by the spectrum camera 32 according to the first imaging condition DB 3101. Details of the first imaging condition DB 3101 and image DB 3102 will be described later.
<Configuration of control unit 311 of spectral camera control device 31>

The control unit 311 of the spectral camera control device 31 includes a reception control unit 3110, a transmission control unit 3111, an imaging determination unit 3112, an attitude position information acquisition unit 3113, an imaging control unit 3114, and a wavelength setting unit 3115. The control unit 311 implements functional units such as the reception control unit 3110 by executing the application program 3100 stored in the storage unit 310.

The reception control unit 3110 controls the process by which the spectral camera control device 31 receives a signal from an external device according to a communication protocol.

The transmission control unit 3111 controls the process by which the spectral camera control device 31 transmits a signal to an external device according to a communication protocol.

The imaging determination unit 3112 determines whether or not to image the ground surface with the spectral camera 32 mounted on the satellite 30 based on the imaging start time and imaging end time stored in the first imaging condition DB 3101. The results are sent to the imaging control unit 3114.

The attitude and position information acquisition unit 3113 acquires the position information, altitude information, and attitude information of the satellite 30 detected by the attitude and position detector 35, and provides the acquisition results to the imaging control unit 3114.

The imaging control unit 3114 receives the determination result from the imaging determination unit 3112, and if the determination result indicates that imaging is to be performed, the imaging control unit 3114 images the ground surface with the spectral camera 32, and stores the captured spectral image 3103 in the storage unit 310. At the same time, the conditions under which the spectrum image 3103 was captured are stored in the image DB 3102.

FIG. 10 conceptually shows a state in which the satellite 30 of this embodiment images an area to be observed and obtains a spectral image. The imaging area of the spectrum camera 32 at one time is, for example, a rectangular (square) area with sides of about 2 to 10 km, and the altitude of the satellite 30 at this time is about 500 km.

<Hardware configuration of information processing device 10>
FIG. 11 is a block diagram showing the basic hardware configuration of the information processing device 10. As shown in FIG. The information processing device 10 includes at least a processor 101, a main storage device 102, an auxiliary storage device 103, a communication IF (Interface) 104, an input IF 105, and an output IF 106. These are electrically connected to each other by a communication bus 107. Further, an input device 110 and an output device 111 are connected to the information processing device 10 via an input IF 105 and an output IF 106, respectively.

The processor 101 is hardware for executing a set of instructions written in a program. The processor 101 includes an arithmetic unit, registers, peripheral circuits, and the like.

The main storage device 102 is for temporarily storing programs, data processed by the programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

The auxiliary storage device 103 is a storage device for storing data and programs. Examples include flash memory, SSD (Solid State Drive), HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, and the like.

The communication IF 104 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.

The input IF 105 functions as an interface with the input device 110 for receiving input operations from the operator of the information processing device 10. The output IF 106 functions as an interface with the output device 111 for presenting information to the operator. The input device 110 is an input device (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving input operations from an operator. The output device 111 is an output device (display, speaker, etc.) for presenting information to the operator.

Note that the information processing device 10 can be virtually realized by distributing all or part of each hardware configuration to a plurality of computers and interconnecting them via a network. In this way, the information processing device 10 is a concept that includes not only a computer housed in a single housing or case, but also a virtualized computer system.

<Functional configuration of information processing device 10>
FIG. 12 shows a functional configuration realized by the hardware configuration of the information processing device 10. The information processing device 10 includes a storage section 120, a control section 130, and a communication section 140. The communication unit 140 is configured by the communication IF 104, the storage unit 120 is configured by the main storage device 102 and the auxiliary storage device 103 of the information processing device 10, and the control unit 130 is mainly configured by the processor 101 of the information processing device 10.

The communication unit 140 communicates with the drone 20 and the like via the network N.

<Configuration of storage unit 120 of information processing device 10>
The storage unit 120 of the information processing device 10 includes a flight plan DB (DataBase) 122, an image DB 123, a wide-area image DB 124, teacher data 125, a learning model 126, a spectral image 127, and a wide-area spectral image 128.

Of these flight plan DB 122, etc., the flight plan DB 122, image DB 123, and wide area image DB 124 are databases. The database referred to here refers to a relational database, which is used to manage data sets called tabular tables, which are structurally defined by rows and columns, in relation to each other. In a database, a table is called a table, a table column is called a column, and a table row is called a record. In a relational database, you can set and associate relationships between tables.

Normally, each table has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit 130 can cause the processor 101 to add, delete, or update records to a specific table stored in the storage unit 120 according to various programs.

The flight plan DB 122 is similar to the flight plan DB 2101 stored in the storage unit 210 of the drone 20. The image DB 123 is similar to the image DB 2103 stored in the storage unit 210 of the drone 20 and the image DB 3102 stored in the storage unit 310 of the satellite 30. In the information processing apparatus 10 of this embodiment, an image DB 123 that is a combination of the image DB 2103 and the image DB 3102 is stored in the storage unit 120. The wide-area image DB 124 is a database for managing the wide-area spectral image 128 generated by the image synthesis unit 136 of the control unit 130 based on the spectral image 127.
<Configuration of control unit 130 of information processing device 10>

The control unit 130 of the information processing device 10 includes a reception control unit 131, a transmission control unit 132, a learning model generation unit 133, a flight plan creation unit 134, an aircraft control unit 135, an image synthesis unit 136, an area identification unit 137, and a type identification unit. 138. The control unit 130 realizes functional units such as the reception control unit 131 by executing the application program 121 stored in the storage unit 120.

The reception control unit 131 controls a process in which the information processing device 10 receives a signal from an external device according to a communication protocol.

The transmission control unit 132 controls a process in which the information processing device 10 transmits a signal to an external device according to a communication protocol.

The learning model generation unit 133 uses the spectral image 127 taken by the spectral camera 32 mounted on the satellite 30 and the image DB 123 sent from the satellite 30, as well as the spectral camera 22 mounted on the drone 20 and the spectral image DB 123 sent from the drone 20. Teacher data 125 is generated based on the image DB 123, and a learning model 126 is generated based on this teacher data 125.

The details of the operation of the learning model generation unit 133 when using the spectrum image 127 taken by the spectrum camera 32 mounted on the satellite 30 will be described. Note that the operation is similar when using the spectrum image 127 taken by the spectrum camera 22 mounted on the drone 20.

The spectral camera 32 mounted on the satellite 30 images the same area of the earth's surface for each wavelength band obtained by subdividing this wavelength band over a wide range of wavelength bands, and images a large number of spectral images 3103 (spectral images) of the same area with different wavelength bands. Image 127) has been acquired. The learning model generation unit 133 presents the operator of the information processing device 10 with a plurality of spectral images 127 of the same region with different wavelength bands, and generates a wavelength band that can clearly distinguish areas where crops are in a specific state in this region. Spectral image 127 is specified by the operator. Next, the learning model generation unit 133 causes the operator to specify a specific state of the crop that can be specified by the specified wavelength band with respect to the spectral image 127 of the specified wavelength band. associated with specific conditions of the crop. Then, the learning model generation unit 133 causes the storage unit 120 to store the specified wavelength band, the spectral image 127 of the specified wavelength band, the specified specific state of the crop, and the relationship thereof as the teacher data 125. .

For example, as shown in FIG. 23, an original image taken of the same area on the ground surface (in this embodiment, multiple spectral images 127 can be acquired for the same area, but to simplify the explanation, only one image is used). It is possible to acquire spectrum images 127 of a plurality of wavelength bands (in FIG. 23, there are three types of wavelengths α, β, and γ, but there is no limit to the number of wavelength bands). Among these spectral images 127, a different density distribution may occur for a certain wavelength (wavelength β in FIG. 23) than for other wavelengths α and γ. Note that since this is a spectral image 127 for a specific wavelength band, it is assumed that when displayed on a display or the like, it will be displayed in a pattern of black and white shading (that is, grayscale display). The operator of the information processing device 10 knows in advance what kind of specific state the crops are in in the same ground surface area (for example, through a field survey), and the operator of the information processing device 10 knows in advance what kind of specific state the crops are in in the same ground surface area (for example, through a field survey), and when the wavelength β is different from other wavelengths α and γ, The operator looks at the unique gray scale distribution that cannot be seen (indicated by hatching in FIG. 23) and inputs the association between this gray scale distribution and a specific state of the crop.

At this time, the learning model generation unit 133 refers to the image DB 123 and calculates the angle between the earth's surface and the sun when the spectral image of the specified wavelength band is captured. This is because the spectrum of light reflected from crops differs depending on the angle of the sun. The learning model generation unit 133 similarly stores the angle of the sun in the storage unit 120 as the teacher data 125. Alternatively, the learning model generation unit 133 calculates a correction formula (or correction value) for correcting the spectral image to a spectral image of a spectrum captured at the sun angle at a predetermined time, based on the calculated sun angle, Teacher data 125 is generated using the spectral image corrected using this correction formula, etc., and is stored in the storage unit 120. At this time, the correction formula itself may be stored in the learning model 126 described later, or the correction formula may be stored separately in the storage unit 120. However, correction based on the solar angle is not essential.

Next, the learning model generation unit 133 generates a learning model 126 in the artificial intelligence technology based on the teacher data 125 generated by the above-described procedure. As for the generation method of the learning model 126 and the type of the learning model 126, known methods can be suitably applied, so further explanation will be omitted here.

Then, the learning model generation unit 133 repeatedly performs the above-described teacher data 125 generation procedure to update the teacher data 125, and re-learns the learning model 126 after updating the teacher data 125 or at every predetermined time interval. This improves the accuracy with which the type identification unit 138 (described later) identifies crops as being in a particular state.

Regarding the updating of the teacher data 125 and the relearning of the learning model 126 by the learning model generation unit 133 described above, in particular, at the beginning of operation of the system 1, a spectral image 127 at a limited solar angle is acquired, and this spectral image 127 is This is a preferable operation if the teacher data 125 and learning model 126 are generated based on the following. That is, initially, the teacher data 125 and the learning model 126 are generated based on the spectral image 127 at a limited solar angle, and then the spectral image obtained from the spectral camera 32 mounted on the satellite 30 and/or the spectral image mounted on the drone 20 is generated. The spectral images obtained from the spectral camera 22 are acquired at various sun angles, training data 125 including the influence of the sun angle is generated, and the learning model 126 is retrained based on this training data 125. Good too. As a result, it is possible to bring forward the start of operation of the system 1, and it is also possible to progressively improve the accuracy of the identification results by the type identification unit 138. In addition, the specific state of the actual crop is compared with the identification result by the type identification unit 138, which will be described later, and the actual specific state of the crop is fed back to create/modify the training data 125, and the learning model 126 is re-trained. You may do so. This also allows the accuracy of the identification results by the type identification unit 138 to be progressively improved.

At this time, the learning model generation unit 133 acquires topographical data of the ground surface imaged by the spectral camera 32 of the satellite 30 and/or the spectral camera 22 of the drone 20, and based on this topographical data, the learning model generation unit 133 determines whether the crops are in a specific state. It is preferable to identify an area that is estimated to be located in the area, and to extract a spectral image 127 obtained by imaging this area with the spectral cameras 22 and 32 and present it to the operator. Whether a crop is in a particular state often depends on the terrain. As an example, it is known that crops are more susceptible to disease if the humidity in the area where they are grown is high. Therefore, the learning model generation unit 133 acquires topographical data of the ground surface, estimates the environment (sunlight, humidity, wind direction, etc.) of the place where crops are grown based on this topographical data, and determines whether this environment is a specific one. A spectrum image 127 obtained by imaging the location is presented to the operator. The terrain data may be stored in advance in the storage unit 120 of the information processing device 10, or may be obtained from an external service.

Note that the specified wavelength band, the spectral image 127 of the specified wavelength band, the specified state of the crop, and the spectral image 127 can be captured without using the learning model generation unit 133, the teacher data 125, and the learning model 126. It is also possible to identify whether a crop is in a particular state based solely on the angle of the sun at the time of the harvest and their relationship.

The flight plan creation unit 134 creates a flight plan for flying the drone 20 over the observation target area and capturing an image of the observation target area using the spectrum camera 22 mounted on the drone 20, and stores the flight plan in the storage unit 120 as the flight plan DB 122. Store in. The flight plan creation unit 134 then transmits the created flight plan DB 122 to the drone 20. In addition, when controlling the flight of the drone 20 (driving the drone 20) by the flying object control part 135 mentioned later, it is not necessary to provide the flight plan creation part 134 and flight plan DB122.

At this time, the flight plan creation unit 134 creates a flight plan in which multiple drones 20 are simultaneously flown over the observation target area, and the spectrum cameras 22 mounted on these multiple drones 20 are to take images of the observation target area in parallel. It is preferable to create one.

It is also preferable that the flight plan creation unit 134 acquires topographical data of the area to be observed and creates a flight plan based on this topographical data. As mentioned above, whether a crop is in a particular state often depends on the terrain. As an example, it is known that crops are more susceptible to disease if the humidity in the area where they are grown is high. Therefore, the flight plan creation unit 134 acquires topographical data of the area to be observed, estimates the environment (sunlight, humidity, wind direction, etc.) of the place where crops are grown based on this topographical data, and determines whether this environment is specific. A drone 20 is flown focusing on a certain place, and a spectral image is acquired by a spectral camera 22. The terrain data may be stored in advance in the storage unit 120 of the information processing device 10, or may be obtained from an external service.

The flight plan creation unit 134 may create a flight plan by having the operator of the information processing device 10 individually designate the flight target position of the drone 20, or the operator may designate only the area to be observed and create the flight plan. The unit 134 may generate the flight target position based on the flight speed of the drone 20 and the spectrum image acquisition speed by the spectrum camera 22.

The flying object control unit 135 controls the flight of the drone 20 based on the flight plan DB 122. However, if the drone 20 can fly autonomously based on the flight plan DB 2101 stored in its own storage unit 210, the flying object control unit 135 may not be provided. Alternatively, if the information processing device 10 controls the drone 20, the flying object control unit 135 controls the drone 20 based on a control signal input by the operator through a remote controller, which is an example of the input device 110 provided in the information processing device 10. to operate.

The image synthesis unit 136 connects the spectral images 2104 and 127 captured by the spectral camera 22 of the drone 20 to generate a wide-range spectral image 128 that preferably captures the entire area to be observed. This is because the imaging range of the spectral camera 22 of the drone 20 is often narrower than the area to be observed where it is necessary to identify whether or not crops are in a specific state. This is because it is preferable to generate a broad-spectrum image 128 of a wide range in order to perform the above-mentioned processing. The image synthesis unit 136 then stores the generated wide spectrum image 128 in the storage unit 120 and updates the wide range image DB 124. Furthermore, the image synthesis unit 136 connects the spectral images 3103 and 127 captured by the spectral camera 32 of the satellite 30 to generate a wide spectrum image 128 that preferably captures a wide range of the ground surface.

At this time, it is preferable that the image synthesis unit 136 generates the wide spectrum image 128 based on a known method of generating orthoimages. An ortho image eliminates the positional shift of the image on the photo and converts the aerial photo into an image that is displayed in the correct size and position without any tilt, as if seen from directly above, just like a map (hereinafter referred to as ``orthogonal conversion''). ). To orthographically convert an aerial photograph, it is necessary to match the position on the aerial photograph with the horizontal position on the ground. This orthographic transformation is performed using a digital elevation model (elevation data) that represents the three-dimensional shape of the earth's surface. Since the method of generating orthoimages is known, further explanation will be omitted here.

FIG. 22 is a diagram showing an example of image synthesis of the spectrum images 2104 and 127 by the image synthesis unit 136. Spectral images 127 obtained by imaging an area 2200 (square in the figure, but not limited to this) by one drone 20 are arranged horizontally in the figure with a predetermined overlapping area, and these spectral images are 127 to generate a wide spectrum image 128. This operation is also performed on the spectral image 127 obtained by imaging the area 2201 with another drone 20 to obtain a wide-area spectral image 128 that includes the area to be observed.

The area specifying unit 137 refers to the wide spectrum image 128 created by the image synthesizing unit 136 and performs information processing to specify the area for the type specifying unit 138 to specify whether or not the crop is in a specific state. received from the operator of the device 10.

The type identification unit 138 determines whether or not the crop is in a specific state in the area identified by the area identification unit 137, and if so, in what state, based on the wide spectrum image 128 and the learning model 126. Identify. At this time, the type specifying unit 138 receives an input of a specific state of the crop to be specified based on an instruction input from the operator of the information processing device 10, and specifies whether the crop is specific in the area specified by the area specifying unit 137. It may also be possible to specify whether or not the person is in a specific state. At this time, the type identifying unit 138 calculates the angle between the earth's surface and the sun when the spectral image 127 that is the source of the wide-spectrum image 128 is captured, Using the angle correction formula, the spectrum is corrected so that it becomes a wide spectrum image 128 captured at a predetermined time.

Next, the type identification unit 138 displays the identification result on a display, which is an example of the output device 111. There are no particular restrictions on the display mode by the type identification unit 138, but as an example, it refers to a map database to display map data of an area to be observed, and also includes an area where crops are in a specific state in this map data. An example of such a mode is superimposed display.

In addition, the type specifying unit 138 may receive input specifying a time axis from the operator of the information processing device 10 and display the spread of crops in a region in a specific state on the time axis. Furthermore, the type identifying unit 138 determines that the crop is in a specific state, and the cost required to restore this specific state (for example, cutting down crops in a specific state, spraying pesticides, etc.) The estimated damage amount may be calculated and displayed on the display.
<Data structure>
FIG. 13 is a diagram showing the data structure of the first imaging condition DB 3101 stored in the storage unit 310 of the satellite 30.

The first imaging condition DB 3101 is a table having columns of imaging start time, imaging end time, and wavelength condition using an imaging condition ID as a key for specifying imaging conditions by the spectrum camera 32 of the satellite 30.

The "imaging condition ID" is information for specifying the imaging condition by the spectrum camera 32 of the satellite 30. The “imaging start time” is information regarding the time at which the spectrum camera 32 of the satellite 30 starts imaging. The “imaging end time” is information regarding the time when imaging by the spectrum camera 32 of the satellite 30 ends. “Wavelength conditions” is information regarding wavelength conditions when imaging is performed by the spectrum camera 32 of the satellite 30. The spectral camera 32 mounted on the satellite 30 is capable of capturing images in multiple wavelength bands (and can therefore generate spectral images in multiple wavelength bands), and in the example shown in FIG. Information about the range of the band and the wavelength width for changing the wavelength band is stored.

Each column in the first imaging condition DB 3101 is generated by an information processing device including the information processing device 10, and is stored in the storage unit 310 by being sent to the satellite 30.

FIG. 14 is a diagram showing the data structure of the image DBs 2103, 3102, and 123 stored in the storage unit 210 of the drone 20, the storage unit 310 of the satellite 30, and the storage unit 120 of the information processing device 10, respectively.

Image DBs 2103, 3102, and 123 use image IDs as keys to identify spectrum images captured by spectrum cameras 22 and 32 of drones 20 and satellites 30, and store information such as image file names, imaging times, location information, altitude information, and city administration. 2 is a table with columns of information and wavelength.

"Image ID" is information for identifying the spectrum image captured by the spectrum cameras 22 and 32 of the drone 20 and the satellite 30. “Image file name” is the file name of the spectral images 2104, 3103, and 127 identified by the image ID and stored in the storage unit 210 of the drone 20, the storage unit 310 of the satellite 30, and the storage unit 120 of the information processing device 10. This is information indicating. "Imaging time" is information indicating the time when the spectrum images 2104, 3103, and 127 specified by the image ID were captured. “Position information” is the position information of the drone 20 and the satellite 30 when the spectrum images 2104, 3103, and 127 specified by the image ID were captured. "Altitude information" is altitude information of the drone 20 and the satellite 30 when the spectrum images 2104, 3103, and 127 specified by the image ID were captured. "Attitude information" is attitude information of the drone 20 and the satellite 30 when the spectrum images 2104, 3103, and 127 specified by the image ID were captured. "Wavelength" is information indicating the wavelength band set in the spectrum cameras 22 and 32 of the drone 20 and the satellite 30 when the spectrum images 2104, 3103, and 127 specified by the image ID were captured.

Each column in the image DB 2103, 3102, 123 is generated by the spectral camera control devices 21, 31 of the drone 20 and satellite 30 when the spectral images 2104, 3103 are generated by the spectral cameras 22, 32 of the drone 20 and satellite 30. do.

FIG. 15 is a diagram showing the data structure of the second imaging condition DB 2102 stored in the storage unit 210 of the drone 20.

The second imaging condition DB 2102 is a table that has columns of imaging position, imaging altitude, and imaging posture, using an imaging condition ID as a key for specifying imaging conditions by the spectrum camera 22 of the drone 20.

The "imaging condition ID" is information for specifying the imaging condition by the spectrum camera 22 of the drone 20. “Imaging position” is position information of the drone 20. “Imaging altitude” is altitude information of the drone 20. “Imaging posture” is posture information of the drone 20.

Each column in the second imaging condition DB 2102 is generated by an information processing device including the information processing device 10, and is stored in the storage unit 210 by being sent to the drone 20.

FIG. 16 is a diagram showing the data structure of the flight plan DBs 122 and 2101 stored in the storage unit 120 of the information processing device 10 and the storage unit 210 of the drone 20.

The flight plan DB 122, 2101 is a table having columns of elapsed time, flight position, and flight attitude using the flight plan ID for specifying the flight plan of the drone 20 as a key.

The "flight plan ID" is information for specifying the flight plan of the drone 20, more specifically, the goal of the drone 20. The "elapsed time" is information indicating the elapsed time from the start of the flight of the drone 20 when the drone 20 should reach the destination specified by the flight plan ID. "Flight position" is target position information that the drone 20 should reach in the target of the drone 20 specified by the flight plan ID. "Flight altitude" is target altitude information that the drone 20 should reach in the target destination specified by the flight plan ID. "Flight attitude" is target attitude information that the drone 20 should reach at the goal specified by the flight plan ID.

Each column in the flight plan DBs 122 and 2101 is generated by the flight plan creation unit 134 of the control unit 130 of the information processing device 10, stored in the storage unit 120, and sent to the drone 20. The spectral camera control device 21 of the drone 20 stores the sent flight plan DB 122, 2101 in the storage unit.

FIG. 17 is a diagram showing the data structure of the wide area image DB 124 stored in the storage unit 120 of the information processing device 10.

The wide-area image DB 124 uses the wide-area image ID for specifying the wide-area spectrum image 128 stored in the storage unit 120 of the information processing device 10 as a key, and stores the wide-area image file name, imaging time, regional information, solar angle, and position information. , altitude information, attitude information, wavelength, and image ID columns.

The “wide area image ID” is information for identifying the wide area spectrum image 128 stored in the storage unit 120 of the information processing device 10. The “wide area image file name” is information indicating the file name of the wide area spectrum image 128 that is specified by the wide area image ID and stored in the storage unit 120 of the information processing device 10.

"Imaging time" is information indicating the time when the wide spectrum image 128 specified by the wide area image ID was captured. Here, the wide spectrum image 128 is a composite of a plurality of spectrum images 127, and the imaging times of the individual spectrum images 127 are strictly different. Therefore, when synthesizing a plurality of spectral images 127 to generate a wide spectrum image 128, the image synthesis unit 136 determines the imaging time of one of the spectral images 127 among the plurality of spectral images 127 that are the source of synthesis. , is representatively used as the imaging time of the wide spectrum image 128. This work is also performed for "sun angle", "position information", "altitude information", and "attitude information".

"Regional information" is information indicating the area where the wide spectrum image 128, which is specified by the wide area image ID and stored in the storage unit 120 of the information processing device 10, was captured. "Sun angle" is information indicating the angle of the sun when the wide spectrum image 128 specified by the wide area image ID was captured. At the beginning of operation of the system 1, the value of the "solar angle" field may be blank. “Position information” is the position information of the drone 20 and the satellite 30 when the spectrum images 2104, 3103, and 127 specified by the image ID were captured. "Altitude information" is altitude information of the drone 20 when the wide spectrum image 128 specified by the wide area image ID was captured. "Attitude information" is attitude information of the drone 20 when the wide spectrum image 128 specified by the wide area image ID is captured. "Wavelength" is information indicating the wavelength band in which the wide spectrum image 128 specified by the wide range image ID was captured. “Image ID” is information for specifying a plurality of spectral images 127 from which the wide-range spectrum image 128 specified by the wide-range image ID is synthesized, and is common to the image ID of the image DB 2103.

Each column in the wide-area image DB 124 is generated by the image synthesis unit 136 of the control unit 130 of the information processing device 10 and stored in the storage unit 120.

<Operation of system 1>
The processing of the system 1 will be described below with reference to the flowcharts of FIGS. 18 to 21.

FIG. 18 is a flowchart showing spectral image capturing processing by the spectral camera control device 31 of the satellite 30.

First, the spectral camera control device 31 refers to the first imaging condition DB 3101 and waits for the imaging start time described in the first imaging condition DB 3101 to arrive (S1800). When the imaging start time arrives (YES in S1800), the spectral camera control device 31 refers to the first imaging condition DB 3101, sets the wavelength band for imaging by the spectral camera 32 (S1801), and uses the set wavelength band. The ground surface is imaged by the spectral camera 32, and a spectral image 3103 is acquired and stored in the storage unit 310, and the conditions at the time of imaging are stored in the image DB 3102 of the storage unit 310 (S1802).

Next, the spectral camera control device 31 refers to the first imaging condition DB 3101 and determines whether the imaging operation in S1802 has been completed in all wavelength bands (S1803). If it is determined that the imaging operation in all wavelength bands has been completed (YES in S1803), the process advances to S1804, and if it is determined that there is a wavelength band in which the imaging operation has not been performed yet (NO in S1803), the process returns to S1801 and the next step is to proceed to S1801. Set the wavelength band and perform the following processing.

In S1804, the spectral camera control device 31 refers to the first imaging condition DB 3101 and determines whether the imaging end time has arrived. If it is determined that the imaging end time has come (YES in S1804), the process advances to S1805, and if it is determined that the imaging end time has not yet arrived (NO in S1804), the process returns to S1800 and waits for the next imaging start time.

In S1805, the spectral camera control device 31 determines whether the imaging operation has been completed for all imaging conditions stored in the first imaging condition DB 3101. If it is determined that the imaging operation has ended for all imaging conditions (YES in S1805), the spectrum image 3103 and image DB 3102 stored in the storage unit 310 are transmitted to the information processing apparatus 10 (S1806), and the imaging operation is still in progress. If it is determined that there is an imaging condition that has not been completed (NO in S1805), the process returns to S1800 and waits for the next imaging start time.

Next, FIG. 19 is a flowchart showing learning model generation processing by the learning model generation unit 133 of the control unit 130 of the information processing device 10.

First, the learning model generation unit 133 refers to the image DB 123 in the storage unit 120 and specifies/selects the location (region, region) where the spectral image 127 from which the learning model is generated is captured (S1900). Next, the learning model generation unit 133 determines the wavelength band in which the spectral image 127 was captured in order to identify the spectral image 127 from which the learning model is generated, among the spectral images 127 stored in the storage unit 120. Specify and select (S1901).

Next, the learning model generation unit 133 extracts from the storage unit 120 the spectral image 127 that matches the conditions specified and selected in S1900 and S1901, and displays the extracted spectral image 127 on the display, which is an example of the output device 111. (S1902).

Further, the learning model generation unit 133 refers to the image DB 123, obtains the time when the spectral image 127 extracted in S1902 was captured from the item "imaging time", and determines from this imaging time that the spectral image 127 was captured. Calculate the angle between the earth's surface and the sun (sun angle). Next, the learning model generation unit 133 calculates a correction formula (a correction value may also be used) for the spectrum when this spectrum image 127 is captured at a specific time (for example, 12:00 in the daytime), and uses this correction formula to The spectrum of the spectrum image 127 is corrected (S1903). This correction formula and the like are stored as a learning model 126.

Thereafter, the learning model generation unit 133 inputs a selection input of a region to be learned from the spectral image 127 displayed on the display using a pointing device such as a mouse, which is an example of the input device 110, to the information processing device. 10 operators (S1904). Then, the learning model generation unit 133 inputs a specification input as to which specific state the crop is in for the area specified in S1903 to the operator of the information processing device 10 using a pointing device such as a mouse, which is an example of the input device 110. (S1905).

Then, the learning model generation unit 133 receives input specifications from S1900 to S1905 from the operator of the information processing device 10, and waits for receiving an instruction indicating that various inputs for generating the learning model have been completed (S1906). When the instruction is received (YES in S1906), the teacher data 125 is generated based on the input specifications from S1900 to S1905 and the image data of the target spectrum image 127 (S1907). The learning model generation unit 133 then generates the learning model 126 based on the teacher data 125 generated in S1907 (S1908). The learning model generation unit 133 stores the teacher data 125 and the learning model 126 generated in S1907 and S1908 in the storage unit 120.

Next, FIG. 20 is a flowchart showing spectral image capturing processing by the spectral camera control device 21 of the drone 20.

First, the spectrum camera control device 21 flies the drone 20 so as to sequentially reach the target position according to the flight plan DB 2101 stored in the storage unit 210 (S2000).

Next, the spectral camera control device 21 refers to the second imaging condition DB 2102 stored in the storage unit 210 and waits for the drone 20 to reach the imaging position described in the second imaging condition DB 2102 (S2001). . Then, when the drone 20 reaches the imaging position (YES in S2001), the spectral camera control device 21 causes the spectral camera 22 to image the area to be observed, and stores the obtained spectral image 2104 in the storage unit 210. At the same time, the conditions at the time of imaging are stored in the image DB 2103 of the storage unit 210 (S2002).

Then, the spectral camera control device 21 determines whether the imaging operation has been completed for all imaging positions stored in the second imaging condition DB 2102 (S2003). Then, if it is determined that the imaging operation has ended for all imaging conditions (YES in S2003), the spectrum image 2104 and image DB 2103 stored in the storage unit 210 are transmitted to the information processing apparatus 10 (S2004), and the imaging operation is still in progress. If it is determined that there is an imaging position that has not been completed (NO in S2003), the process returns to S2001 and waits for the next imaging position to be reached.

After this, the spectrum camera control device 21 ends the flight of the drone 20 (S2005).

FIG. 21 is a flowchart illustrating processing performed by the area specifying unit 137 and type specifying unit 138 of the control unit 130 of the information processing device 10 to specify an area where crops are in a specific state.

First, the area specifying unit 137 refers to the image DB 123 in the storage unit 120 and specifies/selects the position (region, region) where the spectral image 127 is captured, which is the source for specifying the area where the crops are in a specific state. (S2100).

Next, the type identification unit 138 extracts from the storage unit 120 the spectral image 127 that matches the conditions specified and selected in S2100, and displays the extracted spectral image 127 on the display, which is an example of the output device 111 (S2101). .

Next, the image synthesis unit 136 generates a wide spectrum image 128 based on the spectrum image 127 extracted in S2101 (S2102), and displays the generated wide spectrum image 128 on a display or the like (S2103). Note that the operation of generating the wide spectrum image 128 by the image synthesis unit 136 may be performed prior to the processing shown in FIG. 21.

Next, the type specifying unit 138 refers to the image DB 123 and obtains the time when the spectral image 127, which is the source of the wide spectrum image 128, was captured from the item "imaging time", and from this imaging time, the spectral image 127 is Calculates the angle between the earth's surface and the sun (sun angle) when the image is taken. Then, based on the learning model 126 or the correction formula stored in the storage unit 120, the type identification unit 138 determines the spectrum of the spectrum image 127 when it is taken at a specific time (for example, 12:00 in the daytime). The spectrum of the spectrum image 127 is corrected so that the spectrum of the wide spectrum image 128 as a whole is corrected (S2104).

Then, the type identifying unit 138 uses the learning model 126 stored in the storage unit 120 to determine whether or not crops are in a specific state in the observation target area where the wide spectrum image 128 is captured. Then, estimation processing (inference operation) is performed to determine which area it is (S2105), and the inference result is displayed on a display or the like (S2106).

<Screen example>
FIG. 24 is a diagram showing an example of a screen displayed on the display, which is an example of the output device 111 of the information processing device 10, in the learning model generation process of the learning model generation unit 133 shown in the flowchart of FIG.

A button 2401 for specifying a spectral image 127 as a source for generating a learning model is displayed on the screen 2400, and the operator of the information processing device 10 presses this button using a pointing device or the like that is an example of the input device 110. Spectral images 2402 to 2404 are specified. Specified spectral images 2402 to 2404 are displayed on screen 2400 for each wavelength band.

The operator of the information processing device 10 views these spectral images 2402 to 2404 and determines whether the crops are growing based on information about the specific state of the crops in this area and the area where the crops are in the specific state, which has been acquired in advance. Regarding the spectral images 2402 to 2404 (in the example shown in FIG. 24, the spectral image 2403 at wavelength β) that best indicates that the crop is in a specific state, a region 2405 where the crop is in a specific state is detected by pointing, which is an example of the input device 110. Specify using a device, etc. Next, the operator of the information processing apparatus 10 uses the pull-down menu 2406 to specify a specific state of the crops in the area 2405. After specifying the area 2405 and specifying the specific state of the crop using the pull-down menu 2406, the operator of the information processing device 10 clicks the OK button 2407 using a pointing device or the like to input the end of the specified operation. conduct.

Next, FIG. 25 shows an example of the output device 111 of the information processing device 10 in the process of specifying an area where crops are in a specific state by the area specifying unit 137 and the type specifying unit 138, which is shown in the flowchart of FIG. FIG. 2 is a diagram showing an example of a screen displayed on a display.

A spectral image 2501 is displayed on the screen 2500, and an area 2502 is provided in which information indicating the imaging date and time of the spectral image 2501 and the imaging area are displayed. Additionally, map data 2503 corresponding to the area where the spectrum image 2501 was captured is displayed. When the operator of the information processing device 10 specifies a specific state of the crop that he/she wishes to identify using the pull-down menu 2504, a region 2505 that is estimated to be in the specified specific state is displayed in the spectral image 2501. Is displayed. This region 2505 is also the density distribution of the spectrum image 2501.

<Effects of embodiment>
As described above in detail, according to the system 1 of the present embodiment, it becomes possible for a business operator engaged in a harvesting business to manage the growing situation more appropriately.

<Modified example>
Note that the configurations of the embodiments described above are explained in detail in order to explain the present disclosure in an easy-to-understand manner, and the embodiments are not necessarily limited to those having all of the configurations described. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations.

As an example, in the system 1 of the embodiment described above, the spectral camera 32 mounted on the satellite 30 images the ground surface to obtain a spectral image 3103, and the spectral camera 22 mounted on the drone 20 captures the area to be observed. Although the spectral image 2104 is acquired by imaging, the mobile object on which the spectral camera for acquiring the spectral image is mounted is not limited to the satellite 30 or the drone 20, but any mobile object that can image the ground surface from the air can be used. There are no limitations. Furthermore, methods in which a mobile object equipped with a spectral camera moves on the ground surface, such as an operator holding a spectral camera moving on the ground surface, or a car equipped with a spectral camera driving on the ground surface, etc. It may be a method. One example is a mode in which a multispectral camera realized by a smartphone is used as the spectroscopic terminal device disclosed in Japanese Patent No. 6,342,594.

As described above, there may be various combinations and variations of the method of acquiring the spectral image 3103 for library creation and the method of acquiring the spectral image 2104 by imaging the area to be observed. A modification of this variation will be described in detail below.

The LTCF camera, which is the spectrum camera 32 mounted on the satellite 30, can receive reflected light in many (for example, several hundred) wavelength bands to obtain a spectrum image 3103. On the other hand, the spectral camera 22 mounted on the drone 20 uses the library to limit the wavelength bands to (a plurality of) wavelength bands that can detect that crops in a specific state in the observation target area. A spectral image 2104 is obtained from the reflected light.

Such a difference in the configurations of the spectral cameras 22 and 32 is also reflected in the weight and price of the spectral cameras 22 and 32. That is, the spectral camera 32 is more expensive and heavier than the spectral camera 22. The difference in weight between the spectral cameras 22 and 32 also affects the configuration of the aircraft on which the spectral cameras 22 and 32 are mounted. In other words, when the spectral camera 32 is mounted on the drone 20, the configuration of the drone 20 has to be increased in size, and furthermore, the time that it can fly at one time may be limited to a short time.

On the other hand, if the spectral camera 32 is mounted on the satellite 30, the restriction due to the weight of the spectral camera 32 will be somewhat relaxed, but the frequency with which spectral images 3103 of the area to be observed can be acquired will inevitably be lower. That is, the satellite 30 may pass over the area to be observed only once every several months, and therefore the frequency at which the spectral image 3103 can be obtained may also be about once every several months.

For this reason, at the beginning of operation of the system 1, the number of times the spectrum image 3103 of the area to be observed is acquired by the spectral camera 32 is limited to a certain value, and the time at which the spectral camera 32 images the area to be observed is also limited. It is conceivable to fix it (for example, 12:00 in the daytime) and create a library using the solar angle as the solar angle at that time in the area. In this case, the sun angle is not corrected when creating the library.

Next, the time at which the spectrum camera 22 images the area to be observed is set to the time at which the area to be observed is imaged by the spectrum camera 32 (in the above example, 12:00 in the daytime). By capturing images, it is possible to detect whether crops in the area being observed are in a particular state without having to correct the sun's altitude. After this, the drone 20 or the like is repeatedly flown while changing the time at which the spectral camera 22 images the area to be observed, and spectral images 2104 of the area to be observed at multiple times are obtained. By correcting the sun angle, it is possible to detect whether the crops in the area being observed are in a specific state. As described above, the sun angle correction value may be stored in the teacher data 125 and the learning model 126 may be retrained.

The advantage of acquiring spectral images 3103 for library creation with the spectral camera 32, which is an LTCF camera, is that the spectral images 3103 of reflected light in a large number of wavelength bands described above can be acquired with a small number of shots (in extreme case, once). . By acquiring a spectral image 3103 using reflected light in multiple wavelength bands, it is possible to accurately select a suitable wavelength band to detect whether or not crops in a region to be observed are in a specific state. can. Thereby, the number of wavelength bands detected by the spectral camera 22 can be reduced without lowering the accuracy of detecting whether or not crops are in a specific state in the area to be observed. This also leads to a reduction in the weight of the spectral camera 22, and allows the use of a highly versatile flying vehicle such as the drone 20 equipped with the spectral camera 22. Furthermore, it also allows the use of a highly versatile flying vehicle such as the drone 20 for a long period of time and at high frequency. It also leads to flight. If the drone 20 etc. can be flown for a long time, the cost for acquiring the spectral image 2104 can be reduced, and if the drone 20 etc. can be flown frequently, the crops in the area to be observed will be in a specific state. This also helps in quickly detecting whether or not it exists.

Furthermore, by matching the acquisition time of the spectral image 2104 to the acquisition time of the spectral image 3103 without correcting the solar angle when initially creating the library, it is possible to save time and effort when starting up the system 1. The time required to install System 1 can be reduced. By acquiring spectral images 2104 and 3103 at a plurality of times as the system 1 operates, it is possible to improve the accuracy of detecting whether or not crops are in a specific state.

Furthermore, in order to identify the area where the crops are in a particular state, the system 1 of the above-described embodiment uses a spectral image, but in addition to the spectral image, the temperature distribution of the area may also be used. To do this, it is necessary to obtain the temperature of this region using a sensor, etc. when acquiring the spectral image that is the basis for library creation, and also to obtain the temperature of this region when acquiring the spectral image of the area to be observed. It may be acquired by a sensor or the like. At this time, it is preferable to provide a plurality of temperature measurement points so that the temperature distribution has a two-dimensional spread. In this case, the library also includes temperature distribution data for the area.

Furthermore, in the system 1 of the above-described embodiment, the learning model 126 is used to identify areas where crops are in a specific state in the observation target area, but so-called image analysis technology is used to identify areas where crops are in a specific state. You may also specify an area within.

Furthermore, in the system 1 according to the present disclosure, the following configuration is also possible.

A first multispectral camera is installed to store in memory information for detecting that a crop being observed is in a specific state for comparison with a spectral image taken by the multispectral camera. A spectral image is generated by flying an aircraft and photographing the earth's surface.

Next, an aircraft equipped with a second multispectral camera that is different from the first multispectral camera is flown in accordance with the flight plan in the area to be observed, and the wavelengths used to detect that the crops are in a specific state are A spectral image is generated by photographing with a second multispectral camera. Preferably, the aircraft carrying the first multispectral camera and the aircraft carrying the second multispectral camera are different aircraft.

The first multispectral camera has a member, such as a liquid crystal variable wavelength filter 34, that can switch the wavelength to be imaged, and can generate spectral images for a large number of wavelengths. On the other hand, the second multispectral camera can generate spectral images for specific wavelengths. The specific wavelength cannot be switched while the second multispectral camera is imaging the area to be observed. That is, the specific wavelength is a predetermined wavelength.

The second multispectral camera may not have a specific member for capturing spectral images, compared to the first multispectral camera. For example, unlike the first multispectral camera which has a wavelength switchable member, the second multispectral camera does not have a wavelength switchable member and generates a spectral image at a specific wavelength. There may be one. The second flying vehicle carrying the second multispectral camera may have a lower loadable weight, unlike the first flying vehicle carrying the first multispectral camera. This may mean that the first flying object and the second flying object have different weights (the second flying object is less than the first flying object), or based on these differences in weight. Therefore, the airspace (geographical range determined by latitude and longitude, and altitude) in which the first flying object and the second flying object can fly may be different. Even if the second aircraft carries a plurality of (for example, about four) second multispectral cameras, and each second multispectral camera generates spectral images at different wavelengths. good.

The first multispectral camera 32 capable of acquiring spectral images in a large number of wavelength bands, such as the multispectral camera 32 equipped with the liquid crystal variable wavelength filter 34, tends to be expensive and heavy. On the other hand, the second multispectral camera 22, which can generate only spectral images for specific wavelengths, is inexpensive and can be configured to be lightweight.

Since the first multispectral camera 32 generates spectral images for library creation, it is preferable to generate spectral images based on reflected light in multiple wavelength bands. On the other hand, the second multispectral camera 22 is used to identify areas where crops are in a specific state in the observation target area, and is preferably photographed frequently by a flying object such as the drone 20. Therefore, the second multispectral camera 22 has a great advantage of being lighter and cheaper than the first multispectral camera 32.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely realized by hardware, for example, by designing an integrated circuit. Further, the present invention can also be realized by software program codes that realize the functions of the embodiments. In this case, a storage medium on which a program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored on the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the embodiments described above, and the program code itself and the storage medium storing it constitute the present invention. Storage media for supplying such program codes include, for example, flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, and non-volatile memory cards. , ROM, etc. are used.

Furthermore, the program code that implements the functions described in this embodiment can be implemented using a wide range of program or script languages, such as assembler, C/C++, Perl, Shell, PHP, and Java (registered trademark).

Furthermore, by distributing the software program code that realizes the functions of the embodiment via a network, it can be stored in a storage means such as a computer's hard disk or memory, or a storage medium such as a CD-RW or CD-R. Alternatively, a processor included in the computer may read and execute the program code stored in the storage means or the storage medium.

<Additional notes>
The matters explained in each of the above embodiments are additionally described below.

(Additional note 1)
A method for operating a computer (10), wherein a processor (101) of the computer (10) is provided with an observation target for comparison with a spectral image (127) taken by a multispectral camera (22). A first step (S1908) of storing information (126) in the memory (120) for detecting that the crops are in a specific state, and detecting that the crops are in a specific state in the area to be observed. A second step (S2002) of generating a spectral image (127) by photographing with a multispectral camera (22) according to the wavelength of the image, and a spectral image (127) that is the result of the photographing, and a spectral image (127) that shows that the crop is in a specific state. A third step (S2105) of identifying an area where crops in a specific state are located in the observation target area based on the information (126) for detecting this, and a fourth step of outputting information on the identified area. Step (S2106),
How to run it.
(Additional note 2)
In the second step, an aircraft (20) equipped with a multispectral camera (22) is flown in accordance with the flight plan in the area to be observed, and a multispectral camera (20) is used to detect crops using wavelengths to detect specific conditions. The method according to appendix 1, wherein the spectral image (127) is generated by photographing with the camera (22).
(Additional note 3)
In the first step (S1908), the detection information includes information indicating the area where the target crop is grown and a spectral image (127 ), the wavelength information for photographing it, and the sun angle information when photographing crops in the area are associated. A trained model ( 126) is stored in the memory (120), and in the third step (S2105), the crops in a specific state are The method described in Appendix 1 for identifying the area where there is.
(Additional note 4)
In the first step (S1908), machine learning is performed using, as detection information, a database associated with information on the temperature of the area when the area is imaged with the multispectral camera (22) as training data (125). The trained model (126) generated by performing the above steps is stored in the memory (120), and in the third step, the area to be observed when the area to be observed is imaged by the multispectral camera (22) is further stored in the memory (120). The method according to appendix 3, wherein regions with crops in a particular state are identified based on the temperature of the area.
(Appendix 5)
In the first step (S1908), information on a specific state type is further associated with the database as the teacher data (125), and the trained data generated by the database containing the information on the specific state type is The model (126) is stored in the memory (120), and in the third step (S2105), based on the learned model (126), areas where crops are in a specific state and the type of the specific state are determined. The method according to appendix 3 or 4 for identifying.
(Appendix 6)
In the second step (S2002), as an operation plan, multiple aircraft (20) are flown over different areas, and spectral images (127) taken by each aircraft are superimposed based on the photographed areas. The method according to appendix 2, which together generates a spectral image (128).
(Appendix 7)
In the second step (S2002), as an operation plan, the wavelength of the multispectral camera (22) is specified for each region to be observed according to the crops to be observed, and the wavelength is switched in each region according to the wavelength specification. , the method of claim 2, wherein the spectral image (127) is generated by a multispectral camera (22) of the aircraft.
(Appendix 8)
In the second step (S2002), as a flight plan, if the topography around the crop growing place satisfies predetermined conditions, priority is given to the area that satisfies the predetermined conditions, and the multispectral camera (22) of the aircraft The method according to appendix 2, wherein the spectral image (127) is generated by photographing the spectrum image.
(Appendix 9)
In the third step (S2105), the geographical range in which the crops are in a specific state and the geographical range in which they are not are distinguished, and furthermore, based on the point where the spectral image (127) was taken, the spectral image (127) is superimposed on the map image. , the method according to any one of appendices 1 to 8, which displays the range in which the crop is in a specific state.
(Appendix 10)
In the third step (S2105), the designation of the photographing time point is accepted from the user, and based on the spectral image (127) photographed at the designated time point, the geographical range where the crops are in a particular state is displayed. The method described in any of Supplementary Notes 1 to 9.
(Appendix 11)
In the third step (S2105), based on the result of detecting that the crops are in a specific state, information on the crops in the specific state and costs for dealing with the crops in the specific state are provided. The method according to any one of Supplementary Notes 1 to 10, for displaying information.
(Appendix 12)
In the first step (S1908), a flying object (30) equipped with a multispectral camera (32) is flown to photograph the ground surface, and based on the obtained spectral image (127), crops to be observed are grown. information indicating the area where the area is located, a spectral image (127) obtained by photographing the area with a multispectral camera (32), and information on the wavelength for photographing the area, in association with each other, and storing the information in the memory (120); Furthermore, by allowing the user to view the stored spectral image (127) in the processor (101) of the computer (10), the user can specify that the crop is in a particular state in association with the spectral image (127). a fifth step (S1904) of accepting from the user; and a sixth step (S1906) of storing in the memory (120) a database (125) for detecting that the crop is in a specific state according to the user's specification. , the method according to any one of appendices 2 to 8.
(Appendix 13)
The spectral images used to create the database are images taken of the same region for a first wavelength group consisting of a plurality of wavelengths, and the spectral images in the second step are images taken of the same area for a second wavelength group consisting of a plurality of wavelengths. The method according to supplementary note 12, wherein the image is taken of an area to be observed, and the number of wavelengths in the first wavelength group is greater than the number of wavelengths in the second wavelength group.
(Appendix 14)
In the first step (S1908), information indicating the area is associated with information on the spectrum image (127) and the wavelength for photographing the same, and further information on the type of specific state is stored in the memory (120). The method according to appendix 12 or 13.
(Appendix 15)
Based on the specification received from the user in the fifth step (S1904), the wavelength for detecting that the crop is in a specific state by the spectral image (127) is specified, according to any one of appendices 12 to 14. Method.
(Appendix 16)
The method according to any one of appendices 12 to 15, wherein in the first step (S1908), information on the temperature of the area when the area is imaged by the multispectral camera (23) is stored in association with the information in the memory.
(Appendix 17)
Among the information stored in the memory (120), if the topography around the crop growing place satisfies a predetermined condition, a spectral image (127) of the area that satisfies the predetermined condition is extracted, and the extracted spectrum is extracted. The method according to any one of appendices 12 to 16, wherein the image (127) is presented to the user in a designable manner.
(Appendix 18)
Supplementary notes 12 to 12, in which, in response to receiving an input of information regarding a region from a user, a spectral image (127) of the region related to the input is extracted, and the extracted spectral image (127) is presented to the user in a designable manner. 17. The method according to any one of 17.
(Appendix 19)
A program for causing a computer (10) to execute the method according to any one of Supplementary Notes 1 to 18.
(Additional note 20)
An information processing device that executes the method described in any one of Supplementary Notes 1 to 18.

1... System 2, 30... Satellite 3, 20... Drone 4, 10... Information processing device 101... Processor 102... Main storage device 103... Auxiliary storage device 120, 210, 310... Storage unit 121, 2100, 3100... Application program 122 ...Flight plan DB 123, 2103, 3102...Image DB 124...Wide area image DB 125...Teacher data 126...Learning model 127, 2104, 3103...Spectral image 128...Wide area spectral image 130, 211, 311...Control unit 133...Learning model Generation unit 134...Flight plan creation unit 135...Flight control unit 136...Image synthesis unit 137...Area identification unit 138...Type identification unit 21, 31...Spectral camera control device 22, 32...Spectral camera N...Network

Claims

A method for operating a computer, the method comprising: a processor of the computer;
a first step of storing in a memory information for detecting that the crop being observed is in a specific state for comparison with a spectral image taken by a multispectral camera;
a second step of generating the spectral image by photographing with the multispectral camera at a wavelength for detecting that the crop is in a specific state in the observation target area;
Identifying the area where the crop is in a specific state in the area to be observed based on the spectral image that is the photographed result and information for detecting that the crop is in a specific state. The third step is to
a fourth step of outputting information on the identified area;
How to run it.
In the second step, a flying vehicle equipped with the multispectral camera is flown in the area to be observed according to the flight plan, and the multispectral camera is equipped with a wavelength for detecting that the crop is in a specific state. generating the spectral image by photographing with
The method according to claim 1.
In the first step, as the information for detecting,
Information indicating the area where the crop to be observed is grown;
the spectrum image obtained by photographing the area with the multispectral camera and information on the wavelength for photographing the spectrum;
Information on the angle of the sun when photographing the crop in the area;
A trained model generated by performing machine learning using a database associated with as training data is stored in the memory,
In the third step, the area where the crops in the specific state are located is identified based on the spectral image that is the photographed result and the learned model.
The method according to claim 1.
In the first step, the machine learning is performed using, as the detection information, a database associated with information on the temperature of the area when the area is imaged with the multispectral camera as the teacher data. The generated trained model is stored in the memory,
In the third step, further identifying the area where the crops in the specific state are located based on the temperature of the observation area when the multispectral camera images the observation area;
The method according to claim 3.
In the first step, the database, which is the teacher data, is further associated with information on the specific state type and the learned data generated by the database containing the specific state type information. The model is stored in the memory,
In the third step, based on the learned model, specifying the region where the crop is in the specific state and the type of the specific state;
The method according to claim 3 or 4.
In the second step, as the operation plan, a plurality of the aircraft fly over different areas, and the spectral images taken by each of the aircraft are superimposed based on the area where the images were taken. generating the spectral image;
The method according to claim 2.
In the second step, as the operation plan, the wavelength of the multispectral camera is specified for each region to be observed according to the crop to be observed, and the wavelength is set in each region according to the designation of the wavelength. switching to generate the spectral image by the multispectral camera of the aircraft;
The method according to claim 2.
In the second step, as the operation plan, if the topography around the place where the crops are grown satisfies a predetermined condition, the multispectral camera of the aircraft is given priority to the area that satisfies the predetermined condition. generating the spectral image by photographing the spectral image,
The method according to claim 2.
In the fourth step, a geographical range in which the crop is in a specific state and a geographical range in which it is not are distinguished, and further, based on the point where the spectral image was taken, the spectral image is superimposed on a map image, and the crop is in a specific state. Display the range where is in a specific state,
The method according to any one of claims 1 to 8.
In the fourth step, a designation of a photographing time point is accepted from the user, and a geographical range in which the crop is in a particular state is displayed based on the spectral image photographed at the designated time point.
The method according to any one of claims 1 to 9.
In the fourth step, based on the result of detecting that the crop is in a specific state, information on the crop in a specific state and a cost for dealing with the crop being in the specific state. display information and
The method according to any one of claims 1 to 10.
In the first step,
The flying object equipped with the multispectral camera is caused to fly to photograph the ground surface, and based on the obtained spectral image,
Information indicating the area where the crop to be observed is grown;
the spectrum image obtained by photographing the area with the multispectral camera and information on the wavelength for photographing the spectrum;
associated and stored in the memory,
Furthermore, the processor of the computer,
a fifth step of receiving from the user a designation that the crop is in a particular state in association with the stored spectrum image by allowing the user to view the stored spectrum image;
a sixth step of storing in the memory a database for detecting that the crop is in the specific state according to the user's designation;
The method according to any one of claims 2 to 8, wherein the method is performed.
The spectral image for creating the database is an image of the same region for a first wavelength group consisting of a plurality of wavelengths,
The spectral image in the second step is an image of the area of the same observation target for a second wavelength group consisting of a plurality of wavelengths,
13. The method of claim 12, wherein the number of wavelengths in the first wavelength group is greater than the number of wavelengths in the second wavelength group.
In the first step, information indicating the region is associated with information on the spectrum image and the wavelength for photographing the spectrum image, and further information on the type of the specific state is stored in the memory.
The method according to claim 12 or 13.
15. The method according to claim 12, wherein a wavelength for detecting that the crop is in the specific state using the spectral image is specified based on the designation received from the user in the fifth step. Method.
In the first step, further storing information on the temperature of the area when the area is imaged with the multispectral camera in association with the memory;
The method according to any one of claims 12 to 15.
Among the information stored in the memory, when the topography around the place where the crops are grown satisfies a predetermined condition, the spectral image of the area that satisfies the predetermined condition is extracted, and the extracted spectral image to the user in a designable manner;
The method according to any one of claims 12 to 16.
In response to receiving input of information regarding the region from the user, extracting the spectral image of the region related to the input, and presenting the extracted spectral image to the user in a designable manner;
The method according to any one of claims 12 to 17.
A program for causing a computer to execute the method according to claim 1.
An information processing device that executes the method according to claim 1.