JP6690106B1 - Determination device, imaging system, and moving body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To sometimes not properly set an imaging condition suitable for a specific object in a multi-spectral camera which captures an image for each of a plurality of wavelength bands. A determining device determines a first region in a first image based on a first image in a first wavelength region captured by a first image sensor included in the first image capturing device, and a second image capturing device. Is configured to determine the imaging control value of the second imaging device based on the image information of the second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor included in An integrated circuit. [Selection diagram] Figure 10

Description

  The present invention relates to a determination device, an imaging system, and a moving body.

In Patent Document 1, when sensing an area including an inspection object and a reference reflection area, a reference reflection area having a reflectance corresponding to the inspection object for each wavelength band to be sensed by the inspection object. A sensing system for sensing is disclosed.
[Prior Art Document]
[Patent Document]
[Patent Document 1] International Publication No. 2017/221756

  In a multi-spectral camera that captures an image for each of a plurality of wavelength bands, an imaging condition suitable for a specific object may not be set appropriately.

  A determination device according to one aspect of the present invention determines a first region in a first image based on a first image in a first wavelength region imaged by a first image sensor included in a first imaging device, The imaging control value of the second imaging device is determined based on the image information of the second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor included in the second imaging device. A circuit configured as such may be included.

  The image information may be brightness information, and the imaging control value may be an exposure control value.

  The circuit may determine the first region based on the first image and the second image.

  The circuit uses, as an image used for determining the first region, the first image and the second image, or the third image of the third wavelength region captured by the third image sensor included in the first image and the third imaging device. May be selected. The circuit may determine the first region based on the selected first and second images or the first and third images. The circuit determines the imaging control value of the first imaging device based on the image information of the first region of the first image, and the imaging control of the second imaging device based on the image information of the second region of the second image. The value may be determined, and the imaging control value of the third imaging device may be determined based on the image information of the third area corresponding to the first area of the third image.

  The circuit may select the first image and the second image or the first image and the third image based on the characteristics of the subject.

  The first wavelength range may be the near infrared range. The second wavelength region may be a red wavelength region. The third wavelength region may be a wavelength region of a green region or a red edge region.

  The circuit, as an image used for determining the first region, the first image and the second image, the third image of the third wavelength region imaged by the third image sensor included in the first image and the third imaging device, Alternatively, the first image and the fourth image in the fourth wavelength region captured by the fourth image sensor included in the fourth imaging device may be selected. The circuit may determine the first region based on the selected first image and second image, first image and third image, or first image and fourth image. The circuit determines the imaging control value of the first imaging device based on the image information of the first region of the first image, and the imaging control of the second imaging device based on the image information of the second region of the second image. A value is determined, the imaging control value of the third imaging device is determined based on the image information of the third area corresponding to the first area of the third image, and the fourth area corresponding to the first area of the fourth image. The imaging control value of the fourth imaging device may be determined based on the image information of.

  The circuit may select the first image and the second image, the first image and the third image, or the first image and the fourth image based on the characteristics of the subject.

  The first wavelength range may be the near infrared range. The second wavelength region may be a red wavelength region. The third wavelength region may be a wavelength region in the green region. The fourth wavelength region may be the wavelength region of the red edge region.

  A determination device according to one aspect of the present invention includes a first image in a first wavelength region captured by a first image sensor included in a first imaging device, and a second image sensor captured in a second image sensor included in a second imaging device. A first region in the third image of the third wavelength region, which is determined by the third image sensor included in the third imaging device, is determined based on the second image of the two-wavelength region. A circuit configured to determine an imaging control value of the third imaging device based on the image information of the third region corresponding to

  The image information may be brightness information, and the imaging control value may be an exposure control value.

  The first wavelength range may be the near infrared range. The second wavelength region may be a red region, a green region, or a red edge region. The third wavelength region may be a red wavelength region, a green wavelength region, and a blue wavelength region.

  An imaging system according to one aspect of the present invention may include the above determination device. The imaging system may include a first imaging device and a second imaging device.

  The moving body according to one aspect of the present invention may be a moving body that includes the above-described imaging system and moves.

  A determination method according to an aspect of the present invention includes a step of determining a first region in a first image based on a first image in a first wavelength region captured by a first image sensor included in a first imaging device. You may be prepared. The determination method is based on the image information of the second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor included in the second image capturing device. The step of determining a control value may be provided.

  The program according to one aspect of the present invention may be a program for causing a computer to function as the determination device.

  According to one aspect of the present invention, in a multispectral camera that captures an image for each of a plurality of wavelength bands, it is possible to appropriately set imaging conditions suitable for a specific subject.

  Note that the above summary of the invention does not enumerate all the necessary features of the invention. Further, a sub-combination of these feature groups can also be an invention.

It is a figure showing an example of the appearance of an unmanned aerial vehicle (UAV) and a remote control. It is a figure which shows an example of the external appearance of the imaging system mounted in UAV. It is a figure which shows an example of the functional block of UAV. It is a figure which shows an example of the functional block of an imaging system. It is a figure which shows an example of the image imaged by an imaging system. It is a figure which shows the brightness | luminance distribution of a part of image shown in FIG. It is a figure for demonstrating the block which divided the image. It is a figure for demonstrating the region of interest in an image. It is a figure which shows an example of the image imaged by an imaging system. 9 is a flowchart showing an example of a procedure for determining an exposure control value. It is a figure which shows another example of the external appearance of the imaging system mounted in UAV. It is a figure which shows an example of a hardware configuration.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Moreover, not all of the combinations of features described in the embodiments are essential to the means for solving the invention. It is obvious to those skilled in the art that various modifications and improvements can be added to the following embodiments. It is apparent from the scope of the claims that the embodiments added with such modifications or improvements can be included in the technical scope of the present invention.

  The claims, the description, the drawings, and the abstract contain the subject matter of copyright protection. The copyright owner has no objection to the reproduction of any of these documents by anyone as it appears in the JPO file or record. However, in all other cases, all copyrights are reserved.

  Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where a block is (1) a stage of a process in which an operation is performed or (2) a device responsible for performing an operation. "Part" of may be represented. Particular stages and "sections" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. Memory elements and the like.

  Computer-readable media may include any tangible device capable of storing instructions executed by a suitable device. As a result, a computer-readable medium having instructions stored therein will comprise a product that includes instructions that may be executed to create the means for performing the operations specified in the flowcharts or block diagrams. Examples of computer readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), Electrically Erasable Programmable Read Only Memory (EEPROM), Static Random Access Memory (SRAM), Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD), Blu-Ray (RTM) Disc, Memory Stick, Integrated Circuit cards and the like may be included.

  Computer readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes conventional procedural programming languages. Conventional procedural programming languages include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state set data, or Smalltalk®, JAVA®, C ++. It may be an object oriented programming language such as, and the like, and a "C" programming language or similar programming language. Computer-readable instructions are provided to a processor or programmable circuit of a general purpose computer, a special purpose computer, or other programmable data processing device, locally or in a wide area network (WAN) such as a local area network (LAN), the Internet, or the like. ). The processor or programmable circuit may execute computer readable instructions to create a means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

  FIG. 1 shows an example of the appearance of an unmanned aerial vehicle (UAV) 10 and a remote control device 300. The UAV 10 includes a UAV body 20, a gimbal 50, a plurality of image pickup devices 60, and an image pickup system 100. The gimbal 50 and the imaging system 100 are examples of an imaging system. The UAV 10 is an example of a mobile body. The moving body is a concept including a flying body moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like. The flying body moving in the air is a concept including UAV, other aircraft moving in the air, an airship, a helicopter, and the like.

  The UAV body 20 includes a plurality of rotary blades. The plurality of rotary blades is an example of the propulsion unit. The UAV body 20 causes the UAV 10 to fly by controlling the rotation of a plurality of rotor blades. The UAV body 20 flies the UAV 10 by using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed wing aircraft having no rotary wing.

  The imaging system 100 is a multispectral camera for imaging that images an object included in a desired imaging range for each of a plurality of wavelength bands. The gimbal 50 rotatably supports the imaging system 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 supports the imaging system 100 using an actuator so as to be rotatable on the pitch axis. The gimbal 50 further supports the imaging system 100 by using an actuator so as to be rotatable about each of a roll axis and a yaw axis. The gimbal 50 may change the posture of the imaging system 100 by rotating the imaging system 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

  The plurality of imaging devices 60 are sensing cameras that capture images of the surroundings of the UAV 10 in order to control the flight of the UAV 10. The two imaging devices 60 may be provided on the front of the UAV 10, which is the nose. Still another two imaging devices 60 may be provided on the bottom surface of the UAV 10. The two imaging devices 60 on the front side may be paired and may function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired and function as a stereo camera. The imaging device 60 may measure the presence of an object included in the imaging range of the imaging device 60 and the distance to the object. The imaging device 60 is an example of a measuring device that measures an object existing in the imaging direction of the imaging system 100. The measuring device may be another sensor such as an infrared sensor or an ultrasonic sensor that measures an object existing in the imaging direction of the imaging system 100. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of imaging devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV 10 may include at least one imaging device 60 on each of the nose, tail, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the imaging device 60 may be wider than the angle of view that can be set by the imaging system 100. The imaging device 60 may have a single focus lens or a fisheye lens.

  The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may wirelessly communicate with the UAV 10. The remote control device 300 transmits instruction information indicating various commands regarding movement of the UAV 10, such as ascending, descending, accelerating, decelerating, moving forward, moving backward, and rotating, to the UAV 10. The instruction information includes, for example, instruction information for increasing the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at the altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to elevate the UAV 10. The UAV 10 rises while accepting the rise command. Even if the UAV 10 receives the climb command, the UAV 10 may limit the climb when the altitude of the UAV 10 reaches the upper limit altitude.

  FIG. 2 is a diagram showing an example of the external appearance of the imaging system 100 mounted on the UAV 10. The imaging system 100 is a multi-spectral camera that captures image data for each of a plurality of predetermined wavelength bands. The imaging system 100 includes an R imaging device 110, a G imaging device 120, a B imaging device 130, an RE imaging device 140, and an NIR imaging device 150. The imaging system 100 records each image data imaged by the R imaging device 110, the G imaging device 120, the B imaging device 130, the RE imaging device 140, and the NIR imaging device 150 as a multispectral image. be able to. Multi-spectral images may be used, for example, to make predictions about crop health and vitality.

  FIG. 3 shows an example of functional blocks of the UAV 10. The UAV 10 includes a UAV control unit 30, a memory 32, a communication interface 36, a propulsion unit 40, a GPS receiver 41, an inertial measurement device 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a humidity sensor 46, a gimbal 50, and an imaging device. 60 and the imaging system 100.

  The communication interface 36 communicates with other devices such as the remote control device 300. The communication interface 36 may receive instruction information including various commands to the UAV control unit 30 from the remote control device 300. In the memory 32, the UAV control unit 30, the propulsion unit 40, the GPS receiver 41, the inertial measurement device (IMU) 42, the magnetic compass 43, the barometric altimeter 44, the temperature sensor 45, the humidity sensor 46, the gimbal 50, the imaging device 60, Also, it stores programs and the like necessary for controlling the imaging system 100. The memory 32 may be a computer-readable recording medium, and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 32 may be provided inside the UAV body 20. It may be detachably provided from the UAV body 20.

  The UAV control unit 30 controls the flight and imaging of the UAV 10 according to the program stored in the memory 32. The UAV control unit 30 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The UAV control unit 30 controls flight and imaging of the UAV 10 according to a command received from the remote control device 300 via the communication interface 36. The propulsion unit 40 propels the UAV 10. The propulsion unit 40 has a plurality of rotating blades and a plurality of drive motors that rotate the plurality of rotating blades. The propulsion unit 40 causes the UAV 10 to fly by rotating a plurality of rotary wings via a plurality of drive motors according to a command from the UAV control unit 30.

  The GPS receiver 41 receives a plurality of signals indicating the times transmitted from a plurality of GPS satellites. The GPS receiver 41 calculates the position (latitude and longitude) of the GPS receiver 41, that is, the position (latitude and longitude) of the UAV 10 based on the received signals. The IMU 42 detects the attitude of the UAV 10. The IMU 42 detects, as the attitude of the UAV 10, accelerations in the front-rear, left-right, and up-down directions of the UAV 10, and angular velocities in the 3-axis directions of pitch, roll, and yaw. The magnetic compass 43 detects the heading of the UAV 10 nose. Barometric altimeter 44 detects the altitude at which UAV 10 flies. The barometric altimeter 44 detects the atmospheric pressure around the UAV 10, converts the detected atmospheric pressure into an altitude, and detects the altitude. The temperature sensor 45 detects the temperature around the UAV 10. The humidity sensor 46 detects the humidity around the UAV 10.

  FIG. 4 shows an example of functional blocks of the imaging system 100. The imaging system 100 includes an R imaging device 110, a G imaging device 120, a B imaging device 130, an RE imaging device 140, and an NIR imaging device 150. The imaging system 100 includes an image processor 180, a transmission unit 190, and a memory 192.

  The R image pickup device 110 includes an R image sensor 112 and an optical system 114. The R image sensor 112 captures the image formed by the optical system 114. The R image sensor 112 has a filter that transmits light in the red wavelength range and outputs an R image signal that is an image signal in the red wavelength range. The wavelength band in the red region is, for example, 620 nm to 750 nm. The wavelength band in the red region may be a specific wavelength band in the red region, and may be, for example, 663 nm to 673 nm.

  The G imaging device 120 includes a G image sensor 122 and an optical system 124. The G image sensor 122 captures the image formed by the optical system 124. The G image sensor 122 has a filter which transmits light in the green wavelength band, and outputs a G image signal which is an image signal in the green wavelength band. The wavelength band in the green region is, for example, 500 nm to 570 nm. The wavelength band in the green region may be a specific wavelength band in the green region, and may be, for example, 550 nm to 570 nm.

  The B imaging device 130 includes a B image sensor 132 and an optical system 134. The B image sensor 132 captures the image formed by the optical system 134. The B image sensor 132 has a filter that transmits light in the blue wavelength range, and outputs a B image signal that is an image signal in the blue wavelength range. The wavelength band in the blue region is, for example, 450 nm to 500 nm. The wavelength band in the blue region may be a specific wavelength band in the blue region, and may be, for example, 465 nm to 485 nm.

  The RE image pickup device 140 includes a RE image sensor 142 and an optical system 144. The RE image sensor 142 captures the image formed by the optical system 144. The RE image sensor 142 has a filter that transmits light in the wavelength band of the red edge region, and outputs an RE image signal that is an image signal in the wavelength band of the red edge region. The wavelength band of the red edge region is, for example, 705 nm to 745 nm. The wavelength band of the red edge region may be 712 nm to 722 nm.

  The NIR imaging device 150 has an NIR image sensor 152 and an optical system 154. The NIR image sensor 152 captures the image formed by the optical system 154. The NIR image sensor 152 has a filter that transmits light in the wavelength band of the near infrared region, and outputs an NIR image signal which is an image signal in the wavelength band of the near infrared region. The wavelength band in the near infrared region is, for example, 800 nm to 2500 nm. The wavelength band in the near infrared region may be 800 nm to 900 nm.

  The image processor 180 includes a multiplex 170, an input reception unit 172, a demosaic processing unit 174, and a recording processing unit 178. The image processor 180 is an example of a circuit. The image processor 180 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like.

  The multiplex 170 receives the image signal output from each image sensor, selects the image signal output from any one of the image sensors according to a predetermined condition, and inputs the image signal to the input reception unit 172.

  The demosaic processing unit 174 generates image data for display based on the R image signal, the G image signal, and the B image signal input to the input reception unit 172. The demosaic processing unit 174 generates image data for display by performing demosaic processing on the R image signal, the G image signal, and the B image signal. The demosaic processing unit 174 performs a thinning process on the R image signal, the G image signal, and the B image signal, and converts the thinned R image signal, the G image signal, and the B image signal into a Bayer array image signal. Then, image data for display may be generated. The transmission unit 190 transmits the image data for display to the display device. The transmission unit 190 may transmit image data for display to the remote control device 300, for example. The remote control device 300 may display the image data for display on the display unit as a live view image.

  The recording processing unit 178 records image data for recording according to a predetermined recording format based on the R image signal, G image signal, B image signal, RE image signal, and NIR image signal input to the input receiving unit 172. To generate. The recording processing unit 178 may generate RAW data as image data for recording from the R image signal, G image signal, B image signal, RE image signal, and NIR image signal in accordance with the RAW format. The recording processing unit 178 may generate image data for recording full pixels without thinning out each of the R image signal, the G image signal, the B image signal, the RE image signal, and the NIR image signal. The recording processing unit 178 may store the image data for recording in the memory 192. The memory 192 may be a computer-readable recording medium, and may include at least one of SRAM, DRAM, EPROM, EEPROM, and flash memory such as USB memory. The memory 192 may be provided inside the housing of the imaging system 100. The memory 192 may be provided to be removable from the housing of the imaging system 100.

  The image processor 180 further includes a reception unit 184 and a switching unit 186. The receiving unit 184 receives a storage instruction to store the image data for recording in the memory 192. The receiving unit 184 may receive a storage instruction from the user via an external terminal such as the remote control device 300. The reception unit 184 may receive a storage instruction from the UAV control unit 30 when the position of the imaging system 100 is a predetermined position. When the position of the UAV 10 is a predetermined position, the reception unit 184 determines that the position of the imaging system 100 is the predetermined position by the UAV control unit 30 and issues a storage instruction from the UAV control unit 30. You may accept. The imaging system 100 may include a GPS receiver. In this case, the image processor 180 may determine whether the position of the imaging system 100 is a predetermined position according to the position information from its own GPS receiver. The switching unit 186 causes the demosaic processing unit 174 to generate image data for display based on the R image signal, the G image signal, and the B image signal input to the input reception unit 172, or inputs the input reception unit 172. Based on the R image signal, the G image signal, the B image signal, the RE image signal, and the NIR image signal, the recording processing unit 178 switches whether to generate image data for recording according to a predetermined recording format.

  In the imaging system 100 configured as described above, an interest for determining imaging conditions based on image signals captured by the R imaging device 110, the G imaging device 120, and the B imaging device 130, respectively. A region (ROI) may be determined. The image processor 180 determines the ROI based on the pixel value (luminance value) indicated by the image signals captured by each of the R image capturing apparatus 110, the G image capturing apparatus 120, and the B image capturing apparatus 130. There is.

  However, when the image processor 180 determines the respective imaging conditions of the R imaging device 110, the G imaging device 120, and the B imaging device 130 according to such ROI, the imaging conditions suitable for the specific object. It may not be. For example, when the specific object is a plant such as a crop and a car or a road is present around the plant, the image processor 180 may determine an area including the car or the road as the ROI. When the pixel value of the area including the car or the road is higher than the pixel value of the area including the plant, the image processor 180 determines the area including the car or the road as the ROI.

  For example, when an image 500 including a crop 510 and a car 512 as shown in FIG. 5 is captured by the image capturing system 100, the distribution of the luminance value (brightness) of the straight line portion 501 in the image 500 is as shown in FIG. The distribution is as shown in. That is, the brightness value of the car 512 becomes high, and the image processor 180 determines the region including the car 512 as the ROI. The image processor 180 determines such an area as the ROI and determines the exposure based on image information such as the brightness value within the ROI. In this case, the exposure may not be suitable for the crop 510, and the crop 510 in the image 500 may be dark, as shown in FIG.

  Therefore, in the present embodiment, the ROI is determined by an index according to the characteristics of the specific subject so that the imaging condition more suitable for the specific subject can be determined. The characteristic of a subject is a characteristic that allows the position where the subject is present to be estimated or specified from the image captured by the imaging device, and the reflectance, color, shape, and location of the subject for each wavelength of the subject. , Season, season, etc.

  The image processor 180 includes an area determination unit 181 and an imaging condition determination unit 182. The image capturing condition determining unit 182 determines whether or not a specific image is captured by at least one of the R image capturing apparatus 110, the G image capturing apparatus 120, the B image capturing apparatus 130, the RE image capturing apparatus 140, and the NIR image capturing apparatus 150. Based on the image in the wavelength region, the ROI is determined as the first region in the image. The imaging condition determination unit 182 corresponds to each of the first regions of the R imaging device 110, the G imaging device 120, the B imaging device 130, the RE imaging device 140, and the NIR imaging device 150. Based on the image information of the area, the imaging control values of the R imaging device 110, the G imaging device 120, the B imaging device 130, the RE imaging device 140, and the NIR imaging device 150 are determined.

  Here, the area corresponding to the first area is an area including the same subject as the subject included in the first area. For example, the second region of the R image pickup device 110 corresponding to the first region of the NIR image pickup device 150 is defined by the R image pickup device 110 including the same subject as the subject included in the first region of the NIR image pickup device 150. It is the area within the image that is captured. The third region of the G imaging device 120, the fourth region of the B imaging device 130, and the fifth region of the RE imaging device 140 corresponding to the first region of the NIR imaging device 150 are the NIR imaging device 150, This is an area in the images captured by the G image capturing apparatus 120, the B image capturing apparatus 130, and the RE image capturing apparatus 140 that includes the same subject as the subject included in the first region.

  The image information is information regarding an image captured by each image capturing device such as the R image capturing device 110, the G image capturing device 120, the B image capturing device 130, the RE image capturing device 140, or the NIR image capturing device 150, and the brightness. It includes information and color information. The imaging control value may be, for example, at least one of a brightness value, which is a control value when executing the automatic exposure control, a gain value of the image sensor, an aperture value (Iris value), and a shutter speed (accumulation time). The imaging control value may be a focus position (lens position) which is a control value when executing the autofocus control. Further, the imaging control value may be an R gain value and a B gain value which are control values when executing the automatic white balance control.

  The memory 192 stores information indicating the corresponding relationship between the coordinate systems of the images of the R image pickup device 110, the G image pickup device 120, the B image pickup device 130, the RE image pickup device 140, and the NIR image pickup device 150. Good. The imaging condition determination unit 182 may specify the second area, the third area, the fourth area, and the fifth area corresponding to the first area based on the information indicating the correspondence relationship.

  The parameters used to determine the ROI may be determined based on the light reflectance properties that are unique to a particular subject. For example, when the specific subject reflects a relatively large amount of light in the near-infrared region, the region determination unit 181 may determine the ROI based on the near-infrared reflectance.

  The region determination unit 181 may determine the first region in the NIR image as the ROI based on the NIR image of the near infrared region imaged by the NIR imaging device 150. The image capturing condition determining unit 182 determines the exposure control value of the R image capturing apparatus 110 based on the brightness information of the second area corresponding to the first area in the red image of the red area captured by the R image capturing apparatus 110. You can do it. The image capturing condition determining unit 182 determines the exposure control value of the G image capturing apparatus 120 based on the brightness information of the third area corresponding to the first area in the green image of the green area captured by the G image capturing apparatus 120. You can do it. The image capturing condition determining unit 182 determines the exposure control value of the B image capturing apparatus 130 based on the luminance information of the fourth area corresponding to the first area in the blue image of the blue area captured by the B image capturing apparatus 130. You can do it. The imaging condition determination unit 182 determines the exposure control value of the RE imaging device 140 based on the brightness information of the fifth region corresponding to the first region in the red edge image of the red edge region imaged by the RE imaging device 140. May be determined. The imaging condition determination unit 182 may determine the exposure control value of the NIR imaging device 150 based on the brightness information of the first region in the near-infrared image of the near-infrared region imaged by the NIR imaging device 150.

  The area determination unit 181 may determine the first area based on the near-infrared image and the red image. The area determination unit 181 may determine the first area based on the near-infrared image and the green image. The area determination unit 181 may determine the first area based on the near-infrared image and the red edge image.

  For example, when the specific subject is a plant, the standard vegetation index (NDVI) shows a unique value. NDVI shows a higher value as more plants are present. NDVI shows a higher value in plants with more leaves. NDVI shows a higher value in plants with higher activity. When the specific subject is a plant, NDVI indicates a value equal to or larger than a predetermined value. The number of leaves varies depending on the type of plant. The activity depends on the health of the plant. Therefore, when the specific subject is a specific plant, the standard vegetation index (NDVI) indicates a value within a predetermined range. The predetermined range may be determined based on the type of plant, the health condition of the plant to be observed, and the like.

Here, NDVI is represented by the following formula.

  IR represents the reflectance in the near infrared region, and R represents the reflectance in the visible region of red.

  The area determination unit 181 calculates NDVI based on the near-infrared image and the red image, and sets the first area in which NDVI is equal to or more than a predetermined value to the R imaging device 110, the G imaging device 120, and the B imaging device 130. , The RE imaging device 140 and the NIR imaging device 150 are determined as ROIs. The area determination unit 181 may divide each of the near infrared image and the red image into a plurality of blocks, and calculate the NDVI for each block based on the pixel value for each block. The area determination unit 181 may determine at least one block whose NDVI is equal to or greater than a predetermined threshold value as the ROI. The area determination unit 181 divides the image 500 into a plurality of blocks 600, for example, as illustrated in FIG. 7, and calculates NDVI for each block. The area determination unit 181 determines the block having the highest NDVI as the ROI 601, for example, as illustrated in FIG. 8. The image capturing condition determination unit 182 determines the exposure control value of the image capturing device based on the brightness information of each region of each image capturing device corresponding to the first region. In this way, the exposure control value of each imaging device is determined based on the luminance information in the ROI based on NDVI, so that each imaging device can capture an image with an exposure suitable for the crop in the ROI. Therefore, for example, as shown in FIG. 9, it is possible to obtain an image 530 in which the crop 532 is exposed. Note that when the exposure is suitable for the crop 532, a vehicle 534 brighter than the crop 532 may be overexposed. Further, when there is a subject extremely darker than the crop 532, the subject may be blackened in the image. However, since the purpose is to obtain an image focusing on the crop 532, it does not affect the evaluation of the image.

  Depending on the subject imaged by the imaging system 100, it may be preferable for the ROI to be determined by an index other than NDVI. Further, even for the same type of agricultural product, the reflectance characteristics may change depending on the place, time, or season. In this case, it may be better to switch the index for determining the ROI depending on the place, time, season, or the like. That is, the area determination unit 181 may select the index for determining the ROI based on at least one of the type of the subject and the place, the time, or the season when the subject is imaged.

For example, the region determination unit 181 may determine the ROI based on SAVI (Solid Adjusted Vegetation Index). SAVI is represented by the following formula.

  SAVI is an index that takes into consideration the difference in soil reflectance. It is different from NDVI in that it takes into account the difference in soil reflectance. L is 1 when the vegetation is small and 0.25 when the vegetation is large.

The region determination unit 181 may determine the ROI based on gNDVI (Green Normalized Difference Vegation Index). gNDVI is represented by the following formula.

  Here, G represents the reflectance of blue in the visible region.

The region determination unit 181 may determine the ROI based on NDRE (Normalized Difference Red edge Index). NDRE is represented by the following formula.

  Here, NIR represents the reflectance of near infrared rays, and RE represents the reflectance of red edges. By using NDRE, the vegetation distribution can be analyzed deeper. For example, the difference between cedar and cypress can be analyzed. The region determination unit 181 can determine the ROI by distinguishing, for example, the cedar region or the cypress region.

  The area determination unit 181 may select a near-infrared image and a red image or a near-infrared image and a green image as an image used for determining the ROI based on the characteristics of the subject. The area determination unit 181 may determine the ROI based on the selected near-infrared image and red image or near-infrared image and green image. The area determination unit 181 may select a near-infrared image and a red image, a near-infrared image and a green image, or a near-infrared image and a red edge image as an image used for determining the ROI, based on the characteristics of the subject. . The area determination unit 181 may determine the ROI based on the selected near-infrared image and red image, near-infrared image and green image, or near-infrared image and red edge.

  The area determination unit 181 may select NDVI, SAVI, gNDVI, or NDRE based on the characteristics of the subject, and determine the ROI based on the selected NDVI, SAVI, gNDVI, or NDRE. The area determination unit 181 may select NDVI, SAVI, gNDVI, or NDRE based on the characteristics of the plant that is the subject. The area determination unit 181 may select NDVI, SAVI, gNDVI, or NDRE based on at least one of the type of the subject, the place where the subject is imaged, the time, and the season.

  The area determination unit 181 may select NDVI, SAVI, gNDVI, or NDRE as an index for determining the ROI based on the type of subject specified by the user via the reception unit 184. The area determination unit 181 may select NDVI, SAVI, gNDVI, or NDRE as an index for determining the ROI based on the type of crop.

  FIG. 10 is a flowchart showing an example of the procedure for determining the exposure control value. The region determination unit 181 acquires an image necessary for calculating the index for determining the ROI (S100). The region determination unit 181 acquires, for example, the near-infrared image captured by the NIR imaging device 150 and the red image captured by the R imaging device 110 in order to derive the NVDI. The area determination unit 181 sets a plurality of blocks in the image (S102). The area determination unit 181 sets a plurality of blocks in a coordinate system common to each of the R imaging device 110, the G imaging device 120, the B imaging device 130, the RE imaging device 140, and the NIR imaging device 150. You can do it.

  The area determining unit 181 determines the near-infrared reflectance of each block derived from each block of the near-infrared image and the reflectance of each red area of each block derived from each block of the red image based on the reflectance of each block. The NDVI is derived (S104). The area determination unit 181 extracts at least one block indicating the NDVI corresponding to the plant from the NDVI of each of the plurality of blocks (S106). The area determination unit 181 may extract at least one block showing NDVI equal to or higher than a predetermined threshold. The area determination unit 181 may extract at least one block that is equal to or larger than a predetermined threshold and that shows the maximum NDVI. The area determination unit 181 may extract at least one block indicating NDVI that is equal to or higher than a predetermined lower limit threshold and equal to or lower than a predetermined upper limit threshold.

  When there is a block indicating NDVI corresponding to the plant, the area determination unit 181 sets the block indicating NDVI corresponding to the plant in the ROI (S108). On the other hand, when there is no block indicating NDVI corresponding to the plant, the area determination unit 181 sets all blocks to ROI (S110).

  The imaging condition determination unit 182 determines the exposure control value of each imaging device based on the image information of each ROI of each imaging device (S112). The imaging condition determination unit 182 determines the ROI in each image captured by the R imaging device 110, the G imaging device 120, the B imaging device 130, the RE imaging device 140, and the NIR imaging device 150. The exposure control values of the R image pickup device 110, the G image pickup device 120, the B image pickup device 130, the RE image pickup device 140, and the NIR image pickup device 150 are calculated based on the luminance information of the respective areas corresponding to You may decide.

  As described above, according to the imaging system 100 according to the present embodiment, the ROI to be referred to in order to determine the imaging condition can be determined according to the index according to the characteristics of the subject. Accordingly, the imaging system 100 can capture each image with the optimum exposure for the plant even when another subject such as a car having high brightness is present around the plant of interest.

  FIG. 11 is a diagram showing another example of the external appearance of the imaging system 100 mounted on the UAV 10. The imaging system 100 shown in FIG. 2 includes an RGB imaging device 160 in addition to the G imaging device 120, the B imaging device 130, the RE imaging device 140, and the NIR imaging device 150. Different from The RGB image pickup device 160 may be the same as a normal camera, and has an optical system and an image sensor. The image sensor has a filter that transmits light in the wavelength band of the red region, a filter that transmits light in the wavelength band of the green region, and a filter that transmits light in the wavelength band of the blue region, which are arranged in a Bayer array. You may The RGB image pickup device 160 may output an RGB image. The wavelength band in the red region may be, for example, 620 nm to 750 nm. The wavelength band in the green region may be, for example, 500 nm to 570 nm. The wavelength band in the blue region is, for example, 450 nm to 500 nm.

  The region determination unit 181 determines the first region in the near infrared image as the ROI based on the near infrared image captured by the NIR imaging device 150 and the red image captured by the R imaging device 110. You can do it. The area determination unit 181 derives the NDVI based on the near-infrared image captured by the NIR imaging device 150 and the red image captured by the R imaging device 110, and is determined in advance in the near-infrared image. The ROI may be determined as the first region exhibiting NDVI equal to or higher than the threshold. The image capturing condition determining unit 182 may determine the image capturing control value of the RGB image capturing device 160 based on the image information of the region corresponding to the first region in the RGB image captured by the RGB image capturing device 160. The image capturing condition determining unit 182 may determine the exposure control value of the RGB image capturing device 160 based on the luminance information of the region corresponding to the first region in the RGB image captured by the RGB image capturing device 160.

  FIG. 12 illustrates an example computer 1200 in which aspects of the invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with an apparatus according to an embodiment of the present invention or one or more “units” of the apparatus. Alternatively, the program can cause the computer 1200 to execute the operation or the one or more “units”. The program can cause the computer 1200 to execute a process according to the embodiment of the present invention or a stage of the process. Such programs may be executed by CPU 1212 to cause computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

  The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. Computer 1200 also includes a communication interface 1222, input / output units, which are connected to host controller 1210 via input / output controller 1220. Computer 1200 also includes ROM 1230. The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, and thereby controls each unit.

  The communication interface 1222 communicates with other electronic devices via the network. A hard disk drive may store programs and data used by CPU 1212 in computer 1200. The ROM 1230 stores therein a boot program or the like executed by the computer 1200 at the time of activation, and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, a USB memory or an IC card, or a network. The program is installed in the RAM 1214, which is also an example of a computer-readable recording medium, or the ROM 1230, and is executed by the CPU 1212. The information processing described in these programs is read by the computer 1200 and brings about the cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing the operation or processing of information according to the use of the computer 1200.

  For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes the communication program loaded in the RAM 1214, and performs the communication process on the communication interface 1222 based on the process described in the communication program. You may order. The communication interface 1222, under the control of the CPU 1212, reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The reception data received from the network is written in the reception buffer area or the like provided on the recording medium.

  Further, the CPU 1212 causes the RAM 1214 to read all or necessary portions of files or databases stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. Good. CPU 1212 may then write back the processed data to an external storage medium.

  Various types of information such as various types of programs, data, tables, and databases may be stored on the recording medium and processed. The CPU 1212 retrieves the data read from the RAM 1214 for various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information described elsewhere in this disclosure and specified by the instruction sequence of the program. Various types of processing may be performed, including / replacement, etc., and the result is written back to RAM 1214. Further, the CPU 1212 may search for information in files, databases, etc. in the recording medium. For example, when a plurality of entries each having the attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. That is, the entry that matches the condition is searched from the plurality of entries, the attribute value of the second attribute stored in the entry is read, and the entry is associated with the first attribute that satisfies the predetermined condition. The attribute value of the obtained second attribute may be acquired.

  The programs or software modules described above may be stored on a computer-readable storage medium on or near computer 1200. Further, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, whereby the program can be stored in the computer 1200 via the network. provide.

  The execution order of each process such as the operation, procedure, step, and step in the device, system, program, and method shown in the claims, the specification, and the drawings is "preceding" or "prior to". It should be noted that the output of the previous process can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the operation flow in the claims, the description, and the drawings is described using “first,” “next,” and the like for convenience, it means that it is essential to carry out in this order. Not a thing.

  Although the present invention has been described using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is apparent to those skilled in the art that various modifications and improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such modifications or improvements can be included in the technical scope of the present invention.

10 UAV
20 UAV main body 30 UAV control unit 32 Memory 36 Communication interface 40 Propulsion unit 41 GPS receiver 42 Inertial measurement device 43 Magnetic compass 44 Barometric altimeter 45 Temperature sensor 46 Humidity sensor 50 Gimbal 60 Imaging device 100 Imaging system 110 R imaging device 112 R Image sensor 114 optical system 120 G image pickup device 122 G image sensor 124 optical system 130 B image pickup device 132 B image sensor 134 optical system 140 RE image pickup device 142 RE image sensor 144 optical system 150 NIR image pickup device 152 NIR image sensor 154 Optical system 160 RGB image pickup device 170 Multiplex 172 Input reception unit 174 Demosaic processing unit 178 Recording processing unit 180 Image processing processor 181 Area determination unit 182 Imaging condition determination Unit 184 Reception unit 186 Switching unit 190 Transmission unit 192 Memory 300 Remote operation device 1200 Computer 1210 Host controller 1212 CPU
1214 RAM
1220 Input / output controller 1222 Communication interface 1230 ROM

Claims (16)

Determining a first region in the first image based on a first image in a first wavelength region imaged by a first image sensor included in the first imaging device,
An imaging control value of the second imaging device based on image information of a second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor included in the second imaging device. A determining device comprising a circuit configured to determine.   The determination device according to claim 1, wherein the image information is luminance information, and the imaging control value is an exposure control value.   The determination device according to claim 1, wherein the circuit determines the first region based on the first image and the second image. The circuit is
As the image used to determine the first region, the first image and the second image, or the third of the third wavelength region imaged by the third image sensor included in the first image and the third imaging device. Select an image,
Determining the first region based on the selected first image and the second image, or the first image and the third image,
The imaging control value of the first imaging device is determined based on the image information of the first region of the first image, and the second imaging device is determined based on the image information of the second region of the second image. 2. The determination according to claim 1, wherein the imaging control value of the third imaging device is determined based on image information of a third region corresponding to the first region of the third image. apparatus.   The determination device according to claim 4, wherein the circuit selects the first image and the second image or the first image and the third image based on a characteristic of a subject. The first wavelength region is a wavelength region in the near infrared region,
The second wavelength region is a wavelength region of the red region,
The determination device according to claim 4, wherein the third wavelength region is a wavelength region of a green region or a red edge region. The circuit is
As the image used to determine the first region, the third image in the third wavelength region captured by the first image and the second image, the first image, and the third image sensor included in the third imaging device. Or, selecting the fourth image in the fourth wavelength region captured by the fourth image sensor included in the first image and the fourth imaging device,
Determining the first area based on the selected first image and the second image, the first image and the third image, or the first image and the fourth image,
The imaging control value of the first imaging device is determined based on the image information of the first region of the first image, and the second imaging device is determined based on the image information of the second region of the second image. Image pickup control value of the third image, the image pickup control value of the third image pickup device is determined based on image information of a third region corresponding to the first region of the third image, The determination device according to claim 1, wherein the imaging control value of the fourth imaging device is determined based on image information of a fourth region corresponding to one region.   8. The circuit according to claim 7, wherein the circuit selects the first image and the second image, the first image and the third image, or the first image and the fourth image based on a characteristic of a subject. Decision device. The first wavelength region is a wavelength region in the near infrared region,
The second wavelength region is a wavelength region of the red region,
The third wavelength region is a wavelength region of a green region,
The determination device according to claim 7, wherein the fourth wavelength region is a wavelength region of a red edge region. Based on the first image in the first wavelength region captured by the first image sensor included in the first imaging device and the second image in the second wavelength region captured by the second image sensor included in the second imaging device, Determining a first region in the first image,
An imaging control value of the third imaging device based on image information of a third region corresponding to the first region in the third image of the third wavelength region captured by the third image sensor included in the third imaging device. A determining device comprising a circuit configured to determine.   The determination device according to claim 10, wherein the image information is luminance information, and the imaging control value is an exposure control value. The first wavelength region is a wavelength region in the near infrared region,
The second wavelength region is a wavelength region of a red region, a green region, or a red edge region,
The determination device according to claim 10, wherein the third wavelength region is a wavelength region of a red region, a green region, and a blue region. A determination device according to any one of claims 1 to 12,
The first imaging device;
An imaging system comprising the second imaging device.   A moving body comprising the imaging system according to claim 13 and moving. Determining a first region in the first image based on a first image in a first wavelength region captured by a first image sensor included in the first imaging device;
An imaging control value of the second imaging device based on image information of a second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor included in the second imaging device. And a step of determining.   A program for causing a computer to function as the determination device according to any one of claims 1 to 12.