WO2020192385A1 - Determination device, camera system, and moving object - Google Patents

Abstract

A multispectral camera capable of capturing images at multiple spectral bands sometimes cannot appropriately configure photographing conditions suitable for a specific object. Provided is a determination device comprising a circuit, the circuit being configured to: determine, according to a first image of a first wavelength region captured by a first image sensor included in a first camera device, a first region in the first image; and determine, according to image information of a second region in a second image of a second wavelength region captured by a second image sensor included in a second camera device, a camera control value of the second camera device, the second region corresponding to the first region.

Description

Confirmation device, camera system and moving body Technical field

The invention relates to a determining device, a camera system and a mobile body.

Background technique

Patent Document 1 discloses a sensing system that, when sensing an area containing an object to be inspected and a reference reflection area, senses the reference reflection area in each waveband as the sensing object of the object under inspection. It is measured that the reference reflection area has a reflectance corresponding to the object under inspection.

Background technical literature:

[Patent Literature]

[Patent Document 1] International Publication No. 2017/221756

Summary of the invention

The technical problem to be solved by the invention:

A multi-spectral camera that captures images in multiple wavelength bands may not be able to appropriately set imaging conditions suitable for a specific object.

Means used to solve technical problems:

The determining device according to an aspect of the present invention may include a circuit configured to determine the first image in the first image based on the first image in the first wavelength region captured by the first image sensor included in the first imaging device A region, and the imaging control of the second imaging device is determined based on the image information of the second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor of the second imaging device value.

The image information may be luminance information, and the imaging control value may be an exposure control value.

The circuit can determine the first area according to the first image and the second image.

The circuit may select the first image and the second image or the first image and the third image in the third wavelength region captured by the third image sensor of the third imaging device as the image for determining the first region. The circuit may determine the first area according to the selected first image and second image or first image and third image. The circuit can determine the camera control value of the first camera device according to the image information of the first area in the first image, determine the camera control value of the second camera device according to the image information of the second area in the second image, and according to the third image The image information of the third area corresponding to the first area is used to determine the imaging control value of the third imaging device.

The circuit can select the first image and the second image or the first image and the third image according to the characteristics of the object.

The first wavelength region may be a wavelength region of the near infrared region. The second wavelength region may be a wavelength region of the red region. The third wavelength region may be the wavelength region of the green region or the red edge region.

The circuit can select the first image and the second image, the first image and the third image in the third wavelength region captured by the third image sensor of the third imaging device, or the first image and the third image captured by the fourth imaging device. The fourth image in the fourth wavelength region captured by the fourth image sensor is used as an image for determining the first region. The circuit may determine the first area according to the selected first image and second image, first image and third image, or first image and fourth image. The circuit can determine the camera control value of the first camera device according to the image information of the first area in the first image, determine the camera control value of the second camera device according to the image information of the second area in the second image, and according to the third image The image information of the third area corresponding to the first area is used to determine the image capture control value of the third camera, and the image information of the fourth area corresponding to the first area in the fourth image is used to determine the image information of the fourth camera. Camera control value.

The circuit can select the first image and the second image, the first image and the third image, or the first image and the fourth image according to the characteristics of the subject.

The first wavelength region may be a wavelength region of the near infrared region. The second wavelength region may be a wavelength region of the red region. The third wavelength region may be a wavelength region of the green region. The fourth wavelength region may be the wavelength region of the red edge region.

The determining device according to an aspect of the present invention may include a circuit configured to be based on the first image in the first wavelength region captured by the first image sensor provided in the first imaging device and the second imaging device. The second image in the second wavelength region captured by the second image sensor is provided to determine the first region in the first image, and the third region in the third wavelength region captured by the third image sensor of the third image sensor is determined. The image information of the third area corresponding to the first area in the image determines the imaging control value of the third imaging device.

The image information may be luminance information, and the imaging control value may be an exposure control value.

The first wavelength region may be a wavelength region of the near infrared region. The second wavelength region may be a wavelength region of a red region, a green region, or a red edge region. The third wavelength region may be a wavelength region of a red region, a green region, and a blue region.

The camera system according to an aspect of the present invention may include the above-mentioned determining device. The camera system may include a first camera device and a second camera device.

The moving body according to an aspect of the present invention may be a moving body equipped with the aforementioned imaging system and moving.

The determining method according to an aspect of the present invention may include a stage of determining the first region in the first image based on the first image of the first wavelength region captured by the first image sensor included in the first imaging device. The determining method may include imaging of the second imaging device based on image information of the second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor of the second imaging device. The stage in which the control value is determined.

The program according to one aspect of the present invention may be a program for causing a computer to function as the above-mentioned determining device.

According to an aspect of the present invention, a multispectral camera that separately captures images in a plurality of wavelength bands can appropriately set shooting conditions suitable for a specific subject.

In addition, the above summary does not enumerate all the essential features of the present invention. In addition, sub-combinations of these feature groups may also constitute inventions.

Description of the drawings

FIG. 1 is a diagram showing an example of the appearance of an unmanned aerial vehicle (UAV) and a remote operation device.

Fig. 2 is a diagram showing an example of the appearance of an imaging system mounted on a UAV.

Fig. 3 is a diagram showing an example of functional blocks of UAV.

Fig. 4 is a diagram showing an example of functional blocks of the camera system.

Fig. 5 is a diagram showing an example of an image taken by an imaging system.

Fig. 6 is a luminance distribution diagram of a part of the image shown in Fig. 5.

Fig. 7 is a diagram for explaining blocks formed by dividing an image.

Fig. 8 is a diagram for explaining a region of interest in an image.

Fig. 9 is a diagram showing an example of an image taken by an imaging system.

Fig. 10 is a flowchart showing an example of a procedure for determining an exposure control value.

FIG. 11 is a diagram showing another example of the appearance of the imaging system mounted on the UAV.

Fig. 12 is a diagram for explaining an example of the hardware configuration.

Symbol Description:

10 UAV

20 UAV subject

30 UAV Control Department

32 Memory

36 Communication interface

40 Promotion Department

41 GPS receiver

42 Inertial measurement device

43 Magnetic compass

44 Barometric altimeter

45 Temperature sensor

46 Humidity sensor

50 Universal joint

60 Camera device

100 Camera system

110 Camera device for R

112 Image sensor for R

114 Optical system

120 Camera device for G

122 Image sensor for G

124 Optical system

130 Camera device for B

132 Image sensor for B

134 Optical system

140 Camera device for RE

142 Image sensor for RE

144 Optical system

150 NIR camera device

152 Image sensor for NIR

154 Optical system

160 RGB camera

170 Multiplexer

172 Input receiving department

174 Demosaicing Department

178 Record Processing Department

180 Image processor

181 Region Determination Department

182 Shooting Condition Determination Department

184 Receiving Department

186 Switching Department

190 Sending Department

192 Memory

300 remote operation device

1200 Computer

1210 Host Controller

1212 CPU

1214 RAM

1220 Input/Output Controller

1222 Communication interface

1230 ROM

detailed description

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, all the combinations of the features described in the embodiments are not necessarily necessary for the solution of the invention. It is obvious to a person skilled in the art that various changes or improvements can be made to the following embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

The claims, the description, the drawings of the description, and the abstract of the description include matters that are the subject of copyright protection. As long as anyone makes copies of these files as indicated in the patent office files or records, the copyright owner cannot object. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention can be described with reference to flowcharts and block diagrams. Here, a block may represent (1) a stage of a process of performing an operation or (2) a "part" of a device that performs an operation. Specific stages and "parts" can be implemented by programmable circuits and/or processors. Dedicated circuits may include digital and/or analog hardware circuits. May include integrated circuits (ICs) and/or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits can include logical AND, logical OR, logical exclusive OR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate array (FPGA), programmable logic array (PLA) ) And other memory components.

The computer-readable medium may include any tangible device that can store instructions for execution by a suitable device. As a result, the computer-readable medium on which instructions are stored is provided with a product including instructions that can be executed to create means for performing operations specified by the flowchart or block diagram. As an example of a computer-readable medium, it may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. As a more specific example of the computer-readable medium, it may include floppy (registered trademark) disk, floppy disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or Flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick , Integrated circuit cards, etc.

The computer-readable instructions may include any one of source code or object code described in any combination of one or more programming languages. The source code or object code includes traditional procedural programming languages. Traditional procedural programming languages can be assembly instructions, instruction set architecture (1SA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or Smalltalk (registered trademark), JAVA (registered trademark) , C++ and other object-oriented programming languages and "C" programming language or similar programming languages. The computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The processor or programmable circuit can execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and so on.

FIG. 1 shows an example of the appearance of an unmanned aerial vehicle (UAV) 10 and a remote operation device 300. The UAV 10 includes a UAV main body 20, a universal joint 50, a plurality of camera devices 60, and a camera system 100. The universal joint 50 and the camera system 100 are an example of a camera system. UAV10 is an example of a moving body. Moving objects include concepts such as flying objects moving in the air, vehicles moving on the ground, and ships moving on the water. Flying objects moving in the air refer to concepts that include not only UAVs, but also other aircraft, airships, and helicopters that move in the air.

The UAV main body 20 includes a plurality of rotors. Multiple rotors are an example of a propulsion section. The UAV main body 20 makes the UAV 10 fly by controlling the rotation of a plurality of rotors. The UAV main body 20 uses, for example, four rotors to fly the UAV 10. The number of rotors is not limited to four. In addition, UAV10 can also be a fixed-wing aircraft without rotors.

The imaging system 100 is a multispectral camera for imaging that captures objects within a desired imaging range in a plurality of wavelength bands. The universal joint 50 rotatably supports the camera system 100. The universal joint 50 is an example of a supporting mechanism. For example, the gimbal 50 uses an actuator to rotatably support the camera system 100 with a pitch axis. The universal joint 50 uses an actuator to further rotatably support the camera system 100 around the roll axis and the yaw axis, respectively. The gimbal 50 can change the posture of the camera system 100 by rotating the camera system 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that photograph the surroundings of the UAV 10 in order to control the flight of the UAV 10. The two camera devices 60 can be installed on the nose of the UAV 10, that is, on the front side. In addition, the other two camera devices 60 may be provided on the bottom surface of the UAV 10. The two imaging devices 60 on the front side may be paired to function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired to function as a stereo camera. The imaging device 60 can detect the existence of an object included in the imaging range of the imaging device 60 and measure the distance to the object. The imaging device 60 is an example of a measuring device that measures an object existing in the imaging direction of the imaging system 100. The measuring device may be another sensor such as an infrared sensor or an ultrasonic sensor that measures an object existing in the imaging direction of the imaging system 100. The three-dimensional spatial data around the UAV 10 can be generated based on the images taken by the plurality of camera devices 60. The number of imaging devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV10 may be equipped with at least one camera 60 on the nose, tail, side, bottom, and top of the UAV10. The viewing angle that can be set in the camera device 60 may be larger than the viewing angle that can be set in the camera system 100. The imaging device 60 may also include a single focus lens or a fisheye lens.

The remote operation device 300 communicates with the UAV 10 to remotely operate the UAV 10. The remote operation device 300 can wirelessly communicate with the UAV 10. The remote operation device 300 transmits to the UAV 10 instruction information indicating various commands related to the movement of the UAV 10 such as ascending, descending, accelerating, decelerating, forwarding, retreating, and rotating. The instruction information includes, for example, instruction information for raising the height of the UAV 10. The indication information may indicate the height at which the UAV10 should be located. The UAV 10 moves to be located at the height indicated by the instruction information received from the remote operation device 300. The instruction information may include an ascending instruction to raise the UAV10. UAV10 rises while receiving the rise command. When the height of UAV10 has reached the upper limit height, even if the ascending instruction is accepted, the ascent of UAV10 can be restricted.

FIG. 2 is a diagram showing an example of the appearance of the imaging system 100 mounted on the UAV 10. The imaging system 100 is a multispectral camera that separately captures image data of each of a plurality of preset wavebands. The imaging system 100 includes an imaging device 110 for R, an imaging device 120 for G, an imaging device 130 for B, an imaging device 140 for RE, and an imaging device 150 for NIR. The imaging system 100 can record each image data captured by the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR as a multispectral image. For example, multispectral images can be used to predict the health and vitality of crops.

FIG. 3 shows an example of the functional blocks of UAV10. UAV10 includes UAV control unit 30, memory 32, communication interface 36, propulsion unit 40, GPS receiver 41, inertial measurement device 42, magnetic compass 43, barometric altimeter 44, temperature sensor 45, humidity sensor 46, universal joint 50, camera The device 60 and the camera system 100.

The communication interface 36 communicates with other devices such as the remote operation device 300. The communication interface 36 can receive instruction information including various instructions to the UAV control unit 30 from the remote operation device 300. The memory 32 stores the UAV control unit 30 to the propulsion unit 40, GPS receiver 41, inertial measurement unit (IMU) 42, magnetic compass 43, barometric altimeter 44, temperature sensor 45, humidity sensor 46, universal joint 50, camera 60 and The imaging system 100 performs programs and the like necessary for control. The memory 32 may be a computer-readable recording medium, and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 32 may be provided inside the UAV main body 20. It can be configured to be detachable from the UAV main body 20.

The UAV control unit 30 controls the flight and shooting of the UAV 10 in accordance with the program stored in the memory 32. The UAV control unit 30 may be composed of a microprocessor such as a CPU or an MPU, and a microcontroller such as an MCU. The UAV control unit 30 controls the flight and shooting of the UAV 10 in accordance with instructions received from the remote operation device 300 via the communication interface 36. The propulsion unit 40 propels the UAV10. The propulsion part 40 includes a plurality of rotors and a plurality of drive motors that rotate the plurality of rotors. The propulsion unit 40 rotates a plurality of rotors via a plurality of drive motors in accordance with an instruction from the UAV control unit 30 to cause the UAV 10 to fly.

The GPS receiver 41 receives a plurality of signals indicating the time transmitted from a plurality of GPS satellites. The GPS receiver 41 calculates the position (latitude and longitude) of the GPS receiver 41, that is, the position (latitude and longitude) of the UAV 10 based on the received signals. The IMU42 detects the posture of the UAV10. The IMU 42 detects the acceleration of the UAV 10 in the three-axis directions of front and rear, left and right, and up and down, and the angular velocities of the pitch axis, the roll axis, and the yaw axis as the attitude of the UAV 10. The magnetic compass 43 detects the position of the nose of the UAV 10. The barometric altimeter 44 detects the flying altitude of the UAV10. The barometric altimeter 44 detects the air pressure around the UAV 10 and converts the detected air pressure to altitude to detect the altitude. The temperature sensor 45 detects the temperature around the UAV 10. The humidity sensor 46 detects the humidity around the UAV 10.

FIG. 4 shows an example of functional blocks of the camera system 100. The imaging system 100 includes an imaging device 110 for R, an imaging device 120 for G, an imaging device 130 for B, an imaging device 140 for RE, and an imaging device 150 for NIR. The imaging system 100 includes an image processor 180, a transmission unit 190, and a memory 192.

The imaging device 110 for R includes an image sensor 112 for R and an optical system 114. The image sensor 112 for R captures an image formed by the optical system 114. The R image sensor 112 includes a filter that transmits light in the red region, and outputs an R image signal that is an image signal in the red region. For example, the wavelength band of the red region is 620 nm to 750 nm. The wavelength band of the red region may be a specific wavelength band in the red region, for example, it may be 663 nm to 673 nm.

The imaging device 120 for G includes an image sensor 122 for G and an optical system 124. The image sensor 122 for G captures an image formed by the optical system 124. The G image sensor 122 includes a filter that transmits light in the green region, and outputs a G image signal that is an image signal in the green region. For example, the wavelength band of the green region is 500 nm to 570 nm. The wavelength band of the green region may be a specific wavelength band in the green region, for example, it may be 550 nm to 570 nm.

The imaging device 130 for B includes an image sensor 132 for B and an optical system 134. The image sensor 132 for B captures an image formed by the optical system 134. The image sensor for B 132 includes a filter that transmits light in the blue region, and outputs a B image signal that is an image signal in the blue region. For example, the wavelength band of the blue region is 450 nm to 500 nm. The wavelength band of the blue region may be a specific wavelength band in the blue region, for example, it may be 465 nm to 485 nm.

The imaging device 140 for RE includes an image sensor 142 for RE and an optical system 144. The image sensor 142 for RE captures an image formed by the optical system 144. The RE image sensor 142 includes a filter that transmits light in the red edge region, and outputs an RE image signal that is an image signal in the red edge region. For example, the wavelength band of the red edge region is 705 nm to 745 nm. The wavelength band of the red edge region may be 712 nm to 722 nm.

The NIR imaging device 150 includes an NIR image sensor 152 and an optical system 154. The image sensor 152 for NIR captures the image formed by the optical system 154. The image sensor for NIR 152 includes a filter that transmits light in the near infrared region, and outputs an image signal in the near infrared region, that is, an NIR image signal. For example, the wavelength band of the near infrared region is 800 nm to 2500 nm. The wavelength band of the near infrared region may be 800 nm to 900 nm.

The image processor 180 includes a multiplexer 170, an input receiving section 172, a demosaicing processing section 174, and a recording processing section 178. The image processor 180 is an example of a circuit. The image processor 180 may be composed of a microprocessor such as a CPU or an MPU, and a microcontroller such as an MCU.

The multiplexer 170 receives the image signal output from each image sensor, selects the image signal output from any image sensor according to a preset condition, and inputs it to the input receiving unit 172.

The demosaic processing unit 174 generates display image data based on the R image signal, the G image signal, and the B image signal input to the input receiving unit 172. The demosaic processing unit 174 generates display image data by performing demosaic processing on the R image signal, the G image signal, and the B image signal. The demosaic processing unit 174 can perform thinning processing on the R image signal, G image signal, and B image signal, and convert the thinning-processed R image signal, G image signal, and B image signal into Bayer array image signals to generate display signals. Image data. The transmitting unit 190 transmits the image data for display to the display device. For example, the transmitting unit 190 may transmit the image data for display to the remote operation device 300. The remote operation device 300 may display the image data for display on the display unit as an image of a live view.

The recording processing unit 178 generates recording image data based on the R image signal, G image signal, B image signal, RE image signal, and NIR image signal input to the input receiving unit 172 and based on a preset recording format. The recording processing unit 178 can generate RAW data as recording image data from the R image signal, the G image signal, the B image signal, the RE image signal, and the NIR image signal according to the RAW format. The recording processing unit 178 may generate all-pixel recording image data without performing thinning-out processing on the R image signal, G image signal, B image signal, RE image signal, and NIR image signal. The recording processing unit 178 may store the image data for recording in the memory 192. The memory 192 may be a computer-readable storage medium, and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 192 may be provided inside the housing of the camera system 100. The memory 192 may be configured to be detachable from the housing of the camera system 100.

The image processor 180 also includes a receiving unit 184 and a switching unit 186. The receiving unit 184 receives a storage instruction to store the image data for recording in the memory 192. The receiving unit 184 may receive a storage instruction from the user through an external terminal such as the remote operation device 300. When the position of the camera system 100 is a preset position, the receiving unit 184 may receive a storage instruction from the UAV control unit 30. When the position of the UAV 10 is the preset position, the UAV control unit 30 determines that the position of the camera system 100 is the preset position, so the receiving unit 184 can receive the storage instruction from the UAV control unit 30. The camera system 100 may also include a GPS receiver. In this case, the image processor 180 can determine whether the position of the camera system 100 is a preset position according to the position information from its own GPS receiver. The switching unit 186 switches between the following two methods. One is to generate display image data in the demosaicing processing unit 174 based on the R image signal, G image signal, and B image signal input to the input receiving unit 172, and the other is based on the input The R image signal, G image signal, B image signal, RE image signal, and NIR image signal sent to the input receiving unit 172 are generated in the recording processing unit 178 according to a preset recording format.

In the imaging system 100 configured as described above, it is possible to determine a region of interest (ROI) based on image signals captured by the imaging device 110 for R, the imaging device 120 for G, and the imaging device 130 for B, respectively. Determine the shooting conditions. The image processor 180 can determine the ROI based on the pixel values (luminance values) shown by the image signals captured by the R imaging device 110, the G imaging device 120, and the B imaging device 130, respectively.

However, when the image processor 180 determines the imaging conditions of each of the imaging device 110 for R, the imaging device 120 for G, and the imaging device 130 for B based on such an ROI, the imaging conditions may not be suitable for a specific object. For example, when the specific object is a plant such as a crop, and a vehicle or a road exists around the plant, the image processor 180 may determine an area including the vehicle or the road as an ROI. When the pixel value of the area containing the vehicle or road is higher than the pixel value of the area containing the plant, the image processor 180 will determine the area containing the vehicle or the road as the ROI.

For example, when the image 500 including the crop 510 and the vehicle 512 shown in FIG. 5 is captured by the imaging system 100, the luminance value (brightness) distribution of the straight line portion indicated by the symbol 501 in the image 500 is as shown in FIG. 6. That is, the brightness value of the vehicle 512 is high, and the image processor 180 will determine the area containing the vehicle 512 as an ROI. The image processor 180 will determine such a region as an ROI, and determine the exposure based on image information such as the brightness value in the ROI. In this case, as shown in FIG. 5, it is not suitable for the exposure of the crop 510, and the crop 510 in the image 500 is dark.

Therefore, in this embodiment, the ROI is determined by the index corresponding to the characteristic of the specific subject, so that the appropriate shooting conditions can be determined based on the specific subject. The characteristics of the subject refer to the characteristics that can estimate or determine the location of the subject based on the image taken by the imaging device, that is, the reflectance of the subject to each wavelength, the color, the shape, the shooting position of the subject, Time, season, etc.

The image processor 180 includes an area determination unit 181 and a shooting condition determination unit 182. The imaging condition determination unit 182 is based on an image in a specific wavelength region captured by at least one of the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR. The first area within is determined as the ROI. The imaging condition determination unit 182 is based on the image information of each region corresponding to each first region among the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR. The imaging control values of the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR are determined.

The area corresponding to the first area refers to an area containing the same subject as the subject included in the first area. For example, the second area of the R-use imaging device 110 corresponding to the first area of the NIR-use imaging device 150 means that it includes the same subject as the subject included in the first area of the NIR imaging device 150 and is defined by The area within the image captured by the imaging device 110 for R. The third area of the imaging device for G 120, the fourth area of the imaging device for B 130, and the fifth area of the imaging device for RE 140 corresponding to the first area of the imaging device for NIR 150 refer to the The image captured by the device 150, the imaging device for G 120, the imaging device for B 130, and the imaging device for RE 140 is an area of the same subject included in the first area.

Image information refers to image information taken by each imaging device such as the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, or the imaging device 150 for NIR, including luminance information and color information Wait. For example, the imaging control value may be at least one of a luminance value that is a control value when automatic exposure control is executed, a gain value of the image sensor, an aperture value (Iris value), and a shutter speed (accumulation time). The imaging control value may be a focus position (lens position) that is a control value when performing autofocus control. In addition, the imaging control value may be an R gain value and a B gain value that are control values when performing automatic white balance control.

The memory 192 can store information indicating the correspondence of the respective image coordinate systems of the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR. The imaging condition determining unit 182 may determine the second area, the third area, the fourth area, and the fifth area corresponding to the first area based on the information indicating the correspondence relationship.

The parameters used when determining the ROI can be determined according to the characteristics of the light reflectance unique to a specific subject. For example, when a specific subject reflects relatively much light in the near-infrared region, the region determining unit 181 may determine the ROI based on the near-infrared reflectance.

The region determining unit 181 may determine the first region in the NIR image as an ROI based on the NIR image of the near infrared region captured by the NIR imaging device 150. The imaging condition determination unit 182 may determine the exposure control value of the R imaging device 110 based on the luminance information of the second region corresponding to the first region in the red image of the red region captured by the R imaging device 110. The imaging condition determination unit 182 may determine the exposure control value of the G imaging device 120 based on the luminance information of the third region corresponding to the first region in the green image of the green region captured by the G imaging device 120. The imaging condition determination unit 182 may determine the exposure control value of the B imaging device 130 based on the luminance information of the fourth region corresponding to the first region in the blue image of the blue region captured by the B imaging device 130. The imaging condition determination unit 182 may determine the exposure control value of the RE imaging device 140 based on the luminance information of the fifth region corresponding to the first region in the red edge image of the red edge region captured by the RE imaging device 140. The imaging condition determining unit 182 may determine the exposure control value of the NIR imaging device 150 based on the luminance information of the first region in the near infrared image of the near infrared region captured by the NIR imaging device 150.

The area determining unit 181 may determine the first area based on the near infrared image and the red image. The area determining unit 181 may determine the first area based on the near infrared image and the green image. The area determining unit 181 may determine the first area based on the near infrared image and the red edge image.

For example, when the specific subject is a plant, the standard vegetation index (NDVI) represents a unique value. The more plants, the higher the value of NDVI. The more branches and leaves of the plant, the higher the value of NDVI. The higher the activity of the plant, the higher the value of NDVI. When the specific subject is a plant, the value of NDVI will be above the preset value. Depending on the type of plant, the number of branches and leaves will vary. Depending on the health of the plant, the degree of activity will vary. Therefore, when the specific subject is a specific plant, the standard vegetation index (NDVI) is a value within a preset range. The preset range can be determined according to the type of plant and the health status of the plant to be observed.

Among them, NDVI is represented by the following formula.

【Formula 1】

IR represents the reflectance in the near infrared region, and R represents the reflectance of red in the visible light region.

The area determining unit 181 calculates NDVI from the near-infrared image and the red image, and determines the first area whose NDVI is greater than or equal to the preset value as the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, and the imaging device 140 for RE And each ROI of the imaging device 150 for NIR. The area determination unit 181 divides the near-infrared image and the red image into a plurality of blocks, and can calculate the NDVI of each block based on the pixel value of each block. The region determining unit 181 may determine at least one block whose NDVI is above a preset threshold as an ROI. For example, as shown in FIG. 7, the area determination unit 181 divides the image 500 into a plurality of blocks 600 and calculates the NDVI of each block. For example, as shown in FIG. 8, the region determining unit 181 determines the block with the highest NDVI as the ROI601. The imaging condition determination unit 182 determines the exposure control value of the imaging device based on the luminance information of each region of each imaging device corresponding to the first region. In this way, the exposure control value of each imaging device is determined based on the luminance information in the ROI based on NDVI, so that each imaging device can capture images with exposure suitable for the crops in the ROI. Therefore, for example, as shown in FIG. 9, an image 530 suitable for exposure of the crop 532 can be obtained. In addition, when the exposure suitable for the crop 532 is performed, a vehicle 534 or the like that is brighter than the crop 532 may be overexposed. In addition, when there is a subject that is extremely darker than the crop 532, the subject may be underexposed in the image. However, since it is intended to obtain an image focusing on the crop 532, this does not affect the evaluation of the image.

Depending on the subject photographed by the imaging system 100, it may be preferable to determine the ROI by an index other than NDVI. In addition, for example, even if it is the same crop, the reflectance characteristics may change depending on the location, time, or season. In this case, depending on the location, time, season, etc., it is sometimes preferable to switch the index used to determine the ROI. That is, the region determining unit 181 may select an index for determining the ROI according to at least one of the type of the subject and the location, time period, or season where the subject is photographed.

For example, the region determining unit 181 may determine the ROI according to SAVI (Solid Adjusted Vegetation Index, soil adjusted vegetation index). SAVI is represented by the following formula.

[Formula 2]

SAVI is an index that considers the difference in soil reflectance. It is different from NDVI in considering the difference in soil reflectance. L is 1 when the vegetation is small, and 0.25 when it is large.

The area determining unit 181 may determine the ROI according to gNDVI (Green Normalized Difference Vegetation Index, green normalized vegetation index). gNDVI is represented by the following formula.

[Formula 3]

Among them, G represents the blue reflectance in the visible light region.

The region determining unit 181 may determine the ROI according to the NDRE (Normalized Difference Red Edge Index, normalized difference red edge index). NDRE is represented by the following formula.

[Formula 4]

Among them, NIR stands for near infrared reflectance, and RE stands for red edge reflectance. By using NDRE, a deeper analysis of vegetation distribution can be carried out. For example, the difference between cedar and juniper can be analyzed. For example, the area determination unit 181 can distinguish between the area of cedar or the area of cypress to determine the ROI.

The region determining unit 181 may select a near-infrared image and a red image or a near-infrared image and a green image as the image for determining the ROI according to the characteristics of the subject. The region determining unit 181 may determine the ROI based on the selected near-infrared image and red image or near-infrared image and green image. The region determining unit 181 may select a near-infrared image and a red image, a near-infrared image and a green image, or a near-infrared image and a red edge image as the image for determining the ROI according to the characteristics of the subject. The region determining unit 181 may determine the ROI based on the selected near-infrared image and red image, near-infrared image and green image, or near-infrared image and red edge.

The region determining unit 181 selects NDVI, SAVI, gNDVI, or NDRE according to the characteristics of the subject, and can determine the ROI according to the selected NDVI, SAVI, gNDVI, or NDRE. The area determination unit 181 may select NDVI, SAVI, gNDVI, or NDRE according to the characteristics of the plant as the subject. The area determining unit 181 may select NDVI, SAVI, gNDVI, or NDRE according to at least one of the type of the subject, the location where the subject was photographed, the time period, or the season.

The region determining unit 181 may select NDVI, SAVI, gNDVI, or NDRE as an index for determining the ROI according to the type of the subject specified by the user received through the receiving unit 184. The region determining unit 181 may select NDVI, SAVI, gNDVI, or NDRE as an index for determining the ROI according to the type of crop.

Fig. 10 is a flowchart showing an example of a procedure for determining an exposure control value. The area determination unit 181 acquires an image necessary for calculation of an index for determining the ROI (S100). For example, in order to derive NVDI, the area specifying unit 181 acquires a near-infrared image taken by the imaging device 150 for NIR and a red image taken by the imaging device 110 for R. The area specifying unit 181 sets a plurality of blocks in the image (S102). The area specifying unit 181 may set a plurality of blocks in a coordinate system common to each of the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR.

The area determination unit 181 derives the NDVI of each block based on the near-infrared reflectance of each block derived from each block of the near-infrared image and the red area reflectance of each block derived from each block of the red image (S104). The area determination unit 181 extracts at least one block representing the NDVI corresponding to the plant from the NDVI of each of the plurality of blocks (S106). The area determination section 181 may extract at least one block representing NDVI above a preset threshold. The area determining part 181 may extract at least one block representing the largest NDVI above a preset threshold. The area determination part 181 may extract at least one block representing NDVI that is above the preset lower limit threshold and below the preset upper threshold.

When the block indicating the NDVI corresponding to the plant exists, the area determination section 181 sets the block indicating the NDVI corresponding to the plant as an ROI (S108). On the other hand, when the block indicating the NDVI corresponding to the plant does not exist, the region specifying unit 181 sets all the blocks as ROI (S110).

The imaging condition determination unit 182 determines the exposure control value of each imaging device based on the image information of each ROI of each imaging device (S112). The imaging condition determination unit 182 can be based on each image corresponding to the ROI among the respective images captured by the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR. The luminance information of each region determines the exposure control values of the imaging device 110 for R, the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR.

As described above, according to the imaging system 100 according to the present embodiment, it is possible to determine the ROI based on the index corresponding to the characteristics of the subject, and the ROI must be referred to in order to determine the imaging conditions. As a result, for example, even when there are other subjects such as a vehicle with high brightness around the plant to be paid attention to, the imaging system 100 can perform the optimal exposure of the plant and capture the image separately.

FIG. 11 is a diagram showing another example of the appearance of the imaging system 100 mounted on the UAV 10. In addition to the imaging device 120 for G, the imaging device 130 for B, the imaging device 140 for RE, and the imaging device 150 for NIR, the imaging system 100 also includes an imaging device 160 for RGB, which is similar to the imaging system 100 shown in FIG. different. The RGB imaging device 160 may be the same as a normal camera and includes an optical system and an image sensor. The image sensor may include a filter that is arranged in a Bayer array and transmits light in the red region, a filter that transmits light in the green region, and a filter that transmits light in the blue region. The RGB imaging device 160 can output RGB images. For example, the wavelength band of the red region may be 620 nm to 750 nm. For example, the wavelength band of the green region may be 500 nm to 570 nm. For example, the wavelength band of the blue region is 450 nm to 500 nm.

The region determining unit 181 may determine the first region in the near-infrared image as an ROI based on the near-infrared image captured by the NIR imaging device 150 and the red image captured by the R imaging device 110. The region determining unit 181 may derive NDVI from the near-infrared image captured by the NIR camera 150 and the red image captured by the R camera 110, and determine the first region in the near-infrared image where the NDVI is above the preset threshold as an ROI . The imaging condition determination unit 182 may determine the imaging control value of the RGB imaging device 160 based on the image information of the region corresponding to the first region in the RGB image captured by the RGB imaging device 160. The imaging condition determination unit 182 may determine the exposure control value of the RGB imaging device 160 based on the luminance information of the region corresponding to the first region in the RGB image captured by the RGB imaging device 160.

FIG. 12 shows an example of a computer 1200 that may fully or partially embody various aspects of the present invention. The program installed on the computer 1200 can make the computer 1200 function as an operation associated with the device according to the embodiment of the present invention or one or more "parts" of the device. Alternatively, the program can cause the computer 1200 to perform the operation or the one or more "parts". This program enables the computer 1200 to execute the process or stages of the process involved in the embodiment of the present invention. Such a program may be executed by the CPU 1212, so that the computer 1200 executes specified operations associated with some or all blocks in the flowcharts and block diagrams described in this specification.

The computer 1200 according to the present embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other through a host controller 1210. The computer 1200 further includes a communication interface 1222, an input/output unit, which is connected to the host controller 1210 through the input/output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates in accordance with programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit.

The communication interface 1222 communicates with other electronic devices via a network. The hard disk drive can store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program executed by the computer 1200 during operation, and/or a program dependent on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as CR-ROM, USB memory, or IC card, or a network. The program is installed in RAM 1214 or ROM 1230 which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and causes cooperation between the programs and the various types of hardware resources described above. The operation or processing of information can be realized as the computer 1200 is used, thereby constituting an apparatus or method.

For example, when performing communication between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer provided in a recording medium such as RAM 1214 or USB memory, and sends the read transmission data to the network or receives it from the network The received data is written into the receiving buffer provided on the recording medium, etc.

In addition, the CPU 1212 can make the RAM 1214 read all or necessary parts of files or databases stored in an external recording medium such as a USB memory, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 can write the processed data back to the external recording medium.

Various types of information such as various types of programs, data, tables, and databases can be stored in a recording medium and subjected to information processing. For the data read from the RAM 1214, the CPU 1212 can perform various types of operations, information processing, conditional judgment, conditional transfer, unconditional transfer, and information retrieval/retrieval/information specified by the instruction sequence of the program described in various places in this disclosure. Replace various types of processing, and write the results back to RAM 1214. In addition, the CPU 1212 can search for information in files, databases, and the like in the recording medium. For example, when storing a plurality of entries having the attribute value of the first attribute respectively associated with the attribute value of the second attribute in the recording medium, the CPU 1212 may retrieve the attribute value of the specified first attribute from the multiple entries And read the attribute value of the second attribute stored in the entry to obtain the attribute value of the second attribute associated with the first attribute meeting the predetermined condition.

The above-mentioned programs or software modules may be stored on the computer 1200 or on a computer-readable storage medium near the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium so that the program can be provided to the computer 1200 via the network.

It should be noted that the execution order of the actions, sequences, steps, and stages in the devices, systems, programs and methods shown in the claims, description and drawings, as long as there is no special indication that "before..." , "In advance", etc., and can be implemented in any order as long as the output of the previous processing is not used in the subsequent processing. Regarding the operating procedures in the claims, the specification and the drawings, the descriptions are made using "first", "next", etc. for convenience, but it does not mean that it must be implemented in this order.

The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above-mentioned embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

Claims (16)

1. A determining device is characterized by comprising a circuit configured to determine the first image in the first image based on the first image in the first wavelength region captured by the first image sensor provided in the first imaging device. area,

   The imaging control of the second imaging device is determined based on the image information of the second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor of the second imaging device value.
2. The determining device according to claim 1, wherein the image information is luminance information, and the imaging control value is an exposure control value.
3. The determining device according to claim 1, wherein the circuit determines the first area based on the first image and the second image.
4. The determining device according to claim 1, wherein the circuit

   The first image and the second image or the first image and the third image in the third wavelength region captured by the third image sensor of the third imaging device are selected as the third image for determining the first The image of the area,

   Determining the first area according to the selected first image and the second image or the first image and the third image,

   Determine the imaging control value of the first imaging device according to the image information of the first area in the first image, and determine the second imaging device according to the image information of the second area in the second image The imaging control value of the device determines the imaging control value of the third imaging device according to the image information of the third region corresponding to the first region in the third image.
5. The determining device according to claim 4, wherein the circuit selects the first image and the second image or the first image and the third image according to the characteristics of the subject.
6. The determining device according to claim 4, wherein the first wavelength region is a wavelength region of a near infrared region,

   The second wavelength region is a wavelength region of the red region,

   The third wavelength region is the wavelength region of the green region or the red edge region.
7. The determining device according to claim 1, wherein the circuit

   Select the first image and the second image, the first image and the third image in the third wavelength region captured by the third image sensor of the third imaging device, or the first image and the third image The fourth image in the fourth wavelength region captured by the fourth image sensor included in the quad camera device is used as an image for identifying the first region,

   Determining the first area according to the selected first image and the second image, the first image and the third image, or the first image and the fourth image,

   Determine the imaging control value of the first imaging device according to the image information of the first area in the first image, and determine the second imaging device according to the image information of the second area in the second image The imaging control value of the device is determined according to the image information of the third area corresponding to the first area in the third image, and the imaging control value of the third imaging device is determined according to the information in the fourth image. The image information of the fourth area corresponding to the first area determines the imaging control value of the fourth imaging device.
8. The determining device according to claim 7, wherein the circuit selects the first image and the second image, the first image and the third image, or the The first image and the fourth image.
9. The determining device according to claim 7, wherein the first wavelength region is a wavelength region of a near infrared region,

   The second wavelength region is a wavelength region of the red region,

   The third wavelength region is a wavelength region of the green region,

   The fourth wavelength region is the wavelength region of the red edge region.
10. A determining device is characterized by comprising a circuit configured as follows: based on a first image in a first wavelength region captured by a first image sensor included in a first imaging device, and a second image captured by a second imaging device. A second image in the second wavelength region taken by the image sensor to determine the first region in the first image,

    The imaging control of the third imaging device is determined based on the image information of the third region corresponding to the first region in the third image of the third wavelength region captured by the third image sensor of the third imaging device value.
11. The determining device according to claim 10, wherein the image information is luminance information, and the imaging control value is an exposure control value.
12. The determining device according to claim 10, wherein the first wavelength region is a wavelength region of a near infrared region,

    The second wavelength region is a wavelength region of a red region, a green region or a red edge region,

    The third wavelength region is a wavelength region of a red region, a green region, and a blue region.
13. A camera system, characterized by comprising: the determining device according to any one of claims 1 to 12,

    The first camera and

    The second camera device.
14. A mobile body characterized by having the imaging system according to claim 13 and moving.
15. A determining method, characterized in that it comprises: a stage of determining a first region in the first image according to a first image in a first wavelength region captured by a first image sensor of a first imaging device; as well as

    The imaging control value of the second imaging device based on the image information of the second region corresponding to the first region in the second image of the second wavelength region captured by the second image sensor provided in the second imaging device Carry out a certain stage.
16. A program characterized by causing a computer to function as the determining device according to any one of claims 1 to 12.

Images