WO2018163300A1

WO2018163300A1 - Control device, imaging device, imaging system, moving body, control method, and program

Info

Publication number: WO2018163300A1
Application number: PCT/JP2017/009077
Authority: WO
Inventors: 園宏鄭; 本庄　謙一
Original assignee: エスゼットディージェイアイテクノロジーカンパニーリミテッド
Priority date: 2017-03-07
Filing date: 2017-03-07
Publication date: 2018-09-13
Also published as: JPWO2018163300A1; JP6547984B2

Abstract

This control device may be provided with a division unit that divides an image captured by an imaging device on the basis of the field angle of the imaging device, the altitude of the imaging device, and the imaging direction of the imaging device. The control device may be provided with a control unit that controls exposure of the imaging device for each of multiple regions.

Description

Control device, imaging device, imaging system, moving object, control method, and program

The present invention relates to a control device, an imaging device, an imaging system, a moving body, a control method, and a program.

Patent Document 1 discloses that a scene image of a camera is divided into a plurality of areas, and the luminance value of the entire scene is calculated based on the luminance value of each divided area.
Patent Document 1 Japanese Unexamined Patent Application Publication No. 2009-25727

Challenges to be solved

When the imaging device is used in an environment where the brightness changes relatively quickly, such as when the imaging device is mounted on a moving object such as an unmanned aircraft, automatic exposure control by the imaging device may not be performed properly. .

General disclosure

The control device according to one aspect of the present invention includes a dividing unit that divides an image captured by the imaging device into a plurality of regions based on an angle of view of the imaging device, an altitude of the imaging device, and an imaging direction of the imaging device. You may prepare. The control device may include a control unit that controls the exposure of the imaging device for each of a plurality of regions.

The control device may include a determination unit that determines an exposure control value of the imaging device for each of a plurality of regions. The control unit may control the exposure of the imaging device for each of a plurality of regions based on the exposure control value of the imaging device.

The control device may include a derivation unit that derives an evaluation value of brightness of an image captured by the imaging device for each of a plurality of regions. The determining unit may determine an exposure control value for each of the plurality of regions based on the evaluation values of the plurality of regions.

The dividing unit may divide the image into an upper region and a lower region based on the angle of view of the imaging device, the altitude of the imaging device, and the imaging direction of the imaging device. The deriving unit may derive an evaluation value of the brightness of each of the upper region and the lower region. The determination unit may determine the exposure control values of the upper region and the lower region based on the brightness evaluation values of the upper region and the lower region.

The dividing unit divides a first image captured by the imaging device at the first time point into a first upper region and a first lower region based on the angle of view, altitude, and imaging direction of the imaging device at the first time point. You can do it. The deriving unit may derive brightness evaluation values of the first upper region and the first lower region based on the first image. The dividing unit converts the second image captured at the second time point by the image pickup device based on the angle of view, the altitude, and the image pickup direction of the image pickup device at the second time point after the first time point. It may be divided into two lower regions. The determining unit determines an exposure control value for the second upper area based on the evaluation value for the first upper area, and determines an exposure control value for the second lower area based on the evaluation value for the first lower area. Good.

The angle of view, altitude, and imaging direction of the imaging device at the second time point may be the angle of view, altitude, and imaging direction of the imaging device at the second time point specified before the second time point.

The imaging device may be mounted on a moving body. The altitude of the imaging device at the second time point may correspond to the altitude of the moving body at the second time point specified before the second time point.

The dividing unit may divide the image into an upper region, a lower region, and at least one intermediate region between the upper region and the lower region. The determining unit may determine the exposure control value of the at least one intermediate region to a value between the exposure control value of the upper region and the exposure control value of the lower region.

The imaging device may have an image sensor. The control unit may control the exposure of the imaging device for each of a plurality of regions by controlling a gain of an electric signal output from the image sensor according to an exposure amount.

The imaging device may have an image sensor. The imaging device may include an optical filter provided in front of the image sensor and capable of changing the light transmittance for each of a plurality of predetermined regions. The control unit may control the exposure of the imaging device for each of the plurality of regions by controlling the light transmittance for each of the plurality of predetermined regions of the optical filter.

An imaging device according to one embodiment of the present invention may include the control device. The imaging device may include an image sensor.

An imaging system according to an aspect of the present invention may include the imaging device. The imaging system may include a support mechanism that supports the imaging device.

The moving body according to one embodiment of the present invention may move by mounting the imaging system.

A control method according to an aspect of the present invention includes a step of dividing an image captured by an imaging device into a plurality of regions based on an angle of view of the imaging device, an altitude of the imaging device, and an imaging direction of the imaging device. It's okay. The control method may include a step of controlling the exposure of the imaging device for each of a plurality of regions.

A program according to one embodiment of the present invention provides a computer with a step of dividing an image captured by an imaging device into a plurality of regions based on an angle of view of the imaging device, an altitude of the imaging device, and an imaging direction of the imaging device. May be executed. The program may cause the computer to execute a step of controlling the exposure of the imaging device for each of a plurality of regions.

According to one aspect of the present invention, exposure control by the imaging device can be more appropriately executed even when the imaging device is used in an environment where the brightness changes relatively quickly.

The above summary of the invention does not enumerate all the features of the present invention. A sub-combination of these feature groups can also be an invention.

It is a figure which shows an example of the external appearance of an unmanned aerial vehicle (UAV) and a remote control device. It is a figure which shows an example of the functional block of UAV. It is a figure for demonstrating the upper area | region and lower area | region of an image. It is a figure for demonstrating the angle of view of an imaging device. It is a figure for demonstrating the angle range where a horizontal line is included in an image. It is a figure for demonstrating the angle range where a horizontal line is included in an image. It is a flowchart which shows an example of the procedure of the division | segmentation of an image. It is a figure for demonstrating each parameter used in order to estimate the position of the horizon contained in an image, when an imaging device is facing the horizontal direction. It is a figure for demonstrating each parameter used in order to estimate the position of the horizontal line contained in an image, when an imaging device inclines only the angle (alpha) upwards with respect to the horizontal direction. It is a figure for demonstrating the upper area | region and lower area | region of an image. It is a figure which shows the relationship between the ratio of the downward area | region with respect to the whole image, and flight altitude. It is a figure which shows the relationship between the ratio of the downward area | region with respect to the whole image, and attitude | position angle (alpha). It is a flowchart which shows an example of the procedure of exposure control by an imaging device. It is a figure which shows another example of the functional block of UAV. It is a figure for demonstrating an optical filter. It is a figure which shows an example of a hardware constitutions.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the claimed invention. Moreover, not all the combinations of features described in the embodiments are essential for the solution means of the invention. It will be apparent to those skilled in the art that various modifications or improvements can be made to the following embodiments. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

The claims, the description, the drawings, and the abstract include matters that are subject to copyright protection. The copyright owner will not object to any number of copies of these documents as they appear in the JPO file or record. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where a block is either (1) a stage in a process in which an operation is performed or (2) an apparatus responsible for performing the operation. May represent a “part”. Certain stages and “units” may be implemented by programmable circuits and / or processors. Dedicated circuitry may include digital and / or analog hardware circuitry. Integrated circuits (ICs) and / or discrete circuits may be included. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. The memory element or the like may be included.

The computer readable medium may include any tangible device capable of storing instructions to be executed by a suitable device. As a result, a computer readable medium having instructions stored thereon comprises a product that includes instructions that can be executed to create a means for performing the operations specified in the flowcharts or block diagrams. Examples of computer readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer readable media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), Electrically erasable programmable read only memory (EEPROM), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated A circuit card or the like may be included.

The computer readable instructions may include either source code or object code written in any combination of one or more programming languages. The source code or object code includes a conventional procedural programming language. Conventional procedural programming languages include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or Smalltalk, JAVA, C ++, etc. It may be an object-oriented programming language and a “C” programming language or a similar programming language. Computer readable instructions may be directed to a general purpose computer, special purpose computer, or other programmable data processing device processor or programmable circuit locally or in a wide area network (WAN) such as a local area network (LAN), the Internet, etc. ). The processor or programmable circuit may execute computer readable instructions to create a means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

FIG. 1 shows an example of the external appearance of an unmanned aerial vehicle (UAV) 10 and a remote control device 300. The UAV 10 includes a UAV main body 20, a gimbal 50, a plurality of imaging devices 60, and an imaging device 100. The gimbal 50 and the imaging device 100 are an example of an imaging system. The UAV 10 is an example of a moving body propelled by a propulsion unit. The moving body is a concept including a flying body such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, etc. in addition to the UAV.

The UAV main body 20 includes a plurality of rotor blades. The plurality of rotor blades is an example of a propulsion unit. The UAV main body 20 causes the UAV 10 to fly by controlling the rotation of a plurality of rotor blades. For example, the UAV main body 20 causes the UAV 10 to fly using four rotary wings. The number of rotor blades is not limited to four. The UAV 10 may be a fixed wing machine that does not have a rotating wing.

The imaging apparatus 100 is an imaging camera that images a subject included in a desired imaging range. The gimbal 50 supports the imaging device 100 in a rotatable manner. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 supports the imaging device 100 so as to be rotatable about the pitch axis using an actuator. The gimbal 50 further supports the imaging device 100 using an actuator so as to be rotatable about the roll axis and the yaw axis. The gimbal 50 may change the posture of the imaging device 100 by rotating the imaging device 100 about at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two imaging devices 60 may be provided in the front which is the nose of UAV10. Two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two imaging devices 60 on the front side may be paired and function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired and function as a stereo camera. Based on images picked up by a plurality of image pickup devices 60, three-dimensional spatial data around the UAV 10 may be generated. The number of imaging devices 60 included in the UAV 10 is not limited to four. The UAV 10 only needs to include at least one imaging device 60. The UAV 10 may include at least one imaging device 60 on each of the nose, the tail, the side surface, the bottom surface, and the ceiling surface of the UAV 10. The angle of view that can be set by the imaging device 60 may be wider than the angle of view that can be set by the imaging device 100. The imaging device 60 may have a single focus lens or a fisheye lens.

The remote operation device 300 communicates with the UAV 10 to remotely operate the UAV 10. The remote operation device 300 may communicate with the UAV 10 wirelessly. The remote control device 300 transmits to the UAV 10 instruction information indicating various commands related to movement of the UAV 10 such as ascending, descending, accelerating, decelerating, moving forward, moving backward, and rotating. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote operation device 300. The instruction information may include an ascending command that raises the UAV 10. The UAV 10 rises while accepting the ascent command. Even if the UAV 10 receives the ascending command, the UAV 10 may limit the ascent when the altitude of the UAV 10 has reached the upper limit altitude.

FIG. 2 shows an example of functional blocks of the UAV10. The UAV 10 includes a UAV control unit 30, a memory 32, a communication interface 34, a propulsion unit 40, a GPS receiver 41, an inertial measurement device 42, a magnetic compass 43, a barometric altimeter 44, a gimbal 50, and the imaging device 100.

The communication interface 34 communicates with other devices such as the remote operation device 300. The communication interface 34 may receive instruction information including various commands for the UAV control unit 30 from the remote operation device 300. In the memory 32, the UAV control unit 30 controls the propulsion unit 40, the GPS receiver 41, the inertial measurement device (IMU) 42, the magnetic compass 43, the barometric altimeter 44, the gimbal 50, the imaging device 60, and the imaging device 100. Stores necessary programs etc. The memory 32 may be a computer-readable recording medium and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 32 may be provided inside the UAV main body 20. It may be provided so as to be removable from the UAV main body 20.

The UAV control unit 30 controls the flight and imaging of the UAV 10 according to a program stored in the memory 32. The UAV control unit 30 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The UAV control unit 30 controls the flight and imaging of the UAV 10 according to a command received from the remote control device 300 via the communication interface 34. The propulsion unit 40 propels the UAV 10. The propulsion unit 40 includes a plurality of rotating blades and a plurality of drive motors that rotate the plurality of rotating blades. The propulsion unit 40 causes the UAV 10 to fly by rotating a plurality of rotor blades via a plurality of drive motors in accordance with a command from the UAV control unit 30.

The GPS receiver 41 receives a plurality of signals indicating times transmitted from a plurality of GPS satellites. The GPS receiver 41 calculates the position of the GPS receiver 41, that is, the position of the UAV 10 based on the received signals. The IMU 42 detects the posture of the UAV 10. The IMU 42 detects, as the posture of the UAV 10, acceleration in the three axial directions of the front, rear, left, and right of the UAV 10, and angular velocity in the three axial directions of pitch, roll, and yaw. The magnetic compass 43 detects the heading of the UAV 10. The barometric altimeter 44 detects the altitude at which the UAV 10 flies. The barometric altimeter 44 detects the atmospheric pressure around the UAV 10, converts the detected atmospheric pressure into an altitude, and detects the altitude.

The imaging apparatus 100 includes an imaging unit 102 and a lens unit 200. The lens unit 200 is an example of a lens device. The imaging unit 102 includes an image sensor 120, an imaging control unit 110, and a memory 130. The image sensor 120 may be configured by a CCD or a CMOS. The image sensor 120 outputs image data of an optical image formed through the plurality of lenses 210 to the imaging control unit 110. The imaging control unit 110 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The imaging control unit 110 may control the imaging device 100 in accordance with an operation command for the imaging device 100 from the UAV control unit 30. The memory 130 may be a computer-readable recording medium and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 130 stores a program and the like necessary for the imaging control unit 110 to control the image sensor 120 and the like. The memory 130 may be provided inside the housing of the imaging device 100. The memory 130 may be provided so as to be removable from the housing of the imaging apparatus 100.

The lens unit 200 includes a plurality of lenses 210, a lens moving mechanism 212, and a lens control unit 220. The plurality of lenses 210 may function as a zoom lens, a varifocal lens, and a focus lens. At least some or all of the plurality of lenses 210 are arranged to be movable along the optical axis. The lens unit 200 may be an interchangeable lens that is detachably attached to the imaging unit 102. The lens moving mechanism 212 moves at least some or all of the plurality of lenses 210 along the optical axis. The lens control unit 220 drives the lens moving mechanism 212 in accordance with a lens control command from the imaging unit 102 to move one or a plurality of lenses 210 along the optical axis direction. The lens control command is, for example, a zoom control command and a focus control command.

The imaging device 100 mounted on the UAV 10 configured as described above may capture an image including the ground or the sea surface and the sky. Due to the movement of the UAV 10 or the attitude control of the imaging device 100 by the gimbal 50, the ratio of the ground, the sea surface, and the sky in the screen greatly changes. The automatic exposure control by the imaging apparatus 100 cannot follow this change, and the image captured by the imaging apparatus 100 may be temporarily too bright or too dark. Therefore, in the present embodiment, a horizontal line that is a boundary between the ground or the sea surface and the sky is estimated based on the altitude, the angle of view, and the shooting direction of the imaging apparatus 100. An image captured by the imaging apparatus 100 is divided into an upper area above the horizontal line and a lower area below the horizontal line. Then, an evaluation value of the brightness of the image is derived for each of the upper region and the lower region, and an exposure control value for the upper region and the lower region is determined based on each evaluation value. Thereby, the imaging device 100 appropriately performs exposure control.

In order to realize the above exposure control, the imaging control unit 110 includes a dividing unit 112, a derivation unit 114, a determination unit 116, and an exposure control unit 118. The dividing unit 112 divides an image captured by the imaging apparatus 100 into a plurality of regions based on the angle of view of the imaging apparatus 100, the altitude of the imaging apparatus 100, and the imaging direction of the imaging apparatus 100. The dividing unit 112 may divide the image into an upper area and a lower area based on the angle of view of the imaging apparatus 100, the altitude of the imaging apparatus 100, and the imaging direction of the imaging apparatus 100. The dividing unit 112 estimates the position of the horizontal line in the image based on the angle of view of the imaging device 100, the altitude of the imaging device 100, and the imaging direction of the imaging device 100. The dividing unit 112 may divide the image with the upper area above the estimated horizontal line as the upper area and the lower area below the estimated horizontal line as the lower area. The upper region and the lower region in the image differ depending on the posture of the imaging apparatus 100 at the time when the image is captured. For example, when the posture of the imaging apparatus 100 is in a horizontal state, the upper region may be a region on the upper side in the vertical direction in the image, and the lower region may be a region on the lower side in the vertical direction in the image. The upper region may be a region on the sky side in the image including the ground or sea surface and the sky. The lower region may be a region on the ground or sea surface side in the image including the ground or sea surface and the sky.

The deriving unit 114 derives an evaluation value of the brightness of an image captured by the imaging device 100 for each of a plurality of areas. The deriving unit 114 may derive the luminance value of the image captured by the imaging apparatus 100 as a brightness evaluation value for each of a plurality of regions. The deriving unit 114 may derive the luminance value of each region based on the luminance value of each pixel included in each region. The deriving unit 114 may derive the average value of the luminance values of the pixels included in each area as the luminance value of each area. The deriving unit 114 performs predetermined weighting on the luminance value of each pixel included in each region, and derives the luminance value of each region based on each weighted luminance value. Good. The deriving unit 114 may derive the brightness evaluation values of the upper region and the lower region divided by the dividing unit 112.

The determining unit 116 determines the exposure control value of the imaging apparatus 100 for each of a plurality of regions. The determination unit 116 may determine an exposure control value for each of the plurality of regions based on the evaluation values of the plurality of regions. The determination unit 116 may determine an exposure control value for each of the plurality of regions based on the luminance values of the plurality of regions. The determination unit 116 may determine an exposure value (EV value) for each of the plurality of regions as the control value for each of the plurality of regions. The determination unit 116 may determine the exposure control values of the upper region and the lower region based on the brightness evaluation values of the upper region and the lower region.

The exposure control unit 118 controls the exposure of the imaging apparatus 100 for each of a plurality of areas. The exposure control unit 118 may control the exposure of the imaging device 100 for each of the plurality of regions based on the exposure control values of the plurality of regions. The exposure control unit 118 is an example of a control unit. The exposure control unit 118 may control the exposure of the imaging device 100 for each of a plurality of regions by controlling the gain of an electric signal output by the image sensor 120 according to the exposure amount. The exposure control unit 118 controls the exposure of the imaging device 100 for each of the plurality of regions by controlling the gain of the image sensor 120 for each of the plurality of regions based on the exposure control values of the plurality of regions. Good. The exposure control unit 118 may control the exposure of the imaging device 100 for each of the plurality of regions by controlling the exposure time for each of the plurality of regions.

Based on the angle of view, altitude, and imaging direction of the imaging device 100 at the first time point, the dividing unit 112 converts the first image captured by the imaging device 100 at the first time point into the first upper region and the first lower side. It may be divided into regions. The deriving unit 114 may derive brightness evaluation values of the first upper area and the first lower area based on the first image. Based on the angle of view, altitude, and imaging direction of the imaging device 100 at the second time point after the first time point, the dividing unit 112 outputs the second image captured at the second time point by the imaging device 100 in the second upper direction. You may divide | segment into a area | region and a 2nd lower area | region. The angle of view, altitude, and imaging direction of the imaging device 100 at the second time point may be the angle of view, altitude, and imaging direction of the imaging device 100 at the second time point specified before the second time point. The angle of view and imaging direction of the imaging device 100 at the second time point may correspond to the target angle of view and imaging direction to be set in the imaging device 100 at the second time point. The altitude of the imaging device 100 at the second time point may correspond to the target altitude at which the imaging device 100 should be located at the second time point.

The altitude of the imaging device 100 at the second time point may correspond to the altitude of the UAV 10 at the second time point specified before the second time point. While receiving the ascending command from the remote operation device 300, the UAV 10 may ascend at a speed corresponding to the ascending command. In this case, the dividing unit 112 may specify the current rising speed of the UAV 10 based on the rising command. Then, the dividing unit 112 may specify the altitude of the UAV 10 at the second time point after the current time before the second time point based on the current ascending speed and the current altitude. The altitude of the imaging device 100 at the second time point may correspond to the target altitude at which the UAV 10 should be located at the second time point. The target altitude may correspond to the altitude at which the UAV 10 indicated in the instruction information for the UAV 10 should be located. The target altitude may correspond to the altitude at which the UAV 10 should be located after a predetermined time has elapsed since the present time. The target angle of view and imaging direction may be the angle of view and imaging direction to be set when the imaging apparatus 100 captures images at a height at which the UAV 10 should be located after a predetermined time has elapsed since the present time. The imaging direction of the imaging apparatus 100 may be determined based on a control value for the gimbal 50 to control the attitude of the imaging apparatus 100 and a control value for the UAV control unit 30 to control the attitude of the UAV 10. The determination unit 116 may determine a control value for the exposure of the second upper region at the second time point based on the evaluation value of the first upper region at the first time point. The determination unit 116 may determine a control value for the exposure of the second lower region at the second time point based on the evaluation value of the first lower region at the first time point.

The dividing unit 112 may divide the image into an upper region, a lower region, and at least one intermediate region between the upper region and the lower region. The determination unit 116 may determine the exposure control value of at least one intermediate region as a value between the exposure control value of the upper region and the exposure control value of the lower region. When the exposure control value is greatly different between the upper region and the lower region, linear noise may appear near the boundary between the upper region and the lower region of the image captured by the imaging device 100. Therefore, an intermediate region is provided between the upper region and the lower region so that the change in the exposure control value becomes gradual. Thereby, linear noise can be prevented from appearing in the image.

As illustrated in FIG. 3, when an image 500 captured by the imaging apparatus 100 includes a horizontal line, the determination unit 116 emptyes the upper area 502 above the horizontal line and sets the lower area 504 below the horizontal line to the ground or the sea surface. Judge. Then, the determination unit 116 may determine the exposure values of the upper region 502 and the lower region 504 so that the exposure value of the upper region 502 is higher than the exposure value of the lower region 503.

Here, an example of a procedure in which the dividing unit 112 divides an image captured by the imaging device 100 into an upper region and a lower region will be described more specifically.

As shown in FIG. 4, the angle of view of the imaging device 100 determined by the focal length f of the imaging device 100 and the size (width) S of the image sensor 120 is θ. As shown in FIG. 5, let R be the radius of the earth. Let the altitude of UAV10 be h. An angle formed between a virtual line 404 connecting the position of the horizontal line visible from the UAV 10 at the altitude h and the UAV 10 and a virtual line 406 extending from the UAV 10 at the altitude h in the vertical direction is γ. When the UAV 10 exists at such a position, the angle range in which the horizontal line is included in the image captured by the imaging apparatus 100 is from γ−θ / 2 to γ + θ / with respect to the virtual line 406 as shown in FIG. 2 range. When the condition that the angle formed by the optical axis of the imaging apparatus 100 and the virtual line 406 is included in the range of γ−θ / 2 to γ + θ / 2, a horizontal line is included in the image captured by the imaging apparatus 100. . The exposure control unit 118 may control the exposure for each region divided by the dividing unit 112 when the posture of the imaging apparatus 100 satisfies the condition. For example, when the imaging control unit 110 detects an obstacle based on an image captured by the imaging device 60, the exposure control unit 118 does not have to execute exposure control for each of the areas divided by the dividing unit 112. Also good.

FIG. 7 is a flowchart illustrating an example of an image division procedure executed by the dividing unit 112. FIG. 8 is a diagram for describing each parameter used for estimating the position of the horizontal line included in the image when the imaging apparatus 100 is oriented in the horizontal direction. FIG. 9 is a diagram for describing each parameter used for estimating the position of the horizontal line included in the image when the imaging apparatus 100 is inclined upward by an angle α with respect to the horizontal direction.

The dividing unit 112 acquires the flight altitude h of the UAV 10 (S100). The dividing unit 112 derives a distance d from the UAV 10 to the horizontal line (S102). The distance d is derived from the radius R of the earth and the distance d by the following equation.

The dividing unit 112 derives a horizontal distance D indicating the distance between the UAV 10 and the intersection 414 of the virtual line 410 extending in the horizontal direction from the UAV 10 and the virtual line 412 passing through the horizontal line 420 and perpendicular to the virtual line 410 ( S104). The dividing unit 112 derives a horizontal angle β that is an angle formed by the virtual line 410 and the virtual line 416 along the distance d (S106). The horizontal distance D is derived from the following equation.

The horizontal angle β is derived from the following equation.

Subsequently, the dividing unit 112 derives the projection range W when the imaging apparatus 100 captures an image with the angle of view θ (S108), and derives the height V from the lower portion 418 of the projection range W to the horizontal line 420.

As shown in FIG. 8, when the imaging apparatus 100 is oriented in the horizontal direction, the projection range W is derived by the following equation.

In this case, the height V from the lower portion 418 of the projection range W to the horizontal line 420 is derived by the following equation.

On the other hand, as shown in FIG. 9, when the imaging apparatus 100 is inclined by an angle (attitude angle) α with respect to the horizontal direction, the projection range W is derived by the following equation. When the imaging apparatus 100 is oriented upward with respect to the horizontal direction, the posture angle α is a positive value.

The dividing unit 112 identifies the upper region and the lower region based on the derived height V (S112 and S114). For example, as illustrated in FIG. 10, when the number of lines of an image 510 captured by the image sensor 120 is M [pixel], the dividing unit 112 defines the upper region as M × (W on the upper side in the vertical direction of the image. Set to -V) / W [pixel]. The dividing unit 112 sets the lower area to M × V / M [pixel] on the lower side in the vertical direction of the image.

The determination unit 116 determines the exposure value of the upper region to be, for example, 10 [EV] based on the luminance value of the upper region. The determination unit 116 determines the exposure value of the lower region to be, for example, 6 [EV] based on the luminance value of the lower region.

As described above, according to the above procedure, the dividing unit 112 estimates the position of the horizontal line in the image based on the angle of view, the altitude, and the imaging direction of the image capturing apparatus 100, and the upper region above the estimated horizontal line is the lower region. It is possible to divide the image using as a lower area.

FIG. 11 shows the relationship between the ratio of the lower area to the entire image and the flight altitude h. Even if the flight altitude h is the same, the longer the focal length f of the imaging apparatus 100, that is, the smaller the angle of view θ, the lower the proportion of the lower region.

FIG. 12 shows the relationship between the ratio of the lower area to the entire image and the posture angle α. Even if the posture angle α is the same, the longer the focal length f of the imaging apparatus 100, that is, the narrower the angle of view θ, the smaller the proportion of the lower region.

FIG. 13 is a flowchart showing an example of a procedure for exposure control by the imaging apparatus 100. First, the imaging apparatus 100 is turned on by turning on the main power supply of the UAV 10 (S300). The UAV 10 does not fly immediately after the imaging apparatus 100 is turned on. The exposure control unit 118 controls the exposure of the imaging apparatus 100 with the entire image as a single region as usual without dividing the image (S302). Thereafter, the imaging control unit 110 determines whether or not the UAV 10 has received flight instruction information (S304).

Upon receiving the flight instruction information, the UAV 10 starts flying (S306). The dividing unit 112 specifies the current altitude of the UAV 10, the angle of view of the imaging device 100, and the imaging direction of the imaging device 100 (S308). Based on the current altitude of the UAV 10, the angle of view of the imaging device 100, and the imaging direction of the imaging device 100, the dividing unit 112 identifies the current upper region and lower region of the image captured by the imaging device 100 (S310). ). The deriving unit 114 derives, for example, a luminance value as the brightness evaluation value of each of the current upper region and lower region. The determination unit 116 determines the current upper region exposure value and the current lower region exposure value based on the current upper region luminance value and the current lower region luminance value (S312).

The exposure control unit 118 controls the exposure of the imaging device 100 for each of the upper region and the lower region based on the determined exposure value of the upper region and the current exposure value of the lower region (S314).

Next, the dividing unit 112 specifies the height of the UAV 10 at the next time point, the angle of view of the imaging device 100, and the imaging direction of the imaging device 100 based on the instruction information (S316). Based on the instruction information, the dividing unit 112 may specify a target altitude at which the UAV 10 should be located after a predetermined period from the current time, and specify the target altitude as the UAV 10 altitude at the next time point. . The dividing unit 112 may specify the current rising speed of the UAV 10 based on the instruction information. Then, the dividing unit 112 identifies the target altitude at which the UAV 10 should be positioned after a predetermined period from the current time based on the current ascending speed and the current altitude, and sets the target altitude to the UAV 10 at the next time point. It may be specified as an altitude. The dividing unit 112 may specify the angle of view and the imaging direction at the next time point based on the imaging condition indicated in the instruction information.

Subsequently, the dividing unit 112 specifies the upper region and the lower region at the next time point based on the altitude, the angle of view, and the imaging direction at the next time point (S318). The dividing unit 112 may specify the upper region and the lower region according to the procedure illustrated in FIG. 7 based on the altitude, the angle of view, and the imaging direction at the next time point. Next, the determination unit 116 determines the exposure values of the upper region and the lower region at the next time point based on the previous exposure values of the upper region and the lower region (S320). Based on the current exposure values determined for the current upper region and the lower region based on the current image captured by the imaging apparatus 100, the determination unit 116 determines the upper region and the lower region at the next time point. The exposure value may be determined. The determination unit 116 may determine the exposure values of the upper region and the lower region at the next time point based on the respective luminance values derived for the current upper region and the lower region. In the process in which the UAV 10 rises to the target altitude by the next time point, the exposure control unit 118 sets the exposure of the imaging apparatus 100 for each of the upper and lower regions based on the exposure values of the upper and lower regions at the next time point. (S322).

If the UAV 10 is in flight (S324), the imaging control unit 110 repeats the processing after step S316. If the UAV 10 is not in flight, the processing after step S316 is temporarily interrupted, and if the UAV 10 is powered off (S326), the series of processing ends. If the UAV 10 is powered on, the processing from step S302 is repeated.

The imaging control unit 110 may execute the processing after step S308 when the altitude of the UAV 10 becomes equal to or higher than a predetermined altitude. The predetermined altitude may be a lower altitude that is likely to include a horizontal line in an image captured by the imaging apparatus 100. In addition, the imaging control unit 110 may temporarily interrupt the processing after step S316 when the altitude of the imaging control unit 110 becomes equal to or lower than a predetermined altitude.

As described above, the dividing unit 112 determines the target altitude of the UAV 10, the target angle of view of the imaging apparatus 100, and the target imaging direction at a future time after a predetermined period from the current time during the flight of the UAV 10. Identify. Based on the target altitude of the UAV 10 at the future time point, the target angle of view of the image capturing device 100, and the target image capturing direction, the dividing unit 112 is configured to detect the upper region and the lower portion of the image captured by the image capturing device 100 at the future time point. Identify the area. Then, the determination unit 116 determines in advance the exposure values of the upper region and the lower region at a future time point based on the exposure values or luminance values of the upper region and the lower region of the current image. As a result, when the UAV 10 is rising or descending, the ratio of the ground or sea surface area to the sky area is greatly changed, so that the brightness of the image captured by the imaging device 100 is greatly changed, and the imaging is performed. It is possible to prevent the apparatus 100 from appropriately performing exposure control.

The exposure control unit 118 has been described mainly with respect to an example in which exposure is controlled for each of a plurality of regions by controlling the gain of an electric signal output by the image sensor 120 according to the exposure amount. However, the exposure control unit 118 may control the exposure for each of a plurality of regions by other methods.

FIG. 14 shows another example of the functional block of the UAV10. The UAV 10 illustrated in FIG. 14 is different from the UAV 10 illustrated in FIG. 2 in that the imaging unit 102 includes an optical filter 140. The optical filter 140 is an optical filter provided in front of the image sensor 120 and capable of changing the light transmittance for each of a plurality of predetermined regions. The optical filter 140 may be a variable ND filter capable of changing the light transmittance for each predetermined region.

For example, the optical filter 140 may be a variable ND filter capable of changing the light transmittance for each of the plurality of horizontal regions 600 as shown in FIG. The optical filter 140 may have a pair of electrodes for each of the plurality of horizontal regions 600. The exposure control unit 118 controls the voltage applied to the pair of electrodes for each of the plurality of horizontal regions 600 based on the exposure value for each of the plurality of regions. Thereby, the exposure control part 118 can control exposure for every some area | region. For example, the exposure control unit 118 applies a voltage that can realize an exposure value of 10 [EV] to a plurality of horizontal regions 602 corresponding to the upper region of the image. The exposure control unit 118 applies voltages capable of realizing exposure values 9 [EV], 8 [EV], and 7 [EV] to a plurality of horizontal regions 606 corresponding to the intermediate region of the image, respectively. The exposure control unit 118 applies a voltage that can realize an exposure value of 6 [EV] to a plurality of horizontal regions 604 with respect to the lower region of the image. Thereby, the exposure control unit 118 can control the exposure of the upper region, the lower region, and at least one intermediate region.

FIG. 16 illustrates an example of a computer 1200 in which aspects of the present invention may be embodied in whole or in part. A program installed in the computer 1200 can cause the computer 1200 to function as an operation associated with the apparatus according to the embodiment of the present invention or as one or more “units” of the apparatus. Alternatively, the program can cause the computer 1200 to execute the operation or the one or more “units”. The program can cause the computer 1200 to execute a process according to an embodiment of the present invention or a stage of the process. Such a program may be executed by CPU 1212 to cause computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222 and an input / output unit, which are connected to the host controller 1210 via the input / output controller 1220. Computer 1200 also includes ROM 1230. The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

The communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, a USB memory, or an IC card or a network. The program is installed in the RAM 1214 or the ROM 1230 that is also an example of a computer-readable recording medium, and is executed by the CPU 1212. Information processing described in these programs is read by the computer 1200 to bring about cooperation between the programs and the various types of hardware resources. An apparatus or method may be configured by implementing information operations or processing in accordance with the use of computer 1200.

For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214 and performs communication processing on the communication interface 1222 based on the processing described in the communication program. You may order. The communication interface 1222 reads transmission data stored in a RAM 1214 or a transmission buffer area provided in a recording medium such as a USB memory under the control of the CPU 1212 and transmits the read transmission data to a network, or The reception data received from the network is written into a reception buffer area provided on the recording medium.

In addition, the CPU 1212 allows the RAM 1214 to read all or necessary portions of a file or database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. Good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium and subjected to information processing. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval that are described throughout the present disclosure for data read from the RAM 1214 and specified by the instruction sequence of the program. Various types of processing may be performed, including / replacement, etc., and the result is written back to RAM 1214. In addition, the CPU 1212 may search for information in files, databases, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. The entry that matches the condition is searched from the plurality of entries, the attribute value of the second attribute stored in the entry is read, and thereby the first attribute that satisfies the predetermined condition is associated. The attribute value of the obtained second attribute may be acquired.

The program or software module described above may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, whereby the program is transferred to the computer 1200 via the network. provide.

The execution order of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior”. It should be noted that they can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for the sake of convenience, it means that it is essential to carry out in this order. is not.

10 UAV
20 UAV body 30 UAV control unit 32 Memory 34 Communication interface 40 Propulsion unit 41 Receiver 42 Inertial measurement device 43 Magnetic compass 44 Barometric altimeter 50 Gimbal 60 Imaging device 100 Imaging device 102 Imaging unit 110 Imaging control unit 112 Dividing unit 114 Deriving unit 116 Determination unit 118 Exposure control unit 120 Image sensor 130 Memory 140 Optical filter 200 Lens unit 210 Lens 212 Lens moving mechanism 220 Lens control unit 300 Remote operation device 1200 Computer 1210 Host controller 1212 CPU
1214 RAM
1220 Input / output controller 1222 Communication interface 1230 ROM

Claims

A dividing unit that divides an image captured by the imaging device into a plurality of regions based on an angle of view of the imaging device, an altitude of the imaging device, and an imaging direction of the imaging device;
And a control unit that controls exposure of the imaging device for each of the plurality of regions.
For each of the plurality of regions, further comprising a determination unit that determines a control value of exposure of the imaging device,
The control device according to claim 1, wherein the control unit controls exposure of the imaging device for each of the plurality of regions based on a control value of exposure of the imaging device.
A derivation unit that derives an evaluation value of brightness of an image captured by the imaging device for each of the plurality of regions;
The control device according to claim 2, wherein the determination unit determines an exposure control value for each of the plurality of regions based on the evaluation value of each of the plurality of regions.
The dividing unit divides the image into an upper region and a lower region based on an angle of view of the imaging device, an altitude of the imaging device, and an imaging direction of the imaging device,
The derivation unit derives evaluation values of the brightness of the upper region and the lower region,
The control device according to claim 3, wherein the determination unit determines a control value of exposure of each of the upper region and the lower region based on an evaluation value of brightness of each of the upper region and the lower region. .
The dividing unit converts a first image captured by the imaging device at the first time point into a first upper region and a first based on a field angle, an altitude, and an imaging direction of the imaging device at a first time point. Divided into lower areas,
The deriving unit derives brightness evaluation values of the first upper region and the first lower region based on the first image,
The dividing unit generates a second image captured by the imaging device at the second time point based on a field angle, an altitude, and an imaging direction of the imaging device at a second time point after the first time point. 2 divided into an upper region and a second lower region,
The determining unit determines an exposure control value of the second upper region based on the evaluation value of the first upper region, and exposes the second lower region based on the evaluation value of the first lower region. The control device according to claim 4, wherein the control value is determined.
The angle of view, altitude, and imaging direction of the imaging device at the second time point correspond to the angle of view, altitude, and imaging direction of the imaging device at the second time point specified before the second time point. The control device according to claim 5.
The imaging device is mounted on a moving body,
The control device according to claim 6, wherein an altitude of the imaging device corresponds to an altitude of the moving body at the second time point specified before the second time point.
The dividing unit divides the image into the upper region, the lower region, and at least one intermediate region between the upper region and the lower region,
5. The determination unit according to claim 4, wherein the determining unit determines the exposure control value of the at least one intermediate region to a value between the exposure control value of the upper region and the exposure control value of the lower region. Control device.
The imaging device has an image sensor,
The control device according to claim 1, wherein the control unit controls exposure of the imaging device for each of the plurality of regions by controlling a gain of an electric signal output by the image sensor according to an exposure amount.
The imaging device
An image sensor;
An optical filter provided in front of the image sensor and capable of changing light transmittance for each of a plurality of predetermined regions;
The said control part controls exposure of the said imaging device for every said several area | region by controlling the transmittance | permeability of light for every said predetermined several area | region of the said optical filter. Control device.
A control device according to any one of claims 1 to 10,
An imaging device comprising an image sensor.
An imaging device according to claim 11;
An imaging system comprising: a support mechanism that supports the imaging device.
A moving body that carries the imaging system according to claim 12 and moves.
Dividing an image captured by the imaging device into a plurality of regions based on an angle of view of the imaging device, an altitude of the imaging device, and an imaging direction of the imaging device;
Controlling the exposure of the imaging device for each of the plurality of regions.
Dividing an image captured by the imaging device into a plurality of regions based on an angle of view of the imaging device, an altitude of the imaging device, and an imaging direction of the imaging device;
The program for making a computer perform the step which controls exposure of the said imaging device for every said some area | region.