JP6801161B1 - Image processing equipment, imaging equipment, moving objects, image processing methods, and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform image processing for emphasizing a high frequency component with an appropriate intensity on an image obtained by capturing an image of a subject moving relative to an imaging device. An image processing device performs image processing that emphasizes high frequency components on an image captured by the image pickup device while changing its position or imaging direction. The image processing device acquires information indicating the high image quality of each image plane region of the image pickup lens included in the image processing device, acquires information indicating the high contrast of each image area from the image, and refers to the image pickup device. The movement speed of each image area of the subject is specified, the index value for each image area is calculated based on the high imaging performance for each image plane area and the high contrast for each image area, and each image area is calculated. It is provided with a circuit configured to determine the intensity of image processing for each image region based on the index value and the movement speed for each image region, and execute the image processing with the intensity determined for each image region. [Selection diagram] FIG. 10

Description

The present invention relates to an image processing device, an imaging device, a moving body, an image processing method, and a program.

Patent Document 1 discloses an enhancer (image quality improving device) that improves a sense of contrast.
[Prior art literature]
[Patent Document]
[Patent Document 1] Japanese Unexamined Patent Publication No. 2010-28178

For example, when the image pickup device takes a picture while changing the position or the image pickup direction, an image including an image area in which the relative movement speed of the subject with respect to the image pickup device is high and an image area in which the relative movement speed of the subject is low can be obtained. There is. It is desirable to perform image processing that emphasizes high-frequency components with an appropriate intensity for an image in which the subject is moving in this way.

The image processing apparatus according to one aspect of the present invention performs image processing that emphasizes high frequency components on an image captured by the imaging device while changing its position or imaging direction. The image processing device acquires information indicating the high imaging performance of each image plane region of the image pickup lens included in the image processing device. The image processing device acquires information indicating the height of contrast for each image area from the image. The image processing device specifies the movement speed of each image area of the subject with respect to the image pickup device. The image processing apparatus calculates an index value for each image region based on the height of the imaging performance for each image plane region and the height of the contrast for each image region. The image processing apparatus determines the intensity of image processing for each image area based on the index value for each image area and the movement speed for each image area, and executes the image processing with the intensity determined for each image area.

The information indicating the high imaging performance for each image plane region of the lens may be based on the information indicating the MTF for each image height of the lens.

The circuit may be configured so that the higher the imaging performance is, the higher the index value is calculated, and the higher the contrast is, the higher the index value is calculated.

The circuit may be configured so that the higher the index value in the image region and the higher the movement speed, the weaker the intensity of the image processing applied to the image region.

The circuit may be configured so that when the index value in the image region is higher than the reference value and the movement speed in the image region is higher, the intensity of the image processing is weakened.

The circuit may be configured so that when the index value in the image region is lower than the reference value, the lower the movement speed in the image region, the stronger the intensity of the image processing.

The circuit may be configured so that when the movement speed in the image region is higher than the reference value and the index value in the image region is higher, the intensity of the image processing is weakened.

The circuit may be configured so that when the movement speed in the image region is lower than the reference value, the lower the index value in the image region, the stronger the intensity of the image processing.

When the orientation of the image pickup device changes, the circuit captures the movement speed of the subject in the image captured at the next timing based on the movement speed of the subject specified by using the image and the change speed of the orientation of the image pickup device. It may be configured to be specified for each region and to determine the intensity of image processing applied to the image captured at the next timing based on the index value for each image region and the specified movement speed.

The circuit is based on the movement speed of the subject specified by using the image and the amount of change in the movement speed of the image pickup device when the movement speed of the image pickup device changes, and the movement speed of the subject in the image captured at the next timing. Is specified for each image region, and the intensity of image processing applied to the image captured at the next timing may be determined based on the index value for each image region and the specified movement speed.

The image pickup apparatus according to one aspect of the present invention includes an image sensor and the above-mentioned image processing apparatus.

The moving body according to one aspect of the present invention may be a moving body provided with the above-mentioned imaging device.

In the image processing method according to one aspect of the present invention, image processing for emphasizing high frequency components is performed on an image captured by an imaging device while changing its position or imaging direction. The image processing method includes a step of acquiring information indicating the high imaging performance of each image plane region of the imaging lens included in the imaging device. The image processing method includes a step of acquiring information indicating the height of contrast for each image area from the image. The image processing method includes a step of specifying the movement speed of each image area of the subject with respect to the image pickup apparatus. The image processing method includes a step of calculating an index value for each image region based on the height of the imaging performance for each image plane region and the height of the contrast for each image region. The image processing method includes a step of determining the intensity of image processing for each image region based on the index value for each image region and the movement speed for each image region, and executing the image processing with the determined intensity for each image region.

The program according to one aspect of the present invention causes a computer to perform image processing that emphasizes high frequency components in an image captured by an imaging device while changing its position or imaging direction. The program causes the computer to perform a procedure of acquiring information indicating the high imaging performance of each image plane region of the imaging lens provided in the imaging device. The program causes the computer to perform a procedure of acquiring information indicating the height of contrast for each image area from the image. The program causes the computer to perform a procedure for identifying the movement speed of the subject for each image area with respect to the imaging device. The program causes a computer to perform a procedure of calculating an index value for each image region based on the height of imaging performance for each image plane region and the height of contrast for each image region. The program determines the intensity of image processing for each image area based on the index value for each image area and the movement speed for each image area, and causes the computer to execute the procedure for executing the image processing with the determined intensity for each image area. ..

According to one aspect of the present invention, it is possible to perform image processing that emphasizes high-frequency components with an appropriate intensity on an image of an image of a moving subject relative to an image pickup device.

The outline of the above invention does not list all the necessary features of the present invention. Sub-combinations of these feature groups can also be inventions.

An example of the appearance of the unmanned aerial vehicle (UAV) 10 and the remote control device 300 is shown. An example of the functional block of UAV10 is shown. A part of the image height dependence of the MTF (Modulation TransSfer Function) of the lens 210 is shown. The MTF map is shown. The contrast map is shown. The contrast index value map is shown. The relative velocity map is shown. This is an example of the gain information of the enhance filter. The frequency characteristics of the enhance filter are schematically shown. It is a flowchart which shows the procedure of the process which the image pickup control unit 110 executes. An example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part is shown.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters that are subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent the "part" of. Specific stages and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. It may include a memory element such as.

The computer-readable medium may include any tangible device capable of storing instructions executed by the appropriate device. As a result, the computer-readable medium having the instructions stored therein will include the product, including instructions that can be executed to create means for performing the operation specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated A circuit card or the like may be included.

Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes traditional procedural programming languages. Traditional procedural programming languages are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or Smalltalk®, JAVA®, C ++. It may be an object-oriented programming language such as, and a "C" programming language or a similar programming language. Computer-readable instructions are used locally or on a local area network (LAN), wide area network (WAN) such as the Internet, to the processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing device. ) May be provided. The processor or programmable circuit may execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers and the like.

FIG. 1 shows an example of the appearance of the unmanned aerial vehicle (UAV) 10 and the remote control device 300. The UAV 10 includes a UAV main body 20, a gimbal 50, a plurality of imaging devices 60, and an imaging device 100. The gimbal 50 and the imaging device 100 are examples of an imaging system. The UAV 10 is a concept including a moving body, an air vehicle moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like. An airship that moves in the air is a concept that includes UAVs, other aircraft that move in the air, airships, helicopters, and the like.

The UAV main body 20 includes a plurality of rotor blades. The plurality of rotor blades are an example of a propulsion unit. The UAV main body 20 flies the UAV 10 by controlling the rotation of a plurality of rotor blades. The UAV body 20 flies the UAV 10 using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed-wing aircraft having no rotor blades.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The gimbal 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the image pickup device 100 on a pitch axis using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the roll axis and the yaw axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two imaging devices 60 may be provided in front of the nose of the UAV 10. Yet two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two image pickup devices 60 on the front side may form a pair and function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of image pickup devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV 10 may be provided with at least one imaging device 60 on each of the nose, nose, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the image pickup device 60 may be wider than the angle of view that can be set by the image pickup device 100. The image pickup apparatus 60 may have a single focus lens or a fisheye lens.

The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may communicate wirelessly with the UAV 10. The remote control device 300 transmits instruction information indicating various commands related to the movement of the UAV 10, such as ascending, descending, accelerating, decelerating, advancing, reversing, and rotating, to the UAV 10. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to ascend the UAV 10. The UAV10 rises while accepting the rise order. Even if the UAV10 accepts the ascending command, the ascending may be restricted if the altitude of the UAV10 reaches the upper limit altitude.

FIG. 2 shows an example of the functional block of the UAV 10. The UAV 10 includes a UAV control unit 30, a memory 37, a communication interface 36, a propulsion unit 40, a GPS receiver 41, an inertial measurement unit 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a humidity sensor 46, a gimbal 50, and an imaging device. The 60 and the image pickup device 100 are provided.

The communication interface 36 communicates with another device such as the remote control device 300. The communication interface 36 may receive instruction information including various commands from the remote control device 300 to the UAV control unit 30. In the memory 37, the UAV control unit 30 has a propulsion unit 40, a GPS receiver 41, an inertial measurement unit (IMU) 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a humidity sensor 46, a gimbal 50, an image pickup device 60, and the like. The program and the like necessary for controlling the image pickup device 100 are stored. The memory 37 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, USB memory, and solid state drive (SSD). The memory 37 may be provided inside the UAV main body 20. It may be provided so as to be removable from the UAV main body 20.

The UAV control unit 30 controls the flight and imaging of the UAV 10 according to the program stored in the memory 37. The UAV control unit 30 may be composed of a CPU, a microprocessor such as an MPU, a microcontroller such as an MCU, or the like. The UAV control unit 30 controls the flight and imaging of the UAV 10 according to a command received from the remote control device 300 via the communication interface 36. The propulsion unit 40 promotes the UAV 10. The propulsion unit 40 has a plurality of rotary blades and a plurality of drive motors for rotating the plurality of rotary blades. The propulsion unit 40 makes the UAV 10 fly by rotating a plurality of rotor blades via the plurality of drive motors in accordance with a command from the UAV control unit 30.

The GPS receiver 41 receives a plurality of signals indicating the time transmitted from the plurality of GPS satellites. The GPS receiver 41 calculates the position (latitude and longitude) of the GPS receiver 41, that is, the position (latitude and longitude) of the UAV 10 based on the plurality of received signals. The IMU 42 detects the posture of the UAV 10. The IMU 42 detects the acceleration in the three axial directions of the front and rear, the left and right, and the up and down of the UAV 10 and the angular velocity in the three axial directions of pitch, roll, and yaw as the posture of the UAV 10. The magnetic compass 43 detects the nose orientation of the UAV 10. The barometric altimeter 44 detects the altitude at which the UAV 10 flies. The barometric altimeter 44 detects the barometric pressure around the UAV 10, converts the detected barometric pressure into an altitude, and detects the altitude. The temperature sensor 45 detects the ambient temperature of the UAV 10. The humidity sensor 46 detects the humidity around the UAV 10.

The image pickup apparatus 100 includes an image pickup section 102 and a lens section 200. The lens unit 200 is an example of a lens device. The image pickup unit 102 includes an image sensor 120, an image pickup control unit 110, a memory 130, and a distance measurement sensor 140. The image sensor 120 may be configured by CCD or CMOS. The image sensor 120 captures an optical image formed through a plurality of lenses 210, and outputs the captured image to the image pickup control unit 110. The image pickup control unit 110 may be composed of a CPU, a microprocessor such as an MPU, a microcontroller such as an MCU, or the like. The image pickup control unit 110 may control the image pickup device 100 in response to an operation command of the image pickup device 100 from the UAV control unit 30. The image pickup control unit 110 is an example of a circuit. The memory 130 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, USB memory, and solid state drive (SSD). The memory 130 stores a program or the like necessary for the image pickup control unit 110 to control the image sensor 120 or the like. The memory 130 may be provided inside the housing of the image pickup apparatus 100. The memory 130 may be provided so as to be removable from the housing of the image pickup apparatus 100.

The distance measuring sensor 140 measures the distance to the subject. The distance measuring sensor 140 may be an infrared sensor, an ultrasonic sensor, a stereo camera, a TOF (Time Of Flight) sensor, or the like.

The lens unit 200 includes a plurality of lenses 210, a plurality of lens driving units 212, and a lens control unit 220. The plurality of lenses 210 may function as a zoom lens, a varifocal lens, and a focus lens. At least a part or all of the plurality of lenses 210 are arranged so as to be movable along the optical axis. The lens unit 200 may be an interchangeable lens that is detachably provided to the imaging unit 102. The lens driving unit 212 moves at least a part or all of the plurality of lenses 210 along the optical axis via a mechanical member such as a cam ring. The lens driving unit 212 may include an actuator. The actuator may include a stepping motor. The lens control unit 220 drives the lens drive unit 212 in accordance with a lens control command from the image pickup unit 102 to move one or more lenses 210 along the optical axis direction via a mechanical member. The lens control command is, for example, a zoom control command and a focus control command.

The lens unit 200 further includes a memory 222 and a position sensor 214. The lens control unit 220 controls the movement of the lens 210 in the optical axis direction via the lens drive unit 212 in response to a lens operation command from the image pickup unit 102. The lens control unit 220 controls the movement of the lens 210 in the optical axis direction via the lens drive unit 212 in response to a lens operation command from the image pickup unit 102. Part or all of the lens 210 moves along the optical axis. The lens control unit 220 executes at least one of the zoom operation and the focus operation by moving at least one of the lenses 210 along the optical axis. The position sensor 214 detects the position of the lens 210. The position sensor 214 may detect the current zoom position or focus position.

The lens driving unit 212 may include a runout correction mechanism. The lens control unit 220 may execute the shake correction by moving the lens 210 in the direction along the optical axis or in the direction perpendicular to the optical axis via the shake correction mechanism. The lens driving unit 212 may drive the runout correction mechanism by a stepping motor to perform runout correction. The runout correction mechanism may be driven by a stepping motor to move the image sensor 120 in a direction along the optical axis or in a direction perpendicular to the optical axis to perform runout correction.

The memory 222 stores the control values of the plurality of lenses 210 that move via the lens driving unit 212. The memory 222 may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory.

The image processing function included in the image pickup apparatus 100 will be described. The image processing function is carried out by the image pickup control unit 110. The image pickup control unit 110 performs enhancement processing by applying an enhancement filter to the image output from the image sensor 120. The enhancement process is an example of a process for emphasizing a spatial frequency component in a high frequency region of an image, such as an edge enhancement process or a sharpening process. The image pickup control unit 110 generates video data for recording by performing image processing including enhancement processing on the images sequentially output from the image sensor 120, and records the images in the memory 130. In the present embodiment, the enhancement process executed by the image pickup control unit 110 will be described mainly when the UAV 10 takes a picture while flying or when the gimbal 50 takes a picture while changing the direction of the image pickup device 100.

The image pickup control unit 110 acquires information indicating the high imaging performance of each image plane region of the lens 210 included in the image pickup apparatus 100. The information indicating the high imaging performance for each image plane region of the lens may be information based on the information indicating the MTF for each image height of the lens.

The image pickup control unit 110 acquires information indicating the height of contrast for each image region from the image. The image pickup control unit 110 specifies the movement speed of each image region of the subject with respect to the image pickup device 100. The image pickup control unit 110 calculates an index value for each image region based on the height of the imaging performance for each image plane region and the height of the contrast for each image region. The index value may be an index value indicating the height of contrast. The image pickup control unit 110 determines the intensity of the enhancement process for each image area based on the index value for each image area and the movement speed for each image area, and executes the enhancement process with the intensity determined for each image area.

The image pickup control unit 110 calculates the index value higher as the imaging performance is higher, and calculates the index value higher as the contrast is higher. For example, the image pickup control unit 110 may calculate an index value by multiplying a value indicating a high imaging performance and a value indicating a high contrast.

The imaging control unit 110 may weaken the intensity of the enhancement process applied to the image region as the index value in the image region is higher and the movement speed is higher. For example, when the index value in the image region is higher than the reference value, the imaging control unit 110 may weaken the intensity of the enhancement process as the movement speed in the image region is higher. When the index value in the image region is lower than the reference value, the image pickup control unit 110 may increase the intensity of the enhancement process as the movement speed in the image region is lower.

When the movement speed in the image region is higher than the reference value, the image pickup control unit 110 may weaken the intensity of the enhancement processing as the index value in the image region is higher. When the movement speed in the image region is lower than the reference value, the image pickup control unit 110 may increase the intensity of the enhancement processing as the index value in the image region is lower.

When the orientation of the image pickup device 100 changes, the image pickup control unit 110 determines the subject in the image captured at the next timing based on the movement speed of the subject specified by using the image and the change speed of the orientation of the image pickup device 100. The movement speed of the image is specified for each image area, and the intensity of the enhancement process applied to the image captured at the next timing is determined based on the index value for each image area and the specified movement speed.

When the moving speed of the imaging device 100 changes, the image pickup control unit 110 captures an image at the next timing based on the moving speed of the subject specified by using the image and the amount of change in the moving speed of the imaging device 100. The movement speed of the subject in the image may be specified for each image area, and the intensity of the enhancement process applied to the image captured at the next timing may be determined based on the index value for each image area and the specified movement speed.

FIG. 3 shows a part of the image height dependence of the MTF (Modulation Transfer Function) of the lens 210. The horizontal axis of the graph of FIG. 3 represents the image height, and the vertical axis represents the contrast. Graph 310 represents an MTF when the F value (aperture value) is 2.8. Graph 320 represents an MTF when the F value is 11. These graphs show the MTF at a predetermined specific spatial frequency.

FIG. 4 shows an MTF map. The MTF map is information generated based on the image height data shown in FIG. FIG. 4 is an MTF map when the entire image is divided into an image area of 4 rows and 6 columns. In this embodiment, for the purpose of explaining in an easy-to-understand manner, a case where the entire image is divided into an image area of 4 rows and 6 columns is illustrated. However, it is possible to adopt a form in which the entire image is divided into an arbitrary number of image regions. Further, the shape of the image area is not limited to a square or a rectangle, and may have any shape.

The MTF map contains the contrast value of the MTF associated with each image area. As described above, the MTF changes depending on the F value. Therefore, the MTF map is generated for each F value. The memory 130 stores the MTF map in association with the F value. The image pickup control unit 110 reads the MTF map associated with the F value of the current lens 210 from the memory 130.

FIG. 5 shows a contrast map. The contrast map is information indicating the contrast value for each image region detected from the image acquired from the image sensor 120. FIG. 5 shows a contrast map when the entire image is divided into an image area of 4 rows and 6 columns. The contrast map contains the contrast values associated with each image area. The image pickup control unit 110 generates a contrast map by detecting a contrast value from an image of each image region in the image.

FIG. 6 shows a contrast index value map. The contrast index value map is generated based on the MTF map and the contrast map. The contrast index value map shows the contrast index value for each image area. The image pickup control unit 110 calculates the product of the contrast value of the MTF map and the contrast value of the contrast map in each of the image regions as a contrast index value.

The image pickup control unit 110 classifies the contrast index value of each image region into five levels based on the size of the contrast index value of the generated contrast index value map. For example, the image pickup control unit 110 divides the range from the minimum value to the maximum value of the calculated contrast index value into five ranges. The image pickup control unit 110 assigns levels "5" to "1" to five ranges in descending order of contrast index value. The image pickup control unit 110 may determine five ranges of the contrast index values so that the average value of the calculated contrast index values is included in the range of the level "3".

FIG. 7 shows a relative velocity map. The relative velocity map is information indicating the relative velocity with the subject determined for each image area of the captured image. FIG. 6 shows a relative velocity map when the entire image is divided into an image area of 4 rows and 6 columns. The relative velocity map associates relative velocities with each image area. In FIG. 7, the relative speeds are represented by “high”, “medium”, and “low” in descending order of speed.

The imaging control unit 110 specifies the height of the relative velocity based on the relative velocity detected from the optical flow detected from the captured image, the moving speed of the UAV 10, and the rotation speed of the imaging device 100 by the gimbal 50 in the imaging direction. To do. The moving speed of the UAV 10 includes the translation speed of the UAV 10, the rising speed of the UAV 10, the falling speed of the UAV 10, and the overall posture change speed of the UAV 10. The moving speed of the UAV 10 and the rotation speed of the gimbal 50 in the imaging direction are calculated based on the operation amount indicated by the instruction information received from the remote control device 300.

As an example, consider a case where the UAV 10 continuously acquires images while flying at a low altitude along a narrow forest road and shoots a moving image. It is assumed that there are trees on the left side of the UAV 10 in the traveling direction and a flat land spreads on the right side in the traveling direction. Further, it is assumed that the imaging direction of the imaging device 100 coincides with the traveling direction of the UAV 10.

Here, when the UAV 10 turns to the right during shooting, the subject in the left image area is located outside the rotation center of the subject in the right image area, so that the relative speed is higher than that of the subject in the right image area. It gets higher. The image pickup control unit 110 detects the relative velocity of the current subject for each image region based on the optical flow detected from the images acquired so far. When the UAV 10 turns, the image pickup control unit 110 identifies in which image region of the image to be acquired at the next timing the subject in the current image is located based on the predicted turning speed of the UAV 10. To do.

Further, the image pickup control unit 110 predicts the relative speed of each image region of the image acquired at the next timing based on the relative speed of the current subject and the predicted turning speed of the UAV 10, and next Generate a relative velocity map in the image acquired at the timing. Then, the imaging control unit 110 determines the gain of the enhancement filter applied to the image acquired at the next timing based on the relative velocity map and the contrast index value map of the image acquired at the next timing.

The image pickup control unit 110 is a contrast index in the image acquired at the next timing based on the predicted turning speed of the UAV 10, the contrast information of the subject in each image area specified from the current image, and the MTF map. Generate a value map. The contrast information of the subject in each image area specified from the current image may be the contrast map in the current image, or may be the contrast of the subject itself specified from the contrast map in the current image and the MTF map. .. Then, the image pickup control unit 110 may determine the gain of the enhancement filter applied to the image acquired at the next timing based on the relative velocity map and the contrast index value map of the image acquired at the next timing.

FIG. 8 is an example of the gain information of the enhance filter. The gain information is information that associates the gain coefficient of the enhancement filter with the combination of the level of the contrast index value and the relative velocity. In the example of FIG. 3, a gain coefficient “0.5” is associated with the combination of the contrast index value level “5” and the relative velocity “high”. The gain information is stored in the memory 130. The image pickup control unit 110 refers to the gain information stored in the memory 130 to acquire the gain coefficient associated with the combination of the contrast index value and the relative velocity.

The gain coefficient represents the strength of the enhancement process. The larger the gain coefficient, the stronger the enhancement process. The gain coefficient may be, for example, a correction value for correcting the intensity of the default enhancement filter used for the enhancement process executed by the image pickup control unit 110. For example, a gain coefficient "1" means to use the default enhance filter, and a gain coefficient "0.5" means to use an enhance filter having an intensity of 1/2 of the intensity of the default enhance filter. You can do it.

For example, when the relative velocity is "high", the higher the contrast index value, the lower the gain coefficient may be applied. When the relative velocity is "low", the lower the contrast index value, the higher the gain coefficient may be applied. When the contrast index value is level "4" or "5", the higher the relative velocity, the lower the gain coefficient can be applied. When the contrast index value is level "1" or "2", the lower the relative velocity, the higher the gain coefficient can be applied.

Specifically, in the example of the gain information shown in FIG. 8, the lowest gain coefficient is associated with the contrast index value level “5” and the relative velocity “high”. Therefore, the image pickup control unit 110 applies a weak enhancement process to an image region having a high relative speed and a high contrast index value. Therefore, the edges can be made inconspicuous. As a result, it is possible to prevent the edges of the moving image region from being significantly emphasized. The default intensity enhancement process can be applied to an image area with a high relative speed and a low contrast index value, but since the original contrast index value is low, the edges may not be noticeable even after the enhancement process. Absent. In this way, by adjusting the intensity of the enhancement process for each image region based on the gain information shown in FIG. 8, it is possible to suppress the image from losing the sense of motion due to the enhance process.

FIG. 9 schematically shows the frequency characteristics of the enhance filter. The horizontal axis of the graph of FIG. 9 is the spatial frequency, and the vertical axis is the luminance gain. The gain curve 900 in the graph of FIG. 9 represents the spatial frequency characteristics of the default enhance filter used for enhancing the captured image.

The image pickup control unit 110 corrects the gain curve 900 based on the gain coefficient determined from the contrast index value and the relative speed, and executes the enhancement process. For example, assuming that the gain coefficient determined from the contrast index value and the relative velocity is C, the imaging control unit 110 executes the enhancement process using an enhancement filter having a frequency characteristic of peak gain of C × G0.

FIG. 10 is a flowchart showing a procedure of processing executed by the image pickup control unit 110. The processing of this flowchart is started when the moving image shooting by the image pickup apparatus 100 is started.

In S1000, the imaging control unit 110 acquires the MTF map from the memory 130 as a part of the initial processing. Further, the image pickup control unit 110 sets a default enhancement filter as an enhancement filter used for the enhancement process. In S1002, the image pickup control unit 110 acquires an image from the image sensor 120.

In S1004, the image pickup control unit 110 executes image processing of the image acquired from the image sensor 120. The image processing includes the enhancement processing described above. The image pickup control unit 110 executes the enhancement process using the currently set enhancement filter.

In S1006, the image pickup control unit 110 acquires contrast for each image area and generates a contrast map. In S1008, the imaging control unit 110 generates a contrast index value map based on the MTF map and the contrast map.

In S1010, the image pickup control unit 110 detects the optical flow from a plurality of images acquired in the past from the image sensor 120, and acquires the speed of the subject in the image.

In S1012, the image pickup control unit 110 determines whether or not at least one of the operation signal of the UAV 10 and the operation signal of the gimbal 50 has been received. The operation signal of the UAV 10 and the operation signal of the gimbal 50 are generated based on the instruction information received from the remote control device 300. The operation signal of the UAV 10 indicates an acceleration amount, a deceleration amount, an attitude control amount, an ascending amount, a descending amount, and the like of the UAV 10. The operation signal of the gimbal 50 indicates the amount of rotation of the UAV 10 around the yaw axis and the amount of rotation around the pitch axis.

When the operation signal of the UAV 10 is received, the image pickup control unit 110 calculates the speed of the UAV 10 in S1014. When the operation signal of the gimbal 50 is received, the rotation speed of the gimbal 50 is calculated in S1016. When the processes of S1014 and S1016 are executed, the process proceeds to S1018. If the operation signal of the UAV 10 and the operation signal of the gimbal 50 are not received, the process proceeds to S1018 without processing S1014 and S1016.

In S1018, the imaging control unit 110 generates a relative velocity map. Specifically, when neither the operation signal of the UAV 10 nor the operation signal of the gimbal 50 is received, the image pickup control unit 110 generates a relative speed map based on the speed of the subject acquired from the optical flow in S1010. To do. When the operation signal of the UAV 10 is received, the imaging control unit 110 generates a relative speed map in the image acquired at the next timing based on the speed of the subject acquired from the optical flow and the future speed of the UAV 10. When the operation signal of the gimbal 50 is received, the imaging control unit 110 obtains a relative speed map in the image acquired at the next timing based on the speed of the subject acquired from the optical flow and the future rotation speed of the gimbal 50. Generate. When the operation signal of the UAV 10 and the operation signal of the gimbal 50 are received, the image pickup control unit 110 receives the speed of the subject acquired from the optical flow, the future speed of the UAV 10, and the future rotation speed of the gimbal 50. Generate a relative velocity map for the image to be acquired at the next timing.

In S1020, the gain coefficient of the enhancement filter in each image area of the image acquired at the next timing is determined based on the contrast index value map and the relative velocity map. In S1022, an enhancement filter to be applied to the image acquired at the next timing is set based on the gain coefficient determined in S1020.

In S1024, it is determined whether or not to end the moving image shooting. For example, the image pickup control unit 110 determines that the moving image shooting is finished when the operation signal instructing the end of the moving image shooting is received. When the image pickup control unit 110 determines that the moving image shooting is not finished, the process proceeds to S1002. When the image pickup control unit 110 determines that the moving image shooting is finished, the image pickup control unit 110 ends the process of this flowchart.

As described above, the image pickup control unit 110 adjusts the intensity of the enhancement process for each image region according to the flight state of the UAV 10 and the attitude of the gimbal 50 on which the image pickup device is mounted. For example, the image pickup control unit 110 reduces the intensity of the enhancement process when it is predicted that the relative speed of the image region having high contrast will increase. This makes it possible to generate an image in which a moving subject looks natural.

The gain information described in relation to FIG. 8 and the like is an example of gain information for preventing the edges of a moving subject image from being extremely emphasized. The gain information may be appropriately set according to the purpose of image processing. For example, gain information that emphasizes the edges of a moving subject image may be set for the purpose of special effects.

In the above embodiment, the process applied to the image taken by the image pickup apparatus 100 mounted on the UAV 10 as an example of the moving body has been described. However, the image pickup device 100 does not have to be mounted on a moving body such as the UAV 10. For example, the above-described processing may be applied to an image captured by the image pickup device 100 while the user holds the image pickup device 100 in his hand and rotates or moves the image pickup device 100. The above-described processing is performed on an image in which a user holds an image pickup device 100 via a hand-held gimbal, a support rod, or the like, and the image pickup device 100 is imaged while rotating or moving the image pickup device 100 via the hand-held gimbal, the support rod, or the like. May be applied.

FIG. 11 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with the device according to an embodiment of the present invention or as one or more "parts" of the device. Alternatively, the program may cause the computer 1200 to perform the operation or one or more "parts". The program can cause the computer 1200 to perform a process or a step of the process according to an embodiment of the present invention. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. Computer 1200 also includes ROM 1230. The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

Communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by CPU 1212 in computer 1200. The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation, and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, USB stick or IC card or network. The program is installed in RAM 1214 or ROM 1230, which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the operation or processing of information according to the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing on the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The received data received from the network is written to the reception buffer area or the like provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. Good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214 and specified by the instruction sequence of the program. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, a database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or in a computer readable storage medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided in a dedicated communication network or a server system connected to the Internet can be used as a computer-readable storage medium, thereby allowing the program to be transferred to the computer 1200 over the network. provide.

The execution order of each process such as operation, procedure, step, and step in the device, system, program, and method shown in the claims, the specification, and the drawing is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

10 UAV
20 UAV main unit 30 UAV control unit 36 Communication interface 37 Memory 40 Propulsion unit 41 GPS receiver 42 Inertial measurement unit 43 Magnetic compass 44 Atmospheric pressure sensor 45 Temperature sensor 46 Humidity sensor 50 Gimbal 60 Imaging device 100 Imaging device 102 Imaging unit 110 Imaging control unit 120 Image sensor 130 Memory 140 Distance measurement sensor 200 Lens unit 210 Lens 212 Lens drive unit 214 Position sensor 220 Lens control unit 222 Memory 300 Remote control device 310, 320 Graph 900 Gain curve 1200 Computer 1210 Host controller 1212 CPU
1214 RAM
1220 Input / Output Controller 1222 Communication Interface 1230 ROM

Claims (14)

An image processing device that performs image processing that emphasizes high-frequency components on an image captured by the imaging device while changing its position or imaging direction.
Information indicating the high imaging performance of each image plane region of the imaging lens provided in the imaging device is acquired.
Information indicating the height of contrast for each image area is acquired from the image, and the information is obtained.
The movement speed of each image area of the subject with respect to the image pickup device is specified.
An index value for each image region is calculated based on the height of the imaging performance for each image plane region and the height of the contrast for each image region.
The intensity of the image processing is determined for each image region based on the index value for each image region and the movement speed for each image region, and the image processing is executed with the intensity determined for each image region. An image processing device including the circuit. The image processing apparatus according to claim 1, wherein the information indicating the high imaging performance of each image plane region of the imaging lens is based on the information indicating the MTF for each image height of the imaging lens. The image processing apparatus according to claim 1 or 2, wherein the circuit is configured to calculate the index value higher as the imaging performance is higher and to calculate the index value higher as the contrast is higher. .. The image processing apparatus according to claim 3, wherein the circuit is configured so that the higher the index value in the image region and the higher the movement speed, the weaker the intensity of the image processing applied to the image region. The image according to claim 3, wherein the circuit is configured so that when the index value in the image region is higher than the reference value, the higher the movement speed in the image region, the weaker the intensity of the image processing. Processing equipment. The image according to claim 3, wherein the circuit is configured to increase the intensity of the image processing as the movement speed in the image region is lower when the index value in the image region is lower than the reference value. Processing equipment. The image according to claim 3, wherein the circuit is configured so that when the movement speed in the image region is higher than the reference value, the higher the index value in the image region, the weaker the intensity of the image processing. Processing equipment. The image according to claim 3, wherein the circuit is configured to increase the intensity of the image processing as the index value in the image region becomes lower when the movement speed in the image region is lower than the reference value. Processing equipment. When the orientation of the image pickup device changes, the circuit describes the subject in the image captured at the next timing based on the movement speed of the subject specified by using the image and the change speed of the orientation of the image pickup device. The movement speed is specified for each image area, and the intensity of the image processing applied to the image captured at the next timing is determined based on the index value for each image area and the specified movement speed. The image processing apparatus according to claim 1 or 2. The circuit is an image captured at the next timing based on the movement speed of the subject specified by using the image and the amount of change in the movement speed of the image pickup device when the movement speed of the image pickup device changes. The movement speed of the subject is specified for each image area, and the intensity of the image processing applied to the image captured at the next timing is determined based on the index value for each image area and the specified movement speed. The image processing apparatus according to claim 1 or 2. Image sensor and
An imaging device including the image processing device according to claim 1 or 2. A moving body that moves with the imaging device according to claim 11. An image processing method in which an image processing device performs image processing that emphasizes high frequency components on an image captured while changing the position or imaging direction.
The stage of acquiring information indicating the high imaging performance of each image plane region of the imaging lens included in the imaging device, and
At the stage of acquiring information indicating the height of contrast for each image area from the image, and
A step of specifying the movement speed of each image area of the subject with respect to the image pickup device, and
A step of calculating an index value for each image region based on the height of imaging performance for each image plane region and the height of contrast for each image region, and
The step of determining the intensity of the image processing for each image region based on the index value for each image region and the movement speed for each image region, and executing the image processing with the intensity determined for each image region. Image processing method to prepare. A program for causing a computer to perform image processing that emphasizes high-frequency components in an image captured by an imaging device while changing its position or imaging direction.
A procedure for acquiring information indicating the high imaging performance of each image plane region of the imaging lens provided in the imaging device, and a procedure for acquiring information indicating the high imaging performance.
A procedure for acquiring information indicating the height of contrast for each image area from the image, and
A procedure for specifying the movement speed of the subject for each image area with respect to the image pickup device, and
A procedure for calculating an index value for each image region based on the high imaging performance for each image plane region and the high contrast for each image region, and
The procedure of determining the intensity of the image processing for each image region based on the index value for each image region and the movement speed for each image region, and executing the image processing with the intensity determined for each image region. A program to be executed by the computer.

Images