JP2021026581A - Image processing device, imaging apparatus, movable body, image processing method and program - Google Patents

Abstract

To suppress the sharpness of a subject image from being unnaturally changed in accordance with the change of a subject distance.SOLUTION: An image processing device comprises a circuit which performs image processing on an image captured by an imaging apparatus moving along an imaging direction. The circuit may be configured to perform image processing of emphasizing the frequency component of a lower spatial frequency band to an image region including a subject existing at a closer distance. An image processing method for performing image processing on the image captured by the imaging apparatus moving along the imaging direction includes a step of performing the image processing of emphasizing the frequency component of the lower spatial frequency band to the image region including the subject existing at a closer distance.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image processing device, an imaging device, a moving body, an image processing method, and a program.

Patent Document 1 discloses an enhancer (image quality improving device) that improves a sense of contrast.
[Prior art literature]
[Patent Document]
[Patent Document 1] Japanese Unexamined Patent Publication No. 2010-28178

For example, when shooting while approaching a distant subject, the spatial frequency component of the subject image shifts to the low frequency side as it approaches the subject. Therefore, if image processing that emphasizes a specific spatial frequency band is uniformly applied to an image taken while approaching the subject, the sharpness of the subject image may change unnaturally according to a change in the subject distance.

The image processing apparatus according to one aspect of the present invention may include a circuit that performs image processing on an image captured by an imaging device that moves along an imaging direction. The circuit may be configured to perform image processing that emphasizes the frequency components of the lower spatial frequency band in the image region including the subject existing at a closer distance.

The circuit may be configured to perform image processing on an image region containing a subject existing at a closer distance by using a filter having a peak gain at a lower spatial frequency.

The circuit may be configured to reduce the frequency band to be emphasized from the image region including the subject existing at a long distance to the image region including the subject existing at a short distance.

The circuit may be configured to perform image processing with a filter having a lower peak gain in the lower spatial frequency band.

The circuit is configured to perform the image processing described above on an image region containing a subject located closer than at least one of the horizon and the horizon in the image when the image contains at least one of the horizon and the horizon. Good.

The circuit includes an image that includes a subject that is closer than at least one of the horizon and horizon in the image if the altitude of the imager is higher than a predetermined altitude and the image contains at least one of the horizon and horizon. The area may be configured to undergo image processing.

The circuit may be configured to perform the above image processing on the entire image when the altitude of the imaging device is higher than a predetermined altitude and the image includes at least one of the ground and the water surface.

In the circuit, the altitude of the image pickup device is higher than the predetermined altitude, the image includes at least one of the ground surface and the water surface, and the angle formed by the direction of the image pickup device 100 and the direction vertically downward is equal to or larger than the predetermined angle. In some cases, the entire image may be configured to undergo the above image processing.

The circuit may be configured to perform the above image processing on an image when the image pickup apparatus is moving along the imaging direction.

The circuit may be configured to perform the image processing on an image when the image pickup apparatus is moving at a speed equal to or higher than a predetermined threshold value along the image pickup direction.

The image pickup apparatus according to one aspect of the present invention may include the above-mentioned image processing apparatus and an image sensor.

The moving body according to one aspect of the present invention may be a moving body equipped with the above-mentioned imaging device.

In the image processing method according to one aspect of the present invention, image processing is performed on an image captured by an imaging device that moves along an imaging direction. The image processing method includes a step of performing image processing that emphasizes frequency components in a lower spatial frequency band in an image region including a subject existing at a closer distance.

The program according to one aspect of the present invention causes a computer to perform image processing on an image captured by an imaging device that moves along an imaging direction. The program may cause the computer to perform image processing that emphasizes the frequency components of the lower spatial frequency band in the image region containing the subject located at a closer distance.

According to one aspect of the present invention, it is possible to prevent the sharpness of the subject image from changing unnaturally according to the change in the subject distance.

The outline of the above invention does not list all the necessary features of the present invention. Sub-combinations of these feature groups can also be inventions.

An example of the appearance of the unmanned aerial vehicle (UAV) 10 and the remote control device 300 is shown. An example of the functional block of UAV10 is shown. An example of controlling the spatial frequency band emphasized by the enhancement process is shown. The frequency characteristics of the enhance filter are schematically shown. The enhancement filter applied to the subject 330 is schematically shown. It is a graph which shows typically the peak frequency in the height direction of an image. It is a flowchart which shows an example of the procedure of the enhancement process executed by the image pickup control unit 110. It is a figure which shows an example of a hardware configuration.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters that are subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent the "part" of. Specific stages and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. It may include a memory element such as.

The computer-readable medium may include any tangible device capable of storing instructions executed by the appropriate device. As a result, the computer-readable medium having the instructions stored therein will include the product, including instructions that can be executed to create means for performing the operation specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated A circuit card or the like may be included.

Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes traditional procedural programming languages. Traditional procedural programming languages are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or Smalltalk®, JAVA®, C ++. It may be an object-oriented programming language such as, and a "C" programming language or a similar programming language. Computer-readable instructions are used locally or on a local area network (LAN), wide area network (WAN) such as the Internet, to the processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing device. ) May be provided. The processor or programmable circuit may execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers and the like.

FIG. 1 shows an example of the appearance of the unmanned aerial vehicle (UAV) 10 and the remote control device 300. The UAV 10 includes a UAV main body 20, a gimbal 50, a plurality of imaging devices 60, and an imaging device 100. The gimbal 50 and the imaging device 100 are examples of an imaging system. The UAV 10 is a concept including a moving body, an air vehicle moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like. An airship that moves in the air is a concept that includes UAVs, other aircraft that move in the air, airships, helicopters, and the like.

The UAV main body 20 includes a plurality of rotor blades. The plurality of rotor blades are an example of a propulsion unit. The UAV main body 20 flies the UAV 10 by controlling the rotation of a plurality of rotor blades. The UAV body 20 flies the UAV 10 using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed-wing aircraft having no rotor blades.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The gimbal 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the imaging device 100 on a pitch axis using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the roll axis and the yaw axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two imaging devices 60 may be provided in front of the nose of the UAV 10. Yet two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two image pickup devices 60 on the front side may form a pair and function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of image pickup devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV 10 may be provided with at least one imaging device 60 on each of the nose, nose, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the image pickup device 60 may be wider than the angle of view that can be set by the image pickup device 100. The image pickup apparatus 60 may have a single focus lens or a fisheye lens.

The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may communicate wirelessly with the UAV 10. The remote control device 300 transmits instruction information indicating various commands related to the movement of the UAV 10, such as ascending, descending, accelerating, decelerating, advancing, reversing, and rotating, to the UAV 10. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to ascend the UAV 10. The UAV10 rises while accepting the rise order. Even if the UAV10 accepts the ascending command, the ascending may be restricted if the altitude of the UAV10 reaches the upper limit altitude.

FIG. 2 shows an example of the functional block of the UAV 10. The UAV 10 includes a UAV control unit 30, a memory 37, a communication interface 36, a propulsion unit 40, a GPS receiver 41, an inertial measurement unit 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a humidity sensor 46, a gimbal 50, and an imaging device. The 60 and the image pickup device 100 are provided.

The communication interface 36 communicates with another device such as the remote control device 300. The communication interface 36 may receive instruction information including various commands from the remote control device 300 to the UAV control unit 30. The memory 37 has a UAV control unit 30, a propulsion unit 40, a GPS receiver 41, an inertial measurement unit (IMU) 42, a magnetic compass 43, a barometric altimeter 44, a temperature sensor 45, a humidity sensor 46, a gimbal 50, and an imaging device 60. The program and the like necessary for controlling the image pickup device 100 are stored. The memory 37 may be a computer-readable recording medium and may include at least one of flash memory such as SRAM, DRAM, EEPROM, EEPROM, USB memory, and solid state drive (SSD). The memory 37 may be provided inside the UAV main body 20. It may be provided so as to be removable from the UAV main body 20.

The UAV control unit 30 controls the flight and imaging of the UAV 10 according to the program stored in the memory 37. The UAV control unit 30 may be composed of a CPU, a microprocessor such as an MPU, a microcontroller such as an MCU, or the like. The UAV control unit 30 controls the flight and imaging of the UAV 10 according to a command received from the remote control device 300 via the communication interface 36. The propulsion unit 40 promotes the UAV 10. The propulsion unit 40 has a plurality of rotary blades and a plurality of drive motors for rotating the plurality of rotary blades. The propulsion unit 40 rotates a plurality of rotor blades via a plurality of drive motors in accordance with a command from the UAV control unit 30 to fly the UAV 10.

The GPS receiver 41 receives a plurality of signals indicating the time transmitted from the plurality of GPS satellites. The GPS receiver 41 calculates the position (latitude and longitude) of the GPS receiver 41, that is, the position (latitude and longitude) of the UAV 10 based on the plurality of received signals. The IMU 42 detects the posture of the UAV 10. The IMU 42 detects the acceleration in the three axial directions of the front-back, left-right, and up-down of the UAV 10 and the angular velocity in the three-axis directions of pitch, roll, and yaw as the posture of the UAV 10. The magnetic compass 43 detects the nose orientation of the UAV 10. The barometric altimeter 44 detects the altitude at which the UAV 10 flies. The barometric altimeter 44 detects the barometric pressure around the UAV 10, converts the detected barometric pressure into an altitude, and detects the altitude. The temperature sensor 45 detects the ambient temperature of the UAV 10. The humidity sensor 46 detects the humidity around the UAV 10.

The image pickup apparatus 100 includes an image pickup section 102 and a lens section 200. The lens unit 200 is an example of a lens device. The image pickup unit 102 includes an image sensor 120, an image pickup control unit 110, a memory 130, and a distance measurement sensor 140. The image sensor 120 may be composed of a CCD or CMOS. The image sensor 120 captures an optical image formed through a plurality of lenses 210, and outputs the captured image to the image pickup control unit 110. The image pickup control unit 110 may be composed of a CPU, a microprocessor such as an MPU, a microcontroller such as an MCU, or the like. The image pickup control unit 110 may control the image pickup device 100 in response to an operation command of the image pickup device 100 from the UAV control unit 30. The image pickup control unit 110 is an example of a circuit. The memory 130 may be a computer-readable recording medium and may include at least one of flash memory such as SRAM, DRAM, EEPROM, EEPROM, USB memory, and solid state drive (SSD). The memory 130 stores a program or the like necessary for the image pickup control unit 110 to control the image sensor 120 or the like. The memory 130 may be provided inside the housing of the image pickup apparatus 100. The memory 130 may be provided so as to be removable from the housing of the image pickup apparatus 100.

The distance measuring sensor 140 measures the distance to the subject. The distance measuring sensor 140 may be an infrared sensor, an ultrasonic sensor, a stereo camera, a TOF (Time Of Flight) sensor, or the like.

The lens unit 200 includes a plurality of lenses 210, a plurality of lens driving units 212, and a lens control unit 220. The plurality of lenses 210 may function as a zoom lens, a varifocal lens, and a focus lens. At least a part or all of the plurality of lenses 210 are arranged so as to be movable along the optical axis. The lens unit 200 may be an interchangeable lens that is detachably provided to the imaging unit 102. The lens driving unit 212 moves at least a part or all of the plurality of lenses 210 along the optical axis via a mechanical member such as a cam ring. The lens driving unit 212 may include an actuator. The actuator may include a stepping motor. The lens control unit 220 drives the lens drive unit 212 in accordance with a lens control command from the image pickup unit 102 to move one or more lenses 210 along the optical axis direction via a mechanical member. The lens control command is, for example, a zoom control command and a focus control command.

The lens unit 200 further includes a memory 222 and a position sensor 214. The lens control unit 220 controls the movement of the lens 210 in the optical axis direction via the lens drive unit 212 in response to a lens operation command from the image pickup unit 102. The lens control unit 220 controls the movement of the lens 210 in the optical axis direction via the lens drive unit 212 in response to a lens operation command from the image pickup unit 102. Part or all of the lens 210 moves along the optical axis. The lens control unit 220 executes at least one of the zoom operation and the focus operation by moving at least one of the lenses 210 along the optical axis. The position sensor 214 detects the position of the lens 210. The position sensor 214 may detect the current zoom position or focus position.

The lens driving unit 212 may include a runout correction mechanism. The lens control unit 220 may execute the shake correction by moving the lens 210 in the direction along the optical axis or in the direction perpendicular to the optical axis via the shake correction mechanism. The lens driving unit 212 may drive the runout correction mechanism by a stepping motor to perform runout correction. The runout correction mechanism may be driven by a stepping motor to move the image sensor 120 in a direction along the optical axis or in a direction perpendicular to the optical axis to perform runout correction.

The memory 222 stores the control values of the plurality of lenses 210 that move via the lens driving unit 212. The memory 222 may include at least one of flash memories such as SRAM, DRAM, EEPROM, EEPROM, and USB memory.

The image processing function included in the image pickup apparatus 100 will be described. The image pickup control unit 110 performs enhancement processing by applying an enhancement filter to the image output from the image sensor 120. The image pickup control unit 110 controls the spatial frequency characteristic of the enhance filter applied to the image based on the flight state of the UAV 10 and the orientation of the image pickup device 100. The image pickup control unit 110 generates image data for recording by performing image processing including enhancement processing on the images sequentially output from the image sensor 120, and records the images in the memory 130.

The image pickup control unit 110 performs image processing on the image captured by the image pickup device 100 that moves along the image pickup direction. The image pickup control unit 110 performs image processing for emphasizing the frequency component of the lower spatial frequency band in the image region including the subject existing at a closer distance. Specifically, the image pickup control unit 110 performs image processing on an image region including a subject existing at a closer distance by using a filter having a peak gain at a lower spatial frequency. For example, the image pickup control unit 110 may lower the frequency band to be emphasized from the image region including the subject existing at a long distance to the image region including the subject existing at a short distance.

The image pickup control unit 110 may set the peak gain in consideration of the MTF of the lens 210. The image pickup control unit 110 may perform the above image processing using a filter having a lower peak gain in a lower spatial frequency band.

When the image includes at least one of the horizon and the horizon, the image pickup control unit 110 may perform the above image processing on an image region including a subject existing at a closer distance than at least one of the horizon and the horizon in the image.

The image pickup control unit 110 exists at a closer distance than at least one of the horizon and the horizon in the image when the altitude of the image pickup apparatus 100 is higher than a predetermined altitude and the image contains at least one of the horizon and the horizon. The above image processing may be applied to an image area including a subject. When the altitude of the image pickup apparatus 100 is higher than a predetermined altitude and the image includes at least one of the ground surface and the water surface, the image pickup control unit 110 may perform the above image processing on the entire image. When the altitude of the image pickup apparatus 100 is higher than a predetermined altitude and the image includes at least one of the ground surface and the water surface, the image pickup control unit 110 may perform the above image processing on the entire image.

The image pickup control unit 110 may perform the above-mentioned image processing on the image when the image pickup device 100 is moving along the image pickup direction. The image pickup control unit 110 may prohibit the above image processing when the image pickup device 100 is not moving along the image pickup direction. The image pickup control unit 110 may perform the above image processing on the image when the image pickup device 100 is moving along the image pickup direction at a speed equal to or higher than a predetermined threshold value. The image pickup control unit 110 may prohibit the above image processing when the speed in the direction along the image pickup direction of the image pickup apparatus 100 is less than a predetermined threshold value.

FIG. 3 shows an example of controlling the spatial frequency band emphasized by the enhancement process. The image 310 is an image output from the image sensor 120. The image 310 is an image obtained when the UAV 10 is shooting a landscape while flying. The image 310 is a so-called aerial image.

The image pickup control unit 110 performs an enhancement process by applying an enhancement filter to the image 310. The enhancement process is a process for emphasizing a spatial frequency component in a high frequency region of an image, such as an edge enhancement process or a sharpening process.

Image 310 includes a horizon 320 near the center. The image 310 includes a subject 330 located near the horizon 320 and a subject 340 located closer to the image pickup apparatus 100 than the horizon 320. Since the subject 330 exists at a long distance from the image pickup apparatus 100, it appears small in the image 310. Since the subject 340 is located closer to the image pickup apparatus 100 than the subject 330, the subject 340 is relatively large in the image 310. In general, in an image including a horizon or horizon obtained by aerial photography, the subject appears large from the vicinity of the horizon or horizon to the lower end of the image with the position of the horizon or horizon in the vertical intermediate region of the image as a boundary. The spatial frequency of is high.

When the UAV 10 takes a picture while moving forward in the horizontal direction, the subject 330 gradually becomes larger while the position where the subject 330 is captured moves from a position near the horizontal line 320 toward the lower end. The spatial frequency component of an image of the same subject shifts to the lower frequency side as the size on the image increases. Therefore, as the UAV 10 approaches the subject 330, the spatial frequency component of the subject 330 shifts to the low frequency side while the position of the subject 330 moves from near the horizon 320 toward the lower end.

Therefore, if an enhancement filter having a specific frequency characteristic is uniformly applied to the entire image captured by the image pickup apparatus 100, the main spatial frequency in the image of the subject 330 matches the spatial frequency band emphasized by the enhancement filter. In the image obtained when the subject 330 is imaged from a distance, the sharpness of the subject 330 is the highest. Therefore, when the distance from the image pickup apparatus 100 to the subject 330 becomes a specific distance, the sharpness of the image of the subject 330 becomes the highest, and the sharpness of the image of the subject 330 obtained from the positions before and after that becomes relatively high. It gets lower. As a result, if a specific enhancement filter is uniformly applied to an image taken while the UAV 10 moves forward in the horizontal direction and approaches the subject 330, an unnatural image may be obtained.

Therefore, the image pickup control unit 110 changes the spatial frequency band emphasized by the enhancement filter for each image region. Specifically, the image pickup control unit 110 reduces the spatial frequency band emphasized by the enhancement process from the image region of the long-distance subject to the image region of the short-distance subject. For example, the image pickup control unit 110 gradually lowers the spatial frequency band emphasized by the enhance filter from the position of the horizontal line 320 in the image 310 to the image region at the lower end of the image 310.

FIG. 4 schematically shows the frequency characteristics of the enhance filter. Graph 400 shows a gain curve 410 representing the spatial frequency characteristics of the enhance filter applied at the position of the horizon 320 in the image 310. Graph 420 shows a gain curve 430 representing the spatial frequency characteristics of the enhancement filter applied at the lower end of the image 310. The horizontal axis of the graphs 400 and 420 is the spatial frequency, and the vertical axis is the luminance gain.

The gain curve 430 has a peak gain G0 at a peak frequency f0. The gain curve 410 has a peak gain G1 at peak frequencies X and f0. Here, X is a value larger than 1. In this way, the image pickup control unit 110 sets the peak of the spatial frequency characteristic of the enhancement filter to be the highest at the position of the horizontal line 320 on the image and the lowest at the lower end of the image.

The filter characteristic change region 450 is a region for changing the peak frequency of the gain curve of the enhance filter. The gradation shown in the region 450 schematically shows the height of the peak frequency of the gain curve of the enhance filter by shading. The image pickup control unit 110 determines the image region below the position of the horizon 320 as the filter characteristic change region 450. As shown by the gradation of the filter characteristic change region 450, the image pickup control unit 110 sets the height of the peak frequency of the gain curve of the enhance filter from the position of the horizon 320 toward the lower end from X · f0 in the height direction. Gradually decrease to f0. The image pickup control unit 110 may uniformly apply an enhancement filter having a predetermined frequency characteristic to the region 460 other than the filter characteristic change region 450 in the image. In the present embodiment, the "height" mainly represents the distance from the lower end of the image in the image captured with the horizontal axis of the image sensor 120 parallel to the horizontal line. In general, the height may be the distance in the direction corresponding to the vertical direction in the image.

The MTF curve 402 in the graph 400 and the MTF curve 422 in the graph 420 indicate the MTF of the lens 210. The image pickup control unit 110 determines the gain G1 of the gain curve 410 and the gain G0 of the gain curve 430 in consideration of the MTF curve 402 and the MTF curve 422. For example, when the gain of the MTF curve 422 at f0 is larger than the gain of the MTF curve 402 at X / f0, G0 is made smaller than G1. In general, the MTF curve 402 and the MTF curve 422 may differ depending on the subject distance and the image height. Therefore, the image pickup control unit 110 determines the gain curve of the enhancement filter based on the MTF curve determined from the subject distance, the image height, and the optical characteristics of the lens 210. As a result, the influence of the spatial frequency characteristic of the lens can be suppressed.

The "spatial frequency band emphasized by the enhancement filter" may be a predetermined range including the peak frequency. The “spatial frequency band emphasized by the enhancement filter” may be a spatial frequency band in which the peak gain is equal to or higher than a predetermined threshold value. The “spatial frequency band emphasized by the enhancement filter” may be a weighted average value calculated by weighting the spatial frequency with a gain.

FIG. 5 schematically shows an enhancement filter applied to the subject 330. As the UAV 10 approaches the subject 330, the image sensor 120 captures images 501, 502, and 503 in this order and outputs the images to the image control unit 110.

The image pickup control unit 110 applies an enhancement filter having a spatial frequency characteristic represented by a gain curve 510 to an image region at a height at which the subject 330 exists in the image 501. The gain curve 510 has a peak frequency at the spatial frequencies X1 and f0. The image pickup control unit 110 applies an enhancement filter having a spatial frequency characteristic represented by a gain curve 520 to an image region at a height at which the subject 330 exists in the image 502 captured after the image 501. The gain curve 520 has a peak frequency at the spatial frequencies X2 and f0. The image pickup control unit 110 applies an enhancement filter having a spatial frequency characteristic represented by a gain curve 530 to an image region at a height at which the subject 330 exists in the image 503 captured after the image 503. The gain curve 530 has a peak frequency at the spatial frequencies X3 and f0. Here, 1 <X1 <X2 <X3. In this way, the image pickup control unit 110 shifts the peak frequency to the low frequency side as the size of the subject 330 in the image increases, so that it is possible to suppress an unnatural change in the sharpness of the subject 330. ..

The closer to the lower end of the image, the faster the change in the position of the subject 330 in the image. Therefore, the image pickup control unit 110 may increase the intensity of the enhancement filter applied to the image region closer to the lower end of the image. For example, the image pickup control unit 110 may increase the peak gain of the enhancement filter applied to the image region closer to the lower end of the image.

FIG. 6 is a graph schematically showing the peak frequency in the height direction of the image. The horizontal axis of the graph of FIG. 5 is the height, and the vertical axis is the peak frequency. The height indicates the distance from the bottom edge of the image.

As shown in FIG. 6, the imaging control unit 110 linearly reduces the peak frequency from X · f0 to f0 between the height H in which the horizon is reflected and the lower end of the image. The imaging control unit 110 may set the peak frequency represented by the linear function of the height, at least approximately approximately.

The imaging control unit 110 may set the peak frequency represented by the number of higher-order functions of the height, at least approximately approximately. The image pickup control unit 110 may continuously reduce the peak frequency according to the height. The image pickup control unit 110 may change the peak frequency in steps. The imaging control unit 110 may set the peak frequency represented by the step function of the height, at least approximately approximately. The image pickup control unit 110 applies an enhancement filter having a first peak frequency to the first image region, and applies a first peak frequency to a second image region including a subject closer than the first image region. An enhancement filter with a lower second peak frequency may be applied.

As a specific example has been described in relation to FIGS. 3 to 6 and the like, when the image 310 obtained by aerial photography is enhanced, the image pickup control unit 110 is directed from the position of the horizontal line 320 on the image toward the lower end. The spatial frequency emphasized by the enhance filter is shifted from the position of the horizon 320 toward the lower end to the low frequency side so as to match the image characteristic that the spatial frequency of the subject decreases while the position of the subject changes. Let me. This makes it possible to prevent the sharpness of a specific subject from changing unnaturally.

In addition, in FIGS. 3 to 6, the case of taking an image while approaching the subject 330 has been taken up and described. However, the same enhancement processing can be applied when shooting while moving away from the subject 330.

FIG. 7 is a flowchart showing an example of the enhancement processing procedure executed by the image pickup control unit 110. The imaging control unit 110 executes the processing of the flowchart shown in FIG. 7 at any time while the imaging device 100 is performing imaging.

In S702, the image pickup control unit 110 determines whether or not the altitude of the UAV 10 is equal to or higher than the threshold value. For example, a value of 2 m or more may be applied as the threshold value. When the altitude of the UAV 10 is not equal to or higher than the threshold value, in S720, the image pickup control unit 110 performs the enhancement process by a predetermined method. For example, the image pickup control unit 110 performs the enhancement process by uniformly applying an enhancement filter having a predetermined spatial frequency characteristic to the entire image.

When the image pickup control unit 110 determines in S702 that the altitude of the UAV 10 is equal to or higher than the threshold value, in S704, the image pickup control unit 110 determines whether or not the subject includes at least one of the horizon and the horizon. The image pickup control unit 110 may determine whether or not the subject includes at least one of the horizon and the horizon by image analysis. The image pickup control unit 110 may determine whether or not the subject includes at least one of the horizon and the horizon based on the position of the UAV 10 and the map information and the image pickup range of the image pickup apparatus 100.

When the image pickup control unit 110 determines in S704 that the subject includes at least one of the horizon and the horizon, in S706, the horizon or the horizon extends to the image region from the position of the horizon or the horizon to the position including the short-distance subject in the image. The enhancement process is performed by applying an enhancement filter whose peak frequency decreases from the position of 1 to the position including a short-distance subject.

When the image pickup control unit 110 determines in S704 that the subject does not include either the horizon or the horizon, it determines whether or not at least one of the ground and the water surface is included. The image pickup control unit 110 may determine whether or not the subject includes at least one of the ground surface and the water surface by image analysis. The image pickup control unit 110 may determine whether or not the subject includes at least one of the ground surface and the water surface based on the position of the UAV 10 and the map information and the image pickup range of the image pickup apparatus 100.

When the image pickup control unit 110 determines in S710 that the subject does not include either the ground or the water surface, the process shifts to S720. When the image pickup control unit 110 determines in S710 that the subject includes at least one of the ground and the water surface, in S712, the image pickup control unit 110 determines whether or not the image pickup direction of the image pickup apparatus 100 is more horizontal than the vertical direction. .. For example, in the image pickup control unit 110, the angle formed by the image pickup direction of the image pickup apparatus 100 and the vertically downward direction is equal to or higher than a predetermined value based on the posture of the UAV 10 and the yaw angle and pitch angle controlled by the gimbal 50. In some cases, it may be determined that the imaging direction of the imaging device 100 is more horizontal than the vertical direction.

When the image pickup control unit 110 determines in S712 that the image pickup direction of the image pickup apparatus 100 is not oriented horizontally from the vertical direction, the image pickup control unit 110 shifts to S720-like processing. When the image pickup control unit 110 determines in S712 that the image pickup direction of the image pickup apparatus 100 is more horizontal than the vertical direction, in S714, from the position including the subject on the long distance side to the position including the subject on the short distance side. The enhancement process is performed by applying an enhancement filter that reduces the peak frequency to the entire image.

In this way, the image pickup control unit 110 controls the spatial frequency characteristic of the enhance filter and the image region to which the enhance filter is applied according to the flight state of the UAV 10 and the attitude of the gimbal 50 on which the image pickup device is mounted. This makes it possible to generate an image that looks natural.

As described above, the image pickup control unit 110 performs image processing having different spatial frequency characteristics from an image region including a subject existing at a long distance to an image region including a subject existing at a short distance. Thereby, for example, appropriate image processing can be applied to an image obtained by aerial photography. Although the image processing applied to the image obtained by aerial photography has been described in relation to FIGS. 3 to 7, the same applies to images obtained in various situations where the subject distance changes, not limited to aerial photography. Image processing may be applied.

FIG. 8 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with the device according to an embodiment of the present invention or as one or more "parts" of the device. Alternatively, the program may cause the computer 1200 to perform the operation or one or more "parts". The program can cause a computer 1200 to perform a process or a step of the process according to an embodiment of the present invention. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

Communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation, and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, USB memory or IC card or network. The program is installed in RAM 1214 or ROM 1230, which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the operation or processing of information according to the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing on the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The received data received from the network is written to the reception buffer area or the like provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. Good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214 and specified by the instruction sequence of the program. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, a database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for the entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or in a computer readable storage medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided in a dedicated communication network or a server system connected to the Internet can be used as a computer-readable storage medium, thereby allowing the program to be transferred to the computer 1200 over the network. provide.

The execution order of each process such as operation, procedure, step, and step in the device, system, program, and method shown in the claims, the specification, and the drawing is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

10 UAV
20 UAV main unit 30 UAV control unit 36 Communication interface 37 Memory 40 Propulsion unit 41 GPS receiver 42 Inertial measurement unit 43 Magnetic compass 44 Atmospheric pressure sensor 45 Temperature sensor 46 Humidity sensor 50 Gimbal 60 Imaging device 100 Imaging device 102 Imaging unit 110 Imaging control unit 120 Image sensor 130 Memory 140 Distance measurement sensor 200 Lens unit 210 Lens 212 Lens drive unit 214 Position sensor 220 Lens control unit 222 Memory 300 Remote control device 310 Image 320 Horizontal line 330 Subject 340 Subject 400 Graph 410 Gain curve 402 MTF curve 420 Graph 422 MTF curve 430 Gain curve 450 Filter characteristic change area 501, 502, 503 Image 510, 520, 530 Gain curve 1200 Computer 1210 Host controller 1212 CPU
1214 RAM
1220 Input / Output Controller 1222 Communication Interface 1230 ROM

Claims (14)

Equipped with a circuit that performs image processing on the image captured by the imaging device that moves along the imaging direction.
The circuit is an image processing device configured to perform image processing for emphasizing frequency components in a lower spatial frequency band in an image region including a subject existing at a closer distance. The image processing apparatus according to claim 1, wherein the circuit is configured to perform the image processing on an image region including a subject existing at a closer distance by using a filter having a peak gain at a lower spatial frequency. The image according to claim 1 or 2, wherein the circuit is configured to reduce the frequency band to be emphasized from an image region including a subject existing at a long distance to an image region including a subject existing at a short distance. Processing equipment. The image processing apparatus according to claim 2, wherein the circuit is configured to perform image processing using a filter having a lower peak gain in a lower spatial frequency band. The circuit is configured to perform the image processing on an image region including a subject existing at a closer distance than at least one of the horizon and the horizon in the image when the image includes at least one of the horizon and the horizon. The image processing apparatus according to claim 1 or 2. The circuit exists at a closer distance than at least one of the horizon and the horizon in the image when the altitude of the image pickup apparatus is higher than a predetermined altitude and the image contains at least one of the horizon and the horizon. The image processing apparatus according to claim 5, wherein the image processing is performed on an image region including a subject to be processed. The circuit is configured to perform the image processing on the entire image when the altitude of the image pickup apparatus is higher than a predetermined altitude and the image includes at least one of the ground surface and the water surface. The image processing apparatus according to 5. In the circuit, the altitude of the image pickup device is higher than a predetermined altitude, the image includes at least one of the ground surface and the water surface, and the angle formed by the direction of the image pickup device and the vertically downward direction is predetermined. The image processing apparatus according to claim 7, wherein the image processing is performed on the entire image when the angle is equal to or larger than that of the image. The image processing apparatus according to claim 1 or 2, wherein the circuit is configured to perform the image processing on the image when the imaging apparatus is moving along an imaging direction. The circuit according to claim 1 or 2, wherein the image processing is performed on the image when the image pickup device is moving at a speed equal to or higher than a predetermined threshold value along the image pickup direction. Image processing equipment. Image sensor and
An imaging device including the image processing device according to any one of claims 1 to 10. A moving body that moves with the imaging device according to claim 11. An image processing method that performs image processing on an image captured by an imaging device that moves along the imaging direction.
An image processing method comprising a step of performing image processing for emphasizing frequency components in a lower spatial frequency band in an image region including a subject existing at a closer distance. A program for causing a computer to perform image processing on an image captured by an imaging device that moves along the imaging direction.
A program that causes the computer to function in a step of performing image processing that emphasizes frequency components in a lower spatial frequency band in an image region including a subject existing at a closer distance.

Images