JP6746857B2 - Image processing device, imaging device, unmanned aerial vehicle, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an unmanned aerial vehicle, an image processing method, and a program.

Patent Document 1 describes a technique for correcting aberration of a pixel having input screen coordinates with a distortion correction parameter table that holds position coordinate data of each pixel corresponding to a lens parameter.
Patent Document 1 JP 2011-61444 A

When a lens included in an optical system such as a focus lens is moved, an unintended view angle variation may occur due to distortion or the like.

An image processing apparatus according to an aspect of the present invention includes an acquisition unit that acquires information indicating a plurality of positions of a lens that moves during an exposure period of an image sensor. The image processing apparatus includes a correction unit that corrects the image acquired by exposing the image sensor through the lens during the exposure period based on the plurality of positions.

The correction unit may calculate an average position of the lens within the exposure period based on the plurality of positions and correct the image distortion based on the calculated average position of the lens.

The correction unit may correct the distortion of the image based on the average position of the lens and the distortion coefficient associated with the position of the lens.

The correction unit calculates a plurality of pixel positions corresponding to each of the plurality of pixels on the image for each of the plurality of positions and the plurality of pixel combinations based on the plurality of positions and the positions of the plurality of pixels included in the image sensor. Then, the image may be corrected by calculating the average position of the plurality of pixel positions for each of the plurality of pixels.

The image sensor may be exposed for different exposure periods for each of a plurality of pixel columns included in the image sensor. The acquisition unit may acquire information indicating a plurality of positions of the lens that moves during each exposure period of the plurality of pixel columns. The correction unit may correct the image acquired by each of the plurality of pixel columns based on the plurality of positions of the plurality of pixel columns during each exposure period.

The lens may be a focus lens that is movable in the optical axis direction.

The lens may reciprocate in the optical axis direction during the exposure period to adjust the focus of the optical system including the lens. The acquisition unit may acquire information indicating a plurality of positions of the lens in the reciprocating movement during the exposure period.

The lens may move in one direction along the optical axis during the exposure period to adjust the focus of the optical system including the lens. The acquisition unit may acquire information indicating a plurality of positions of the lens during movement in one direction during the exposure period.

The exposure period may be a period in which the image sensor is repeatedly exposed to obtain each of a plurality of moving image constituent images that form a moving image. The correction unit may correct each moving image constituent image based on the plurality of positions in the exposure period for acquiring each of the plurality of moving image constituent images.

An imaging device according to one aspect of the present invention may include the image processing device. The imaging device may include an image sensor.

The lens may be a focus lens that is movable in the optical axis direction. The imaging device may include a focus adjustment unit that adjusts the focus of the optical system including the lens based on the image corrected by the correction unit.

An unmanned aerial vehicle according to an aspect of the present invention includes the above imaging device and moves.

An image processing method according to an aspect of the present invention includes a step of acquiring information indicating a plurality of positions of a lens that moves during an exposure period of an image sensor. The image processing method includes a step of correcting an image acquired by exposing the image sensor through a lens during an exposure period based on the plurality of positions.

The program according to one aspect of the present invention may be a program for causing a computer to function as the above-described image processing device.

According to one aspect of the present invention, it is possible to suppress the influence of the change in the angle of view due to the movement of the lens.

Note that the above summary of the invention does not enumerate all necessary features of the present invention. Further, a sub-combination of these feature groups can also be an invention.

It is a figure which shows an example of the external appearance perspective view of the imaging device 100 which concerns on this embodiment. It is a figure which shows the functional block of the imaging device 100 which concerns on this embodiment. FIG. 6 is a diagram schematically illustrating a process of the correction unit 140. An example of a distortion coefficient is shown. An example of the positional relationship between the coordinates on the image sensor 120 and the image coordinates (x, y) is schematically shown. FIG. 6 is a diagram illustrating a method of calculating an average value of positions of a focus lens 210. 9 is a flowchart showing an example of a procedure executed by the imaging device 100. It is a figure explaining the other correction method which the correction|amendment part 140 performs. It is a figure which roughly explains the correction process at the time of reading pixel data by rolling reading. 1 illustrates an unmanned aerial vehicle (UAV) equipped with the imaging device 100. 1 illustrates an example computer 1200 in which aspects of the present invention may be embodied in whole or in part.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Further, not all combinations of the features described in the embodiments are essential to the means for solving the invention. It is apparent to those skilled in the art that various modifications and improvements can be added to the following embodiments. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

The claims, the description, the drawings, and the abstract contain the subject matter of copyright protection. The copyright owner has no objection to the reproduction of any of these documents by anyone as it appears in the JPO file or record. However, in all other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where a block is (1) a stage of a process in which an operation is performed or (2) a device responsible for performing an operation. "Part" of may be represented. Particular stages and "sections" may be implemented by programmable circuits and/or processors. Dedicated circuitry may include digital and/or analog hardware circuitry. It may include integrated circuits (ICs) and/or discrete circuits. Programmable circuits may include reconfigurable hardware circuits. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. Memory elements, etc. may be included.

Computer-readable media may include any tangible device capable of storing instructions executed by a suitable device. As a result, a computer-readable medium having instructions stored therein will comprise a product that includes instructions that may be executed to create the means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer readable media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), Electrically erasable programmable read only memory (EEPROM), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (registered trademark) disc, memory stick, An integrated circuit card or the like may be included.

Computer readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes conventional procedural programming languages. Conventional procedural programming languages include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or Smalltalk, JAVA, C++, etc. It may be an object-oriented programming language, and the "C" programming language or similar programming languages. Computer readable instructions can be for a general purpose computer, a special purpose computer, or another programmable data processing device processor or programmable circuit, either locally or in a wide area network (WAN) such as a local area network (LAN), the Internet, or the like. ). The processor or programmable circuit may execute computer readable instructions to create a means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

FIG. 1 is a diagram showing an example of an external perspective view of the image pickup apparatus 100 according to the present embodiment. FIG. 2 is a diagram showing functional blocks of the image pickup apparatus 100 according to the present embodiment.

The image pickup apparatus 100 includes an image pickup unit 102 and a lens unit 200. The image capturing section 102 includes an image sensor 120, an image processing section 104, an image capturing control section 110, a memory 130, an instruction section 162, and a display section 160.

The image sensor 120 is an image sensor such as CCD or CMOS. The image sensor 120 receives light via the optical system included in the lens unit 200. The image sensor 120 outputs image data of an optical image formed via the optical system of the lens unit 200 to the image processing unit 104.

The imaging control unit 110 and the image processing unit 104 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The memory 130 may be a computer-readable recording medium and may include at least one of SRAM, DRAM, EPROM, EEPROM, and flash memory such as USB memory. The memory 130 stores a program necessary for the imaging control unit 110 to control the image sensor 120 and the like, a program necessary for the image processing unit 104 to perform image processing, and the like. The memory 130 may be provided inside the housing of the imaging device 100. The memory 130 may be provided so as to be removable from the housing of the imaging device 100.

The instruction unit 162 is a user interface that receives an instruction for the imaging device 100 from a user. The display unit 160 displays an image captured by the image sensor 120 and processed by the image processing unit 104, various setting information of the image capturing apparatus 100, and the like. The display unit 160 may include a touch panel.

The imaging control unit 110 controls the lens unit 200 and the image sensor 120. The imaging control unit 110 controls the adjustment of the focal position and the focal length of the optical system included in the lens unit 200. The imaging control unit 110 controls the lens unit 200 by outputting a control command to the lens control unit 220 included in the lens unit 200 based on the information indicating the instruction from the user.

The lens unit 200 includes a focus lens 210, a zoom lens 211, a lens driving unit 212, a lens driving unit 213, a lens control unit 220, a memory 222, a position sensor 214, and a position sensor 215. The focus lens 210 and the zoom lens 211 may include at least one lens. The focus lens 210 and the zoom lens 211 are lenses included in the optical system. At least a part or all of the focus lens 210 and the zoom lens 211 are arranged so as to be movable along the optical axis of the optical system.

The imaging control unit 110 includes a focus adjustment unit 112. The focus adjustment unit 112 controls the focus lens 210 to adjust the focus of the optical system included in the lens unit 200.

The lens unit 200 may be an interchangeable lens that is detachably attached to the imaging unit 102. The lens driving unit 212 may include a driving device such as a stepping motor, a DC motor, a coreless motor, or an ultrasonic motor. At least a part or all of the focus lens 210 moves by receiving power from a driving device included in the lens driving unit 212 via a mechanical member such as a cam ring and a guide shaft. The lens driving unit 213 may include a driving device such as a stepping motor, a DC motor, a coreless motor, or an ultrasonic motor. At least a part or all of the zoom lens 211 moves by receiving power from a driving device included in the lens driving unit 213 via a mechanical member such as a cam ring and a guide shaft.

The lens control unit 220 drives at least one of the lens driving unit 212 and the lens driving unit 213 in accordance with a lens control command from the imaging unit 102, and causes at least one of the focus lens 210 and the zoom lens 211 to emit light through a mechanical member. By moving along the axial direction, at least one of zoom operation and focus operation is executed. The lens control command is, for example, a zoom control command and a focus control command.

The memory 222 stores control values for the focus lens and the zoom lens that move via the lens driving unit 212. The memory 222 may include at least one of SRAM, DRAM, EPROM, EEPROM, and flash memory such as USB memory.

The position sensor 214 detects the position of the focus lens 210. The position sensor 215 detects the position of the zoom lens 211. The position sensor 214 and the position sensor 215 may be a magnetoresistive (MR) sensor or the like.

The imaging control unit 110 outputs a control command to the image sensor 120 based on the information indicating the instruction from the user through the instruction unit 162 and the like, thereby executing the control including the control of the imaging operation on the image sensor 120. The image captured by the image sensor 120 is processed by the image processing unit 104 and stored in the memory 130.

The image acquired by the image sensor 120 is input to the image processing unit 104. The correction unit 140 corrects the image acquired by the image sensor 120. The display unit 160 displays the image corrected by the correction unit 140. The memory 130 stores the image corrected by the correction unit 140. The image corrected by the correction unit 140 may be transferred from the memory 130 to a recording medium such as a memory card.

The image processing unit 104 includes a correction unit 140 and an acquisition unit 142. The acquisition unit 142 acquires information indicating a plurality of positions of the focus lens 210 that moves during the exposure period of the image sensor 120. For example, the acquisition unit 142 acquires information indicating a plurality of positions of the focus lens 210 from the focus adjustment unit 112. The correction unit 140 corrects the image acquired by exposing the image sensor 120 through the focus lens 210 during the exposure period based on the plurality of positions.

For example, the correction unit 140 calculates an average position of the focus lens 210 within the exposure period based on the acquired plurality of positions of the focus lens 210, and based on the calculated average position of the focus lens 210, Correct image distortion. Specifically, the correction unit 140 corrects the image distortion based on the average position of the focus lens 210 and the distortion coefficient associated with the position of the focus lens 210.

The correction unit 140, based on the plurality of positions of the focus lens 210 and the plurality of pixels of the image sensor 120, displays on the image for each combination of the plurality of positions of the focus lens 210 and the plurality of pixels of the image sensor 120. A plurality of pixel positions corresponding to each of the plurality of pixels may be calculated. The correction unit 140 may correct the image by calculating an average position of the calculated plurality of pixel positions for each of the plurality of pixels.

The image sensor 120 may be exposed for different exposure periods for each of a plurality of pixel columns included in the image sensor 120. For example, the imaging control unit 110 may read pixel information from the image sensor 120 by rolling reading. In this case, the acquisition unit 142 acquires information indicating a plurality of positions of the focus lens 210 that moves during each exposure period of a plurality of pixel columns. The correction unit 140 corrects the image acquired by each of the plurality of pixel rows based on the plurality of positions of the plurality of pixel rows during each exposure period.

The focus lens 210 reciprocates in the optical axis direction during the exposure period in order to adjust the focus of the optical system including the focus lens 210. For example, the focus adjustment unit 112 causes the focus lens 210 to perform a wobbling operation during the exposure period. The acquisition unit 142 acquires information indicating a plurality of positions of the focus lens 210 during the reciprocating movement during the exposure period. The correction unit 140 corrects the image based on the plurality of positions of the focus lens 210 during the reciprocating movement during the exposure period.

The focus lens 210 may move in one direction along the optical axis during the exposure period in order to adjust the focus of the optical system including the focus lens 210. The acquisition unit 142 acquires information indicating a plurality of positions of the focus lens 210 during movement in one direction during the exposure period. The correction unit 140 corrects the image based on the plurality of positions of the focus lens 210 in the movement in one direction during the exposure period.

The exposure period may be a period in which the image sensor 120 is repeatedly exposed to acquire each of a plurality of moving image constituent images that form a moving image. The correction unit 140 may correct each moving image constituent image based on the plurality of positions in the exposure period for acquiring each of the plurality of moving image constituent images.

The focus adjustment unit 112 adjusts the focus of the optical system including the focus lens 210 based on the image corrected by the correction unit 140. For example, the focus adjustment unit 112 determines the position of the focus lens 210 in the optical axis direction based on the contrast value of the image corrected by the correction unit 140, and moves the focus lens 210 to the determined position.

FIG. 3 is a diagram schematically illustrating the process of the correction unit 140. With reference to FIG. 3, a process of the correction unit 140 when the exposure periods of all the horizontal pixel columns included in the image sensor 120 are the same will be described. Specifically, a process in the case where images are continuously picked up by the global reading of the image sensor 120 will be described. In this embodiment, the image sensor 120 has N (N is a natural number) horizontal pixel rows.

The imaging control unit 110 exposes the horizontal pixel row 1 to the horizontal pixel row N included in the image sensor 120 at a timing based on the vertical synchronization signal VD. FIG. 3 shows an exposure period from time t1 to time t7 and an exposure period from time t9 to time t15.

The focus adjustment unit 112 detects the position of the focus lens 210 in the optical axis direction a plurality of times within the period from the trigger of the vertical synchronization signal VD to the next vertical synchronization signal VD. Specifically, the focus adjustment unit 112 detects the position of the focus lens 210 in the optical axis direction a predetermined number of times at a predetermined time interval from the vertical synchronization signal VD at time t0. Further, the focus adjustment unit 112 detects the position of the focus lens 210 in the optical axis direction a predetermined number of times at a predetermined time interval from the vertical synchronization signal VD at time t7.

As shown in FIG. 3, the lens controller 220 detects LP1, LP2, LP3, LP4, LP5, LP6, and LP7 in the exposure period from time t1 to time t7. In addition, the lens control unit 220 detects LP9, LP10, LP11, LP12, LP13, LP14, and LP15 in the exposure period from time t9 to time t15. LPi (i is a natural number) indicates the position of the focus lens 210 in the optical axis direction.

In response to the vertical synchronization signal VD at time t7, the focus adjustment unit 112 outputs lens position information indicating LP1, LP2, LP3, LP4, LP5, LP6, and LP7 detected during the exposure period from time t1 to t7 to the lens. It is acquired from the control unit 220 and output to the image processing unit 104. The acquisition unit 142 acquires the lens position information output from the focus adjustment unit 112.

The correction unit 140 calculates the average value of LP1, LP2, LP3, LP4, LP5, LP6, and LP7 based on the lens position information. The correction unit 140 calculates the distortion coefficient based on the average value of LP1, LP2, LP3, LP4, LP5, LP6, and LP7. The distortion coefficient is information that represents distortion that is determined from the position of the focus lens 210. The distortion coefficient will be described with reference to FIG. The average value is a value calculated by dividing the sum of LP1 to LP7 by the time from time t1 to time t7. A specific method of calculating the average value will be described with reference to FIG.

The image processing unit 104 acquires the pixel data 310 of the horizontal pixel row 1 to the horizontal pixel row N from the image sensor 120 according to the vertical synchronization signal VD at time t7. The image processing unit 104 corrects the image of the pixel data 310 based on the pixel data 310 acquired from the image sensor 120 and the distortion coefficient corresponding to the average value of the position of the focus lens 210 to generate a corrected image 311. .. The corrected image 311 generated by the correction unit 140 is stored in the memory 130 and output to the display unit 160 as an image for display on the display unit 160.

Similarly, according to the vertical synchronization signal VD at time t7, the focus adjustment unit 112 causes the lens positions indicating LP9, LP10, LP11, LP12, LP13, LP14, and LP15 detected during the exposure period from time t9 to t15. The information is acquired from the lens control unit 220 and output to the image processing unit 104. The acquisition unit 142 acquires the lens position information output from the focus adjustment unit 112.

The correction unit 140 calculates the average value of LP9, LP10, LP11, LP12, LP13, LP14, and LP15 based on the lens position information. The correction unit 140 calculates the distortion coefficient based on the average value of LP9, LP10, LP11, LP12, LP13, LP14, and LP15.

The image processing unit 104 acquires the pixel data 320 of the horizontal pixel row 1 to the horizontal pixel row N from the image sensor 120 according to the vertical synchronization signal VD at time t15. The image processing unit 104 corrects the image of the pixel data 320 based on the pixel data 320 acquired from the image sensor 120 and the distortion coefficient corresponding to the average value of the position of the focus lens 210 to generate a corrected image 321. .. The corrected image 321 generated by the correction unit 140 is stored in the memory 130 and output to the display unit 160 as an image for display on the display unit 160.

The imaging apparatus 100 executes the one-time exposure process, the pixel data read process, the image correction process, and the process of displaying on the display unit 160, as described above, according to the vertical synchronization signal VD. The image pickup apparatus 100 repeats the above-described processing for each vertical synchronization signal VD.

FIG. 4 shows an example of the distortion coefficient. The horizontal axis of FIG. 4 is the position of the focus lens 210, and the vertical axis is the value of the distortion coefficient. The distortion coefficient includes k1, k2, and k3. The memory 130 stores distortion coefficient data indicating the dependence of the focus lens 210 on k1, k2, and k3. The distortion coefficient data may be calculated in advance based on the lens design data of the optical system included in the lens unit 200 and stored in the memory 130. The distortion coefficient data may be preliminarily calculated by experiments and stored in the memory 130. The distortion coefficient data of k1, k2, and k3 may be data representing a function having the position of the focus lens 210 as a variable. The function may be obtained by fitting the distortion coefficients k1, k2, and k3 calculated in advance to a function having the position of the focus lens 210 as a variable. The distortion coefficient data of k1, k2, and k3 may be map data that maps the position of the focus lens 210 to k1, k2, and k3.

The relationship between the coordinates (x _distorted , y _distorted ) on the image sensor 120 and the normalized image coordinates (x, y) is expressed by the following equation 1 using distortion coefficients k1, k2, and k3.

The coordinates (x, y) represent coordinates in the corrected image. LP in Expression 1 represents the position of the focus lens 210. Since the distortion coefficients k1, k2, and k3 depend on LP, they are represented as k1(LP), k2(LP), and k3(LP) in Expression 1. FIG. 5 schematically shows an example of the positional relationship between the coordinates (x _distorted , y _distorted ) on the image sensor 120 and the image coordinates (x, y).

The correction unit 140 applies the coordinates of each pixel of the pixel data to (x _distorted , y _distorted ) in Expression 1 to calculate k1, k2, and k3 calculated from the average value of the position of the focus lens 210 and the distortion coefficient data. Coordinates (x, y) satisfying the equation 1 are calculated by applying them to k1 (LP), k2 (LP), and k3 (LP), respectively. Then, the correction unit 140 generates a corrected image by applying the pixel value of the coordinate (x _distorted , y _distorted ) in the pixel data acquired from the image sensor 120 as the pixel value of the coordinate (x, y).

As described with reference to FIG. 3, FIG. 4, and FIG. 5, the correction unit 140 corrects the image using the distortion coefficient determined from the average position of the focus lens 210 during the exposure period. As a result, as shown in FIG. 3, it is possible to generate the corrected image 311 and the corrected image 321 in which the change in the angle of view caused by the movement of the focus lens 210 is suppressed.

FIG. 6 is a diagram for explaining a method of calculating the average value of the positions of the focus lens 210. In FIG. 6, T is the exposure time. That is, Te is the time from t1 to t7. Td is a time interval for detecting the position of the focus lens 210. In the example shown in FIG. 6, T=6Td. The average value of the position of the focus lens 210 is a value calculated by the time average of LP1 to LP7.

Specifically, the average value of the position of the focus lens 210 is calculated by dividing the weighted sum of LP1 to LP7 by T. The weighted sum of LP1 to LP7 is calculated by Σαi×LPi. i is a natural number from 1 to 7.

αi is a weighted sum weighting coefficient. The sum of α1 to α7 is T. αi is determined based on the time of the period included in the exposure period in the period from time ti-Td/2 to time ti+Td/2. Specifically, α1 and α7 are Td/2, and α2 to α6 are Td. In this way, by determining the weighting coefficient αi according to the timing at which the position of the focus lens 210 is detected, the temporal average value of the position of the focus lens 210 can be calculated.

The average value may be a value calculated by dividing the sum of LP1 to LP7 by 7. That is, the average value may be a value calculated by dividing the sum of the positions of the focus lens 210 detected during the exposure period by the number of detections of the position of the focus lens 210.

FIG. 7 is a flowchart showing an example of a procedure executed by the image capturing apparatus 100. This flowchart is started when the vertical synchronizing signal VD is triggered.

In step S600, the focus adjustment unit 112 detects the position of the focus lens 210 by outputting a timing signal for detecting the position of the focus lens 210 to the lens control unit 220.

In S600, the imaging control unit 110 starts the exposure of the image sensor 120 when a predetermined time has elapsed from the vertical synchronization signal VD. In step S604, the lens control unit 220 detects the position of the focus lens 210 according to the timing signal output from the focus adjustment unit 112.

In step S606, the imaging control unit 110 determines whether to end the exposure. For example, the imaging control unit 110 determines to end the exposure when detecting the trigger of the new vertical synchronization signal VD. The process of S604 is repeated while exposing the image sensor 120 until it is determined that the exposure is finished.

If it is determined in step S606 that the exposure is to be ended, in step S608, the correction unit 140 acquires the lens position information of the focus lens 210 from the lens control unit 220 through the focus adjustment unit 112, and the average value of the positions of the focus lens 210. To calculate.

In S610, the correction unit 140 acquires the pixel data output from the image sensor 120. In S612, the correction unit 140 calculates the image coordinates (x, y) corresponding to the coordinates (x _distorted , y _distorted ) of each pixel of the image sensor 120 based on the distortion coefficient data.

In step S614, the correction unit 140 generates a corrected image by reflecting the pixel value of each coordinate of the pixel data output from the image sensor 120 as the pixel value of the image coordinate calculated in step S612.

In S616, the image processing unit 104 stores the corrected image generated by the correction unit 140 in the memory 130. When the processing of S616 is completed, the processing of this flowchart ends.

The corrected image stored in the memory 130 in S616 is output to the display unit 160 as a moving image constituent image for display, for example. When the above correction process is performed during moving image shooting, the corrected image stored in the memory 130 is recorded in the memory 130 as a moving image constituent image of moving image data after the moving image shooting is completed.

The correction unit 140 may store the image coordinates calculated in S612 in the memory 130 in association with the average value of the position of the focus lens 210. If the difference between the newly calculated average value of the position of the focus lens 210 and the average value stored in the memory 130 is smaller than a predetermined value, the correction unit 140 replaces the processing of S612 and replaces the memory 130. The image coordinates stored in can be used. This may reduce the amount of calculation for calculating the image coordinates.

As described with reference to FIGS. 3 to 7, the correction unit 140 corrects the image distortion based on the average value of the position of the focus lens 210 during the exposure period. Accordingly, it is possible to generate a corrected image in which the influence of the change in the angle of view due to the movement of the focus lens 210 during the exposure period is suppressed.

FIG. 8 is a diagram illustrating another correction method executed by the correction unit 140. FIG. 8 shows the relationship between the pixel coordinates (x _distorted , y _distorted ) of the image sensor 120 and the corrected image coordinates (x, y) with a part of the image coordinates enlarged.

The correction unit 140 calculates the image coordinates (x, y) based on each of the positions of the focus lens 210 detected during the exposure period. Hereinafter, correction processing based on another correction method will be described with reference to the example shown in FIG.

The correction unit 140 calculates the image coordinates (x, y) from each of LP1, LP2, LP3, LP4, LP5, LP6, and LP7 in the exposure period from time t1 to time t7. Specifically, the correction unit 140 applies the coordinates of the pixel in the pixel data acquired from the image sensor 120 to (x _distorted , y _distorted ) of Expression 1, and calculates k1, k2 calculated from LP1 and the distortion coefficient data, and k3 is applied to k1(LP), k2(LP), and k3(LP) in Equation 1, respectively, to calculate the coordinates (x, y) that satisfy Equation 1. The calculated coordinates (x, y) are indicated by (x1, y1) in FIG.

Similarly for LP2, the correction unit 140 applies the coordinates of the pixel in the pixel data to (x _distorted , y _distorted ) in Equation 1 to calculate k1, k2, and k3 calculated from LP2 and the distortion coefficient data, respectively, using Equation 1 To k1(LP), k2(LP), and k3(LP), to calculate the coordinates (x, y) that satisfy Expression 1. The calculated coordinates (x, y) are indicated by (x2, y2) in FIG. Similarly, for LP3, LP4, LP5, LP6, and LP7, Expression 1 is satisfied by using LPi (i is a natural number from 3 to 7) and k1, k2, and k3 calculated from the distortion coefficient data (x, y ) Is calculated and calculated as (xi, yi) corresponding to each LPi.

The correction unit 140 _applies the pixel value of the coordinates (x _distorted , y _distorted ) in the pixel data as the pixel value of the average coordinates (x, y) of (xi, yi) (i is a natural number from 1 to 7). The correction unit 140 performs the same process on each pixel of the pixel data acquired from the image sensor 120 to generate a corrected image.

The correction method described with reference to FIG. 8 can also provide a corrected image in which the influence of the change in the angle of view due to the movement of the focus lens 210 during the exposure period is suppressed. In some cases, it is possible to generate a corrected image in which the influence of the change in the angle of view is further reduced as compared with the correction method described with reference to FIGS.

FIG. 9 is a diagram schematically illustrating a correction process of the correction unit 140 when pixel data is read from the image sensor 120 by rolling read. When performing the rolling reading, the imaging control unit 110 sequentially shifts the exposure start time from the horizontal pixel row 1 to the horizontal pixel row N to perform exposure. Accordingly, the exposure period of the image sensor 120 differs for each pixel column included in the image sensor 120.

As shown in FIG. 9, the exposure period of the horizontal pixel row 1 is from time t1 to time t7. The exposure period of the horizontal pixel row N is from time t7 to time t13. In general, the exposure period of the horizontal pixel row 1+i is from time t1+δT×(i−1) to time t7+δT×(i−1) (i is a natural number from 1 to N). δT is an interval between exposure start times of adjacent horizontal pixel rows.

The positions of the focus lens 210 detected during the exposure period of the horizontal pixel row 1 are LP1, LP2, LP3, LP4, LP5, LP6, and LP7. The correction unit 140 corresponds to the average value of LP1, LP2, LP3, LP5, LP6, and LP7 based on the average value of LP1, LP2, LP3, LP4, LP5, LP6, and LP7 and the distortion coefficient data. The distortion coefficients k1, k2, and k3 to be calculated are calculated. The correction unit 140 applies the calculated distortion coefficients k1, k2, and k3 to Expression 1 to correct the pixel data 810 of the horizontal pixel row 1, thereby generating corrected pixel data 811.

Similarly, the positions of the focus lens 210 detected during the exposure period of the horizontal pixel row 2 are LP2, LP3, LP4, LP5, LP6, and LP7. The correction unit 140, based on the average value of LP2, LP3, LP4, LP5, LP6, and LP7 and the distortion coefficient data, the distortion coefficient k1 corresponding to the average value of LP2, LP3, LP4, LP5, LP6, and LP7. , K2 and k3 are calculated. The correction unit 140 applies the calculated distortion coefficients k1, k2, and k3 to the calculation formula 1 to correct the pixel data of the horizontal pixel row 2, thereby generating corrected pixel data. The average value is a value calculated by dividing the weighted sum of LP1 to LP7 by the time from time t1 to time t7. Specifically, the average value is a value calculated by the calculation method described with reference to FIG.

The correction unit 140 applies the same processing from the horizontal pixel row 3 to the horizontal pixel row N to generate corrected pixel data. For example, since the positions of the focus lens 210 detected during the exposure period of the horizontal pixel row N are LP7, LP8, LP9, LP10, LP11, LP12, and LP13, the correction unit 140 uses the LP7, LP8, LP9, and LP10. , LP11, LP12, and LP13 based on the average value and distortion coefficient data, the average value of LP7, LP8, LP9, LP10, LP11, LP12, and LP13 is calculated, and the distortion coefficient k1 corresponding to the calculated average value. , K2 and k3 are calculated. The correction unit 140 applies the calculated distortion coefficients k1, k2, and k3 to Expression 1 to correct the pixel data 820 of the horizontal pixel row N, thereby generating corrected pixel data 821. The correction unit 140 generates one corrected image based on the corrected pixel data generated from each horizontal pixel row.

As described above, when performing rolling reading from the image sensor 120, the detected position of the image sensor 120 may differ depending on the pixel column because the exposure period differs for each horizontal pixel column. Therefore, the correction unit 140 sets the distortion coefficients k1, k2, and k3 based on the average value of the positions of the focus lens 210 in the exposure period for the pixel column groups in which the position of the focus lens 210 detected in the exposure period is the same. It is applied to Equation 1 to generate corrected pixel data.

With reference to FIG. 9 and the like, the process of correcting the image data read by the rolling read based on the average value of the position of the focus lens 210 has been described. The correction method described with reference to FIG. 8 and the like can be applied to correction of image data read by rolling reading. For example, the correction unit 140 may calculate the image coordinates corresponding to each position of the focus lens 210 for each pixel row, and apply the average coordinates of the calculated image coordinates as the pixel value of the corrected image. ..

As described above, according to the imaging device 100, it is possible to provide an image in which the influence of the change in the angle of view due to the movement of the focus lens 210 is suppressed. This effect is particularly effective for a lens device that is relatively small with respect to the size of the image sensor. For example, when the optical system is downsized with respect to the size of the image sensor, the influence of the distortion aberration due to the movement of the focus lens becomes remarkable. Therefore, for example, if the wobbling operation of the focus lens is performed during live view shooting, a change in the angle of view due to the wobbling operation may be observed on the live view image. As described above, according to the imaging device 100, the influence of the angle of view variation due to the movement of the focus lens 210 can be suppressed, and thus the angle of view variation due to the wobbling operation is hard to be observed on the live view image. You can Further, by detecting the contrast value used for the focus control from the corrected image, it is possible to suppress the change of the image area where the contrast value is to be detected due to the movement of the focus lens 210. Therefore, focusing control based on the contrast value can be performed more accurately.

The processing described in relation to the image pickup apparatus 100 of the present embodiment can be applied not only to the wobbling operation of the focus lens 210, but also to processing when the focus lens 210 is moved in one direction during the exposure period. Further, the processing described in relation to the image capturing apparatus 100 of the present embodiment is not only the image during live view shooting, but also correction processing when generating moving image data for recording, and generating still image data for recording. It can also be applied to the correction processing in the case of doing. Further, the processing described in relation to the image pickup apparatus 100 of the present embodiment can be applied to the movement of lenses other than the focus lens 210. That is, the correction unit 140 corrects the image acquired by exposing the image sensor 120 during the exposure period, based on a plurality of positions of lenses other than the focus lens 210 that move during the exposure period of the image sensor 120. You may

The imaging device 100 as described above may be mounted on a moving body. The imaging device 100 may be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. The UAV 10 may include a UAV body 20, a gimbal 50, a plurality of imaging devices 60, and an imaging device 100. The gimbal 50 and the imaging device 100 are an example of an imaging system. The UAV 10 is an example of a moving body propelled by the propulsion unit. The moving body is a concept including a UAV, a flying body such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like.

The UAV body 20 includes a plurality of rotary blades. The plurality of rotary blades is an example of the propulsion unit. The UAV body 20 causes the UAV 10 to fly by controlling the rotation of a plurality of rotor blades. The UAV body 20 flies the UAV 10 by using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed wing aircraft having no rotary wing.

The image capturing apparatus 100 is an image capturing camera that captures an image of a subject included in a desired image capturing range. The gimbal 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 supports the image pickup apparatus 100 using an actuator so as to be rotatable on the pitch axis. The gimbal 50 further supports the imaging device 100 by using an actuator so as to be rotatable about each of a roll axis and a yaw axis. The gimbal 50 may change the posture of the imaging device 100 by rotating the imaging device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are sensing cameras that capture images of the periphery of the UAV 10 in order to control the flight of the UAV 10. The two imaging devices 60 may be provided on the front surface of the UAV 10, which is the nose. Still another two imaging devices 60 may be provided on the bottom surface of the UAV 10. The two imaging devices 60 on the front side may be paired and may function as a so-called stereo camera. The two imaging devices 60 on the bottom side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of imaging devices 60 included in the UAV 10 is not limited to four. The UAV 10 only needs to include at least one imaging device 60. The UAV 10 may include at least one imaging device 60 on each of the nose, tail, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the imaging device 60 may be wider than the angle of view that can be set by the imaging device 100. The imaging device 60 may have a single focus lens or a fisheye lens.

The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may communicate with the UAV 10 wirelessly. The remote control device 300 transmits instruction information indicating various commands regarding movement of the UAV 10, such as ascending, descending, accelerating, decelerating, moving forward, moving backward, and rotating, to the UAV 10. The instruction information includes, for example, instruction information for increasing the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at the altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to elevate the UAV 10. The UAV 10 rises while accepting the rise command. Even if the UAV 10 receives the climb command, the UAV 10 may limit the climb when the altitude of the UAV 10 reaches the upper limit altitude.

FIG. 11 illustrates an example computer 1200 in which aspects of the invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with an apparatus according to an embodiment of the present invention or one or more “units” of the apparatus. For example, the program installed in the computer 1200 can cause the computer 1200 to function as the correction unit 140 or the image processing unit 104. Alternatively, the program can cause the computer 1200 to execute the operation or the function of the one or more “units”. The program can cause the computer 1200 to execute the process according to the embodiment of the present invention or the stage of the process. Such programs may be executed by CPU 1212 to cause computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. Computer 1200 also includes a communication interface 1222, an input/output unit, which is connected to host controller 1210 via input/output controller 1220. Computer 1200 also includes ROM 1230. The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

The communication interface 1222 communicates with other electronic devices via the network. A hard disk drive may store programs and data used by CPU 1212 in computer 1200. The ROM 1230 stores therein a boot program or the like executed by the computer 1200 at the time of activation, and/or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, a USB memory or an IC card, or a network. The program is installed in the RAM 1214 or the ROM 1230, which is also an example of a computer-readable recording medium, and is executed by the CPU 1212. The information processing described in these programs is read by the computer 1200 and brings about the cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing the operation or processing of information according to the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes the communication program loaded in the RAM 1214, and executes the communication process on the communication interface 1222 based on the process described in the communication program. You may order. The communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the recording medium such as the RAM 1214 or the USB memory under the control of the CPU 1212, transmits the read transmission data to the network, or The reception data received from the network is written in the reception buffer area or the like provided on the recording medium.

Further, the CPU 1212 causes the RAM 1214 to read all or a necessary portion of a file or database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. Good. CPU 1212 may then write back the processed data to an external storage medium.

Various types of information such as various types of programs, data, tables, and databases may be stored on the recording medium and processed. The CPU 1212 can retrieve various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval from the data read from the RAM 1214, described in various parts of this disclosure and specified by the instruction sequence of the program. Various types of processing may be performed, including /replacement, etc., and the result is written back to RAM 1214. Further, the CPU 1212 may search for information in files, databases, etc. in the recording medium. For example, when a plurality of entries each having the attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. That is, the entry that matches the condition is searched from the plurality of entries, the attribute value of the second attribute stored in the entry is read, and thereby the first attribute satisfying the predetermined condition is associated. The attribute value of the acquired second attribute may be acquired.

The programs or software modules described above may be stored on a computer-readable storage medium on or near computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, whereby the program can be stored in the computer 1200 via the network. provide.

The execution order of each process such as operation, procedure, step, and step in the device, system, program, and method shown in the claims, the specification, and the drawings is "preceding" or "prior to It should be noted that the output of the previous process can be realized in any order unless the output of the previous process is used in the subsequent process. The operation flow in the claims, the description, and the drawings is described by using “first,” “next,” and the like for the sake of convenience, but it is essential that the operations are performed in this order. Not a thing.

Although the present invention has been described above using the embodiment, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is apparent to those skilled in the art that various changes or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

10 UAV
20 UAV body 50 Gimbal 60 Imaging device 100 Imaging device 102 Imaging unit 104 Image processing unit 110 Imaging control unit 112 Focus adjustment unit 120 Image sensor 130 Memory 140 Correction unit 142 Acquisition unit 160 Display unit 162 Indication unit 200 Lens unit 210 Focus lens 211 Zoom lens 212 Lens drive unit 213 Lens drive unit 220 Lens control unit 222 Memory 214, 215 Position sensor 300 Remote operation device 310 Pixel data
311 corrected image
320 pixel data
321 corrected image
810 pixel data
811 corrected pixel data
820 pixel data
821 corrected pixel data
1200 computer 1210 host controller 1212 CPU
1214 RAM
1220 Input/output controller 1222 Communication interface 1230 ROM

Claims (14)

An acquisition unit that acquires information indicating a plurality of positions of the lens that moves during the exposure period of the image sensor,
An image processing apparatus comprising: a correction unit that corrects distortion of an image acquired by exposing the image sensor through the lens during the exposure period based on the plurality of positions. The correction unit calculates an average position of the lens within the exposure period based on the plurality of positions, and corrects the distortion of the image based on the calculated average position of the lens. Item 1. The image processing device according to item 1. The image processing apparatus according to claim 2, wherein the correction unit corrects the distortion of the image based on an average position of the lens and a distortion coefficient associated with the position of the lens. The correction unit corresponds to each of the plurality of pixels on the image for each combination of the plurality of positions and the plurality of pixels based on the plurality of positions and the positions of the plurality of pixels included in the image sensor. The image processing apparatus according to claim 1, wherein the distortion of the image is corrected by calculating a plurality of pixel positions to be performed and calculating an average position of the plurality of pixel positions for each of the plurality of pixels. The image sensor is exposed in a different exposure period for each of a plurality of horizontal pixel columns included in the image sensor,
The acquisition unit acquires information indicating a plurality of positions of a lens that moves during each exposure period of the plurality of horizontal pixel rows,
The correction unit 4 based on the plurality of positions in each of the exposure periods of the plurality of horizontal pixel rows, from claim 1 for correcting the distortion of the image acquired by each of the plurality of horizontal pixel rows The image processing device according to claim 1. The image processing apparatus according to claim 1, wherein the lens is a focus lens that is movable in the optical axis direction. It said lens is in order Sessu the focus of the optical system including the lens adjustment, reciprocating in the optical axis direction during the exposure period,
The image processing apparatus according to claim 6, wherein the acquisition unit acquires information indicating a plurality of positions of the lens in the reciprocating movement during the exposure period. It said lens is in order Sessu the focus of the optical system including the lens adjustment, moves in one direction to the optical axis direction during the exposure period,
The image processing apparatus according to claim 6, wherein the acquisition unit acquires information indicating a plurality of positions of the lens in the movement in the one direction during the exposure period. The exposure period is a period in which the image sensor is repeatedly exposed to obtain each of a plurality of moving image constituent images that form a moving image,
The correction part, based on the plurality of position in the exposure period for obtaining each of the plurality of video configuration images, any one of claims 1 to 8 for correcting the distortion of each video configuration images The image processing device according to item 1. An image processing apparatus according to any one of claims 1 to 9,
An image pickup apparatus comprising the image pickup device. The lens is a focus lens that is movable in the optical axis direction,
The imaging device is
The imaging device according to claim 10, further comprising a focus adjustment unit that adjusts a focus of an optical system including the lens based on the image whose distortion is corrected by the correction unit. An unmanned aerial vehicle comprising the imaging device according to claim 10 and moving. Acquiring information indicating a plurality of positions of a lens that moves during an exposure period of the image sensor,
Correcting the distortion of the image acquired by exposing the image pickup device through the lens during the exposure period based on the plurality of positions. A program for causing a computer to function as the image processing device according to claim 1.

Images