JP2014131177A - Image processing device, camera, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct positional deviation of a frame image and distortion caused by a rolling shutter while suppressing an arithmetic load.SOLUTION: An image processing device 100 comprises: change amount calculation means 104 that calculates a change amount for each frame image of a plurality of frame images which are captured by an image sensor 103 and are consecutively acquired using a geometric transform model; and correction means 104 that corrects the frame image using the change amount calculated by the change amount calculation means 104. The change amount includes information on distortion caused by a rolling shutter of the image sensor 103.

Description

  The present invention relates to an image processing device, a camera, an image processing method, and an image processing program.

  An image processing apparatus that obtains a positional deviation amount between two images using a six-axis affine transformation model, performs alignment between the two images using the positional deviation amount, and combines these two images is known. (See Patent Document 1).

JP 2012-126405 A

  When the above-described conventional technique is used to correct positional deviation due to camera shake during shooting and distortion due to a rolling shutter in a plurality of frame images continuous in the time direction, there are as many as six variables used in the affine transformation model. There has been a problem that the calculation load when calculating the variables becomes large.

(1) The image processing apparatus according to the first aspect of the present invention calculates, using a geometric transformation model, a change amount for each frame image in a plurality of frame images captured and continuously acquired by an image sensor. A change amount calculation unit; and a correction unit that corrects the frame image using the change amount calculated by the change amount calculation unit, wherein the change amount includes information on distortion caused by a rolling shutter of the image sensor. Features.
(2) A camera according to a sixth aspect of the invention includes an image pickup device and the image processing apparatus according to any one of the first to fifth aspects.
(3) In the image processing method according to the seventh aspect of the present invention, a change amount for each frame image is calculated using a geometric transformation model in a plurality of frame images captured and continuously acquired by the image sensor. A change amount calculating step, and a correction step of correcting the frame image using the change amount calculated by the change amount calculating step, wherein the change amount includes information on distortion caused by a rolling shutter of the image sensor. It is characterized by.
(4) An image processing program according to an eighth aspect of the present invention calculates, using a geometric transformation model, a change amount for each frame image in a plurality of frame images captured and continuously acquired by an image sensor. An image processing program for causing a computer to execute a change amount calculation process and a correction process for correcting a frame image using the change amount calculated by the change amount calculation process. It includes information on distortion caused by a rolling shutter.

  According to the present invention, it is possible to correct misalignment of a frame image and distortion due to a rolling shutter while suppressing a calculation load.

It is a block diagram explaining the structural example of the camera by one embodiment of this invention. It is a flowchart which shows the flow of an image correction process. (A) is a figure explaining the frame image which comprises a moving image, (B) is a figure explaining adjacent position variation | change_quantity, (C) is a figure explaining reference | standard variation | change_quantity. It is a figure explaining rolling shutter distortion. It is a figure which shows an example of a frame image. It is a figure explaining the mode of provision of a program.

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of a camera equipped with an image processing apparatus according to an embodiment of the present invention. The camera 100 includes an operation member 101, an imaging lens 102, an imaging element 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various input members operated by the user, such as a power button, a release button, a recording button, a cross key, an enter button, a playback button, a delete button, and the like.

  The imaging lens 102 is disposed so as to form a subject image on the imaging surface of the imaging element 103. The imaging lens 102 is represented by a single lens in FIG. 1 as a representative, but actually includes a plurality of optical lenses.

  The image sensor 103 is an XY address type photoelectric conversion element (CMOS image sensor) having a large number of pixels arranged in a matrix. The image sensor 103 is driven under the control of the control device 104 to capture a subject image formed by the imaging lens 102 and outputs an image signal obtained by the imaging to the control device 104. In the present embodiment, the image sensor 103 is driven by a method (so-called rolling shutter method) that sequentially reads out image signals for each scanning line (pixel row).

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the entire camera 100 in addition to the processing of the image signal. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for developing a program when the CPU executes a program, or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium. The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100.

  When the power switch of the camera 100 is operated, the control device 104 turns on the power, and displays image data for display (through image) obtained by performing image processing for display on the image acquired from the image sensor 103 in time series on the monitor 106. Display.

  When the release button is operated to instruct to take a still image, the control device 104 performs predetermined image processing on the image signal input from the image sensor 103. Then, the image data after image processing is compressed into compressed image data such as JPEG format, and the compressed image data is stored as a still image file in a memory card inserted in the memory card slot 105.

  On the other hand, when the recording button is operated and the start of moving image shooting is instructed, the control device 104 performs predetermined image processing on the image signal of each frame output from the image sensor 103. Then, the image data after image processing is compressed into compressed image data such as MPEG format or Motion JPEG, and the compressed image data is stored in a memory card inserted in the memory card slot 105.

  When the recording button is operated again to instruct the end of moving image shooting, the control device 104 stores the compressed image data up to the end of moving image shooting in the memory card and completes the moving image file on the memory card. Let Note that when capturing a moving image, the camera 100 captures an image of a predetermined number of frames (for example, 24 to 60 frames) per second.

<Image correction processing>
The camera 100 according to the present embodiment corrects a positional deviation between frame images due to camera shake at the time of shooting with respect to a moving image, and also generates distortion (hereinafter referred to as a frame image) due to a rolling shutter method of the image sensor 103. Image correction processing for correcting (rolling shutter distortion) is performed. FIG. 2 is a flowchart for explaining the flow of image correction processing executed by the control device 104. When the moving image for executing the image correction processing is selected from the moving images recorded in the memory card inserted in the memory card slot 105 by the operation of the operation member 101, the control device 104 is illustrated in FIG. Start the program that executes the process.

In step S1, the control device 104 reads each frame image of the moving image data corresponding to the selected moving image from the memory card, and proceeds to step S2. It is assumed that the selected moving image is composed of _n frame images (F _{1 to} F _n ) that are temporally continuous, as shown in FIG.

In step S2, the control device 104, as shown in FIG. 3B, in the _second to _n-th frame images (F _{2 to} F _n ), adjacent immediately preceding frame images (F _{1 to} F _n-1 ). Is calculated as an adjacent change amount P (P _{2 to} P _n ). That is, in the second piece of the frame image F ₂ calculates the next change amount P ₂ of the frame image F ₁ of the first sheet, from the frame image F ₂ of the second sheet in the frame image F ₃ of the third sheet of calculating the adjacent variation P _3, ..., in the frame image F _n of n-th calculating the adjacent variation P _n from the frame image F _n-1 of the n-1 th.

The adjacent change amount P is calculated using a geometric transformation model (affine transformation model) shown in Expression (1). Expression (1) is an expression for converting the coordinates (x, y) of the image into coordinates (x ′, y ′) by affine transformation.

n-th coordinates in the frame image _{F n (x ', y'} ) and the pixel value of the difference between the pixel value of coordinates (x, y) in the n-1 th frame image F _n-1 is smaller Such geometric transformation coefficients p0 to p5 represent the amounts of translation, rotation, deformation, and enlargement / reduction between these two frame images. In the present embodiment, these geometric transformation coefficients p0 to p5 are calculated as the adjacent change amount P. Note that these geometric transformation coefficients p0 to p5 will be described in detail later.

Controller in step S3 104, as shown in FIG. 3 (C), in each frame image 2~n sheet _(F 2 to F _n), from the start frame image (the first frame image) _{F 1} Is calculated as a reference change amount Q (Q _{2 to} Q _n ). That is, in the second sheet of frame image F ₂ calculates the reference change amount Q ₂ from the start frame image F _1, the reference change amount Q ₃ from the start frame image F ₁ in the third piece of the frame image F ₃ ,..., The reference change amount Q _n from the start frame image F ₁ is calculated for the nth frame image F _n . This reference change amount Q is calculated by integrating the adjacent change amounts P (p0 to p5) expressed as a matrix in the equation (1). Specifically, the reference change amount Q is Q ₂ = P ₂ , Q ₃ = Q ₂ × P ₃ , Q ₄ = Q ₃ × P ₄ ,..., Q _n = Q _n−1 × P _n . Is calculated.

In step S <b> 4, the control device 104 uses the reference change amount Q (Q _{2 to} Q _n ) in each of the _second to _n-th frame images (F _{2 to} F _n ) to change from the start frame image (position shift And a correction process for correcting (rolling shutter distortion). Note that a known method is used as the image correction method using the reference change amount Q. Thereafter, the control device 104 records the corrected moving image data on the memory card, and ends the processing of FIG.

<Geometric transformation model>
Next, the geometric transformation model used for calculating the above-described adjacent change amount P will be described in detail. In the geometric transformation model shown in the above equation (1), when the six geometric transformation coefficients p0 to p5 are variables, it is necessary to calculate six variables, which increases the amount of calculation. On the other hand, if it is a geometric transformation model that represents only the amount of displacement between the frame images (the amount of translation and the amount of rotation), it is represented by three variables a, b, and θ as shown in equation (2). Can do. In other words, the expression (2) is an expression in which p0 in the expression (1) is expressed as cos θ, p1 as −sin θ, p2 as sin θ, p3 as cos θ, p4 as a, and p5 as b. In Equation (2), a is a parallel movement amount in the x direction (lateral direction), b is a parallel movement amount in the y direction (vertical direction), and θ is a variable representing the rotation amount.

Further, since the amount of rotation of the camera 100 due to camera shake at the time of shooting is predicted to be small, by approximating cos θ = 1 assuming that θ in Equation (2) is very small, as shown in Equation (3), Equation (2) can be further simplified. That is, the expression (3) is an expression in which p0 in the expression (1) is 1, p1 is c, p2 is -c, p3 is 1, p4 is a, and p5 is b. In Equation (3), a is a parallel movement amount in the x direction (horizontal direction), b is a parallel movement amount in the y direction (vertical direction), and c is a variable representing a rotation amount.

  However, since Equation (3) can only represent the amount of misalignment between frame images (the amount of translation and the amount of rotation), the misalignment between frames can be corrected, but the rolling shutter distortion can be corrected. I can't. Therefore, in the present embodiment, based on the geometric transformation model represented by Expression (3), the rolling shutter distortion is corrected in addition to the correction of the positional deviation between the frame images without increasing the variables (that is, with three variables). A geometric transformation model is proposed.

  Here, before explaining the geometric transformation model for correcting the rolling shutter distortion, the rolling shutter distortion will be considered. Rolling shutter distortion occurs due to the image signal readout time being different for each pixel row of the image sensor 103. In other words, since positions with different readout times occur in the same subject, when the camera 100 moves (that is, shakes) or the subject moves at a high speed, the subject is distorted on the acquired image.

  FIG. 4 is a diagram for explaining rolling shutter distortion. In FIG. 4, the actual shape of the subject is indicated by a dotted line, and the shape of the subject in which rolling shutter distortion has occurred is indicated by a solid line. When the camera 100 moves rightward, in the frame image shown in FIG. 4A, the subject moves to the left and is deformed obliquely so that the upper part is shifted to the right rather than the lower part of the subject. When the camera 100 moves leftward, in the frame image shown in FIG. 4B, the subject moves to the right and is deformed obliquely so that the upper part is shifted to the left rather than the lower part of the subject. When the camera 100 moves downward, in the frame image shown in FIG. 4C, the subject moves upward and expands between the upper and lower portions of the subject, that is, vertically. It deforms as follows. When the camera 100 moves in the upward direction, the subject moves down in the frame image shown in FIG. 4D, and the space between the upper portion and the lower portion of the subject is reduced, that is, reduced in the vertical direction. It deforms as follows. Note that when the subject is moving instead of the camera 100, distortion in the reverse direction occurs.

As described above, the rolling shutter distortion is represented by two types of deformation, that is, deformation that is obliquely inclined in the horizontal direction and deformation that is enlarged / reduced in the vertical direction. Based on this, in the geometric transformation model used in the present embodiment, as shown in the following equation (4), the geometric transformation coefficient (one line) corresponding to the deformation distorted obliquely in the horizontal direction with respect to the above equation (3). Correction terms α and β for correcting rolling shutter distortion are added to the coefficient in the second column) and the geometric transformation coefficient (the coefficient in the second row and the second column) corresponding to the deformation to be expanded / reduced in the vertical direction. . The correction terms α and β represent the amount of change in rolling shutter distortion between two frame images (adjacent frames).

  In Expression (4), w and h represent coordinate values serving as a reference for rolling shutter distortion, for example, coordinate values of the center point of the frame image. Therefore, in Equation (4), the correction term −wα added to the geometric transformation coefficient corresponding to the horizontal translation (the coefficient in the first row and the third column) and the geometric transformation coefficient corresponding to the vertical translation ( The correction term -hβ added to the coefficient in the 2nd row and the 3rd column is for setting the rolling shutter distortion reference at the center of the frame image.

  Next, a method of calculating the rolling shutter distortion change amounts (correction terms) α and β will be considered. FIG. 5 shows an example of a frame image taken by the camera 100. FIG. 5A shows a frame image at time t0, FIG. 5B shows a frame image at time t1, and FIG. 5C shows a frame image at time t2. These frame images are taken in the order of times t0, t1, and t2. In addition, the camera 100 and the subject Ob were stationary at the time t0, but the camera 100 moved leftward with respect to the subject Ob from the time t0 to the time t1 and from the time t1 to the time t2. To do. In this case, in the frame image at time t0, as shown in FIG. 5A, no rolling shutter distortion occurs in the subject Ob. In the frame image at the time t1, as shown in FIG. 5B, the subject Ob moves from the frame image at the time t0 to the right, and rolling shutter distortion that deforms obliquely in the horizontal direction occurs. In the frame image at the time t2, as shown in FIG. 5C, the subject Ob moves in the right direction from the frame image at the time t1, and rolling shutter distortion that deforms obliquely in the horizontal direction occurs.

  The rolling shutter distortion changes in accordance with the scanning speed (reading scanning speed) of the image sensor 103 and the moving speed of the subject on the image sensor 103. That is, the slower the scan speed, the greater the amount of rolling shutter distortion that occurs in the frame image. Further, as the moving speed of the subject increases, the amount of rolling shutter distortion that occurs in the frame image increases. Since the scan speed does not change during moving image shooting, the fluctuation of the rolling shutter distortion during moving image shooting is considered to depend on the moving speed of the subject on the image sensor 103.

  Further, in FIG. 5, since the rolling shutter distortion does not occur (is zero) at time t0, the amount of change in rolling shutter distortion from time t0 to time t1 and the rolling shutter distortion from time t0 to time t1. It is considered that the amount of rolling shutter distortion at time t2 is obtained by integrating the amount of change. That is, it is considered that the amount of rolling shutter distortion can be obtained by obtaining the amount of change in rolling shutter distortion between frame images, and the rolling shutter distortion can be corrected.

  Here, in the frame image, the movement amount of the subject Ob from the time t0 to the time t1 is m1, and the movement amount of the subject Ob from the time t1 to the time t2 is m2. Since the subject is stationary at time t0 and the subject is moving from time t0 to time t1, the amount of change in rolling shutter distortion from time t0 to time t1 is the moving speed of the subject, that is, the movement of the subject Ob. It can be determined from the quantity m1.

  When the movement amount m1 of the subject Ob is the same as the movement amount m2 of the subject Ob, the movement speed of the subject Ob is the same from time t0 to time t1 and from time t1 to time t2. In this case, it is considered that the amount of rolling shutter distortion in the frame image at time t1 is the same as the amount of rolling shutter distortion in the frame image at time t2. Therefore, the amount of change in rolling shutter distortion from time t1 to time t2 is zero.

  On the other hand, when the moving amount m1 of the subject Ob is different from the moving amount m2 of the subject Ob, the moving speed of the subject Ob changes from time t0 to time t1 and from time t1 to time t2. Since the amount of rolling shutter distortion differs by the amount of movement of the subject Ob, the amount of change in rolling shutter distortion from time t1 to time t2 is the amount of change in the movement speed of the subject Ob, that is, the amount of movement m2 of the subject Ob. It is considered that it fluctuates according to the difference from the movement amount m1 of the subject Ob.

  In this way, the amount of change in rolling shutter distortion from the previous frame image can be obtained from the movement amount of the subject Ob in the second frame image, that is, the parallel movement amount of the image. It is considered that can be obtained from the amount of change in the amount of movement of the subject Ob, that is, the amount of change in the amount of parallel movement of the image.

Based on this, in the geometric transformation model of the present embodiment, the changes (correction terms) α and β of the rolling shutter distortion in the equation (4) are expressed by the following equations (5) and (6). In equations (5) and (6), k1 and k2 are constants. a ′ is the horizontal translation amount a calculated in the frame image immediately before the current frame image (the frame image for which the adjacent change amount P is calculated). b ′ is the vertical translation amount b calculated in the frame image immediately before the current frame image (the frame image for which the adjacent change amount P is calculated).
α = k1 (aa ′) (5)
β = k2 (bb ′) (6)

  That is, the change amount α corresponding to the deformation distorted obliquely in the horizontal direction is the difference between the horizontal translation amount a in the current frame image and the horizontal translation amount a ′ in the immediately previous frame image, that is, the horizontal amount. It is calculated according to the amount of change in the direction parallel movement amount a. Further, the amount of change β corresponding to the deformation expanding / reducing in the vertical direction is the difference between the vertical translation amount b in the current frame image and the vertical translation amount b ′ in the immediately preceding frame image, that is, It is calculated according to the change amount of the parallel translation amount b in the vertical direction.

  The constants k1 and k2 are set according to the scan speed of the image sensor 103. As the scanning speed is higher, the rolling shutter distortion generated in the frame image becomes smaller. Therefore, the constants k1 and k2 are set to be small and the correction amount of the rolling shutter distortion is reduced. On the other hand, since the rolling shutter distortion generated in the frame image increases as the scanning speed is slower, the constants k1 and k2 are set larger to increase the correction amount of the rolling shutter distortion.

Since the adjacent change amount P is calculated in chronological order, the adjacent change amount P _n−1 in the immediately preceding frame image has already been calculated at the time of calculating the adjacent change amount P _n in the current frame image. Therefore, at the time of calculating the adjacent change amount P _n in the current frame image, the parallel movement amounts a ′ and b ′ in the immediately previous frame image are known. However, when calculating the adjacent change amount P2 in the _second frame image, since a ′ and b ′ are not calculated in the first frame image, for example, a ′ = b ′ = 0.

  In step S2 (FIG. 2) described above, the control device 104 updates the geometric transformation model shown in the equation (4) using the equations (5) and (6) in each frame image, and the updated geometric transformation. The adjacent change amount P is calculated using the model. According to the equations (4) to (6), since there are three variables a, b, and c, it is possible to reduce the calculation load compared to the case where there are six variables. It should be noted that a known method is used as a variable calculation method in the geometric transformation model.

  Further, the adjacent change amount P represents a positional shift amount (parallel movement amount and rotation amount) and a change amount of rolling shutter distortion between adjacent frame images. For this reason, in step S3 (FIG. 2), the control device 104 can calculate a reference change amount Q representing the amount of positional deviation from the start frame and the change amount of rolling shutter distortion for each frame image. Accordingly, in step S4 (FIG. 2), the control device 104 can correct the positional deviation between the frame images and the rolling shutter distortion in the moving image using the reference change amount Q of each frame.

According to the embodiment described above, the following operational effects can be obtained.
(1) The camera 100 captures a change amount (adjacent change amount P) for each frame image in a plurality of frame images captured continuously by the image sensor 103 and a geometric transformation model (formula (4)). And a control device 104 that corrects the frame image using the adjacent change amount P. The geometric transformation model (formula (4)) is obtained by adding correction terms (α, β) for correcting distortion due to the rolling shutter of the image sensor 103 to the geometric transformation coefficient for correcting the positional deviation of the frame image. It is. The correction terms (α, β) are expressed by using the amount of displacement (parallel displacement) of the frame image as a variable (a, b), and for each frame image, the amount of displacement (a ′, b ′) of the past frame image. ) To update. As a result, it is possible to correct the frame image misalignment and the distortion caused by the rolling shutter while suppressing the calculation load.

(2) In the camera 100 of the above (1), the correction terms (α, β) include a variable (a, b) indicating a positional deviation amount of the frame image and a positional deviation amount (a ′, b ′) and the difference (aa ′, bb ′). Thereby, the rolling shutter distortion can be appropriately corrected according to the moving speed of the subject on the image sensor 103.

(3) In the camera 100 of (2), the correction terms (α, β) are constants set according to the difference (aa ′, bb ′) and the scanning speed of the image sensor 103. (K1, k2). Thereby, the rolling shutter distortion can be appropriately corrected according to the scanning speed of the image sensor 103.

(Modification 1)
In the above-described embodiment, the case where the moving image data recorded in the memory card is read and the adjacent change amount P is calculated based on each frame of the read moving image data has been described. However, the control device 104 may calculate the adjacent change amount P every time a frame of moving image data is captured by the image sensor 103 during capturing of moving image data.

  The control device 104 according to the first modification calculates the change amount from the immediately preceding frame image as the adjacent change amount P every time each frame image is output from the image sensor 103 after the start of moving image capturing. The control device 104 records the frame image data and the adjacent change amount P in association with each other on the memory card.

  When shooting of the moving image ends, the control device 104 starts the image correction process (FIG. 2), and instead of steps S1 and S2, the adjacent change amount P associated with the frame image of the moving image data is read from the memory card. read out. Then, the control device 104 performs the processing after step S3 based on the read adjacent change amount P, and records the corrected moving image data in the memory card.

  As described above, by calculating the adjacent change amount P in real time during moving image shooting and performing image correction processing after moving image shooting, the corrected moving image data can be acquired early.

(Modification 2)
In the above-described embodiment, the example in which the image correction process (FIG. 2) is performed on the moving image data has been described. However, the image correction process is performed on a plurality of frame images continuously acquired by continuous shooting. (FIG. 2) may be performed.

(Modification 3)
In the embodiment described above, correction terms α and β for correcting rolling shutter distortion in the geometric transformation model (formula (4)) are obtained using the parallel movement amounts a ′ and b ′ in the past frame images. An example was described. However, the camera 100 is provided with a sensor (for example, an angular velocity sensor) for detecting the moving speed of the camera 100, and the correction terms α and β are obtained based on the moving speed of the camera 100 detected by the sensor. Also good.

(Modification 4)
In the embodiment described above, correction terms α and β for correcting rolling shutter distortion in the geometric transformation model (formula (4)) are obtained using the parallel movement amounts a ′ and b ′ in the immediately preceding frame image. An example was described. However, the translation amount used for obtaining the correction terms α and β is not limited to one past frame image, and the correction terms α and β are calculated using the translation amounts in a plurality of past frame images. You may make it ask.

(Modification 5)
In the above-described embodiment, the example in which the control device 104 of the camera 100 performs the image correction process (FIG. 2) by executing a program recorded in a memory (not illustrated) has been described. This program may be recorded in advance at the time of product shipment, or may be provided through a recording medium such as a memory card or a data signal such as the Internet after the product shipment. FIG. 6 is a diagram showing this state. The camera 100 is provided with a program via a recording medium 200 such as a memory card. The camera 100 also has a connection function with the communication line 201. A computer 202 is a server computer that provides the program, and stores the program in a recording medium such as a hard disk 203. The communication line 201 is a communication line such as the Internet or personal computer communication, or a dedicated communication line. The computer 202 reads the program using the hard disk 203 and transmits the program to the camera 100 via the communication line 201. That is, the program is transmitted as a data signal via a communication line 201 via a carrier wave. As described above, the program can be supplied as a computer-readable computer program product in various forms such as a recording medium and a data signal (carrier wave).

(Modification 6)
In the above-described embodiment, the example in which the control device 104 included in the camera 100 executes the image correction process (FIG. 2) has been described. However, it is also possible to record a program for executing the image correction process in another terminal such as a personal computer and execute the process on the terminal. In this case, the moving image data captured by the camera is taken into the terminal side, and the image correction process is performed on this. The present invention can also be applied to a camera-equipped mobile phone.

  The above description is merely an example, and is not limited to the configuration of the above embodiment. Moreover, you may combine the structure of each modification suitably with the said embodiment.

DESCRIPTION OF SYMBOLS 100 ... Camera, 101 ... Operation member, 102 ... Imaging lens, 103 ... Imaging element, 104 ... Control apparatus, 105 ... Memory card slot, 106 ... Monitor

Claims (8)

Change amount calculating means for calculating a change amount for each frame image using a geometric transformation model in a plurality of frame images captured and continuously acquired by the image sensor;
Correction means for correcting the frame image using the change amount calculated by the change amount calculation means;
With
The image processing apparatus according to claim 1, wherein the change amount includes information related to distortion caused by a rolling shutter of the image sensor. The image processing apparatus according to claim 1.
The geometric transformation model is obtained by adding a correction term for correcting distortion caused by a rolling shutter of the image sensor to a geometric transformation coefficient for correcting a positional deviation of a frame image.
The image processing apparatus is characterized in that the correction term is expressed by using a positional deviation amount of a frame image as a variable, and is updated using a positional deviation amount of a past frame image for each frame image. The image processing apparatus according to claim 2,
The image processing apparatus, wherein the correction term is expressed using a difference between the variable and a positional deviation amount of the past frame image. The image processing apparatus according to claim 3.
The image processing apparatus, wherein the correction term is expressed using the difference and a constant set in accordance with a scanning speed of the image sensor. In the image processing apparatus according to any one of claims 2 to 4,
The image processing apparatus according to claim 1, wherein the displacement amount represents a parallel movement amount between adjacent frame images. The imaging element;
The image processing apparatus according to any one of claims 1 to 5,
A camera comprising: A change amount calculating step of calculating a change amount for each frame image using a geometric transformation model in a plurality of frame images captured and continuously acquired by the image sensor;
A correction step of correcting the frame image using the change amount calculated by the change amount calculation step;
Have
The image processing method, wherein the change amount includes information on distortion caused by a rolling shutter of the image sensor. Change amount calculation processing for calculating a change amount for each frame image using a geometric transformation model in a plurality of frame images captured and continuously acquired by the image sensor;
A correction process for correcting a frame image using the change amount calculated by the change amount calculation process;
An image processing program for causing a computer to execute
The image processing program characterized in that the amount of change includes information related to distortion caused by a rolling shutter of the image sensor.

Images