JP2006003926A - Image processing device, image processing method and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance the quality of a generated image when generating image data of higher resolution from a plurality of image data. <P>SOLUTION: A CPU 200 acquires a plurality of time series image data F0 to F3, and acquires angular velocities V<SB>X</SB>, V<SB>Y</SB>and V<SB>Z</SB>including an angular velocity corresponding to a rotational displacement of a subject for a straight line perpendicular to a pickup image, as information on the movement of a digital still camera 10 associated with the generation of each of the image data F0 to F3. The CPU 200 uses the angular velocities V<SB>X</SB>, V<SB>Y</SB>and V<SB>Z</SB>to calculate displacement correction amount to the subject between images represented by the image data F0 to F3. The CPU 200 corrects displacements according to the calculated displacement correction amount, and synthesizes the corrected image data F0 to F3 to generate high resolution image data of higher resolution. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to image processing, and more particularly to a technique for generating high-resolution image data from a plurality of relatively low-resolution image data.

  A plurality of approximate image data arranged in time series may be acquired, such as frame images constituting moving image data generated by a digital video camera or image data continuously captured by a digital still camera. By calculating the subject shift that occurs between the images represented by these approximated multiple image data in units smaller than the pixel unit and combining the multiple image data in consideration of this shift, it is higher than the original image data. Techniques for generating resolution image data are known. Also, such a subject shift is regarded as being caused by an angle change of the imaging device corresponding to a vertical shift of the subject and an angle change of the imaging device corresponding to a horizontal shift, and is detected by an angular velocity sensor of the imaging device. A technique for calculating the shift of the subject based on the detected angle change information is known (for example, Patent Document 1).

JP 2001-309234 A JP-A-11-164264

  However, the shift of the subject actually includes a shift in the rotation direction about a straight line perpendicular to the captured image. In the above-described prior art, when such a deviation in the rotation direction is so large that it cannot be ignored, the deviation of the subject may not be calculated with sufficiently high accuracy. In such a case, in the generated image, there is a possibility that image quality deterioration such as a doubled subject occurs.

  In addition, the above-described problem is not limited to the case of using moving image data generated by a digital video camera, but is also a problem common to the case of using a plurality of image data continuously photographed by a digital still camera.

  The present invention has been made to solve the above-described problem, and an object of the present invention is to improve the quality of a generated image in image processing for generating higher-resolution image data based on a plurality of image data. .

  In order to solve the above problems, a first aspect of the present invention provides an image processing apparatus. An image processing apparatus according to a first aspect of the present invention includes image data acquisition means for acquiring a plurality of image data arranged in time series, which is image data generated by an imaging apparatus, and each of the plurality of image data Information relating to a change in the orientation of the imaging device corresponding to the generation of the image, wherein the information includes at least information relating to an angular change corresponding to a shift in the rotation direction of the subject about a straight line perpendicular to the captured image. Information acquisition means for acquiring information associated with the image data, and a correction amount for correcting the positional deviation of the subject between the images represented by the plurality of image data, the acquired information regarding the change in the orientation And a misregistration correction amount calculation means for calculating a correction amount estimated by using the correction amount, and based on the calculated correction amount, the plurality of image data in the plurality of image data While correcting the location shift, by combining the corrected plurality of image data, characterized by comprising a high resolution image generation means for generating a high resolution high-resolution image data from said plurality of image data.

  According to the image processing device of the first aspect of the present invention, the information regarding the change in the orientation of the imaging device corresponding to the generation of each of the plurality of image data together with the plurality of image data, Get associated information. Then, a correction amount (position shift correction amount) for correcting the positional deviation of the subject between the images, which is estimated using the acquired information on the change in the orientation of the imaging device, is calculated. The information related to the change in the orientation of the imaging device includes information related to the change in angle corresponding to the shift in the rotation direction of the subject about the straight line perpendicular to the captured image, so that the correction amount calculation accuracy is improved. As a result, it is possible to improve the image quality of the image represented by the high-resolution image data to be synthesized.

  In the image processing apparatus according to the first aspect of the present invention, it is preferable that the calculation of the misregistration correction amount is performed with an accuracy finer than a pixel unit constituting an image represented by the plurality of image data. By calculating the misregistration correction amount with an accuracy finer than a pixel unit, it is possible to improve the image quality of the image represented by the combined high-resolution image data.

  In the image processing device according to the first aspect of the present invention, the information related to the change in the orientation of the imaging device further includes information related to an angle change corresponding to a vertical displacement of the subject in the captured image, and information on the subject in the captured image. It may also include information on angle change corresponding to the shift in the left-right direction. In such a case, since the positional deviation correction amount is calculated using an angle change corresponding to the deviation in the three directions of the subject in the vertical direction, the horizontal direction, and the rotation direction, the calculation accuracy of the correction amount is further improved. As a result, the image represented by the high-resolution image data to be synthesized can have higher image quality.

  In the image processing device according to the first aspect of the present invention, the information related to the change in the orientation of the imaging device is information detected by a sensor provided in the imaging device or information detected by the sensor. It may be the information obtained. In such a case, the positional deviation correction amount can be accurately calculated based on the information detected by the sensor.

  In the image processing device according to the first aspect of the present invention, the information related to the change in the orientation of the imaging device may be any of an angular displacement amount, an angular velocity, and an angular acceleration related to the change in the orientation of the imaging device. In such a case, the misregistration correction amount can be accurately calculated based on any one of the angular displacement amount, the angular velocity, and the angular acceleration.

  A second aspect of the present invention provides an image processing method. The image processing method according to the second aspect of the present invention is image data generated by an imaging device, acquires a plurality of image data arranged in time series, and corresponds to the generation time of each of the plurality of image data. Information relating to a change in orientation of the imaging device, the information including at least information relating to a change in angle corresponding to a shift in the rotation direction of the subject about a straight line perpendicular to the captured image and associated with the plurality of image data A correction amount for correcting the positional deviation of the subject between the images represented by the plurality of image data, and a correction amount estimated using the acquired information regarding the change in the orientation. Calculating, correcting the positional deviation in the plurality of image data based on the calculated correction amount, and combining the plurality of corrected image data. , And generates a high high-resolution image data resolution than the plurality of image data.

  A third aspect of the present invention provides a computer program that executes image processing. A computer program according to a third aspect of the present invention is an image data generated by an imaging device, the image data acquisition function for acquiring a plurality of image data arranged in time series, and each of the plurality of image data A plurality of images including at least information on a change in orientation of the imaging device corresponding to the generation time, and information on a change in angle corresponding to a shift in the rotation direction of the subject about a straight line perpendicular to the captured image An information acquisition function for acquiring information associated with the data, and a correction amount for correcting the positional deviation of the subject between the images represented by the plurality of image data, the information regarding the acquired change in the orientation; A positional deviation correction amount calculation function for calculating a correction amount estimated by using the correction amount; and the plurality of image data based on the calculated correction amount. And a computer that realizes a high-resolution image generation function for generating high-resolution image data having a higher resolution than the plurality of image data by correcting the misalignment of the image and combining the corrected image data. It is characterized by.

  According to the image processing method according to the second aspect of the present invention and the computer program according to the third aspect of the present invention, it is possible to obtain the same operational effects as those of the image processing apparatus according to the first aspect of the present invention. . The image processing method according to the second aspect of the present invention and the computer program according to the third aspect of the present invention are realized in various aspects in the same manner as the image processing apparatus according to the first aspect of the present invention. Can be done. The computer program according to the third aspect of the present invention can also be realized as a computer-readable recording medium that records the computer program.

  A fourth aspect of the present invention provides an imaging device. An imaging device according to a fourth aspect of the present invention includes an image data generation unit that generates a plurality of image data arranged in time series, a sensor that detects a change in the orientation of the imaging device, and a change in the detected orientation. And an image data output unit for generating information relating to the change in the orientation corresponding to the generation time of the plurality of image data, and outputting the plurality of image data and the generated information in association with each other Means.

  According to the imaging device according to the fourth aspect of the present invention, the information regarding the change in the orientation of the imaging device corresponding to the generation of each of the plurality of image data is output in association with the plurality of image data. In an external image processing apparatus that has acquired a plurality of image data from the imaging apparatus, it is possible to calculate a positional deviation correction amount for each image represented by the plurality of image data, using the output information regarding the change in orientation. As a result, when an external image processing apparatus generates a higher resolution image data by synthesizing a plurality of image data, the misregistration correction amount can be accurately calculated, and the image represented by the generated high resolution image data Can be improved in image quality.

  In the imaging device according to the fourth aspect of the present invention, the information related to the change in orientation is the information related to the change in orientation, the angle change corresponding to the vertical displacement of the subject in the captured image, and the left and right of the subject in the captured image. Information on all or part of an angle change corresponding to a direction shift and an angle change corresponding to a rotation direction shift of the subject about a straight line perpendicular to the captured image may be used. In such a case, the external image processing apparatus calculates the positional deviation correction amount with higher accuracy by using all or part of the angular change corresponding to the deviation in the three directions of the subject in the vertical direction, the horizontal direction, and the rotation direction. can do.

  In the imaging device according to the fourth aspect of the present invention, the angle change detected by the sensor may be any one of an angular displacement amount, an angular velocity, and an angular acceleration. In such a case, the positional deviation correction amount can be calculated with high accuracy using any one of the angular displacement amount, the angular velocity, and the angular acceleration in the external image processing apparatus.

  Hereinafter, an image processing apparatus according to the present invention will be described based on examples with reference to the drawings.

A. Configuration of Example / Image Processing System:
FIG. 1 is an explanatory diagram illustrating an example of an image processing system including an imaging device and an image processing device according to an embodiment of the present invention. A configuration of an image processing system to which an imaging apparatus and an image processing apparatus according to an embodiment of the present invention can be applied will be described with reference to FIG.

  The image processing system includes a digital still camera 10 as an imaging device that generates image data, and a personal computer 20 as an image processing device that generates high-resolution image data from a plurality of image data generated by the digital still camera 10. A color printer 30 is provided as an output device that outputs an image using image data. In addition to the printer 30, for example, a monitor 25 of an LCD display and a display device 40 can be used as the output device.

  The personal computer 20 is a commonly used type of computer. The CPU 200 executes an image processing program including processing for generating high-resolution image data, and the RAM 201 temporarily stores calculation results in the CPU 200, image data, and the like. A hard disk drive (HDD) 202 for storing an image processing program. The personal computer 20 includes a card slot 203 for inserting a memory card MC and an input / output terminal 204 for connecting a connection cable from the digital still camera 10.

  The printer 30 is a printer that can output a color image. For example, an inkjet that forms an image by ejecting four color inks of cyan, magenta, yellow, and black onto a print medium to form a dot pattern. Printer. Alternatively, it is an electrophotographic printer that forms an image by transferring and fixing color toner onto a printing medium. In addition to the above four colors, light cyan (light cyan), light magenta (light magenta), red, and blue may be used as the color ink.

  The display device 40 includes a display display 45 for displaying an image, and is a display device that functions as an electronic photo frame, for example. As the display 45, for example, a liquid crystal display or an organic EL display can be used.

  The printer 30 and the display device 40 may include an image processing function included in the personal computer 20 in order to execute image processing and image output in a stand-alone manner. In such a case, the printer 30 and the display device 40 acquire image data from the digital still camera 10 via, for example, a storage medium such as a memory card MC or a cable without using the personal computer 20, and the printer 30. The display device 40 can function as an image processing device in this embodiment.

  In the following description, a case will be described in which image data generated by the digital still camera 10 is sent to the personal computer 20 and image processing for generating high-resolution image data is executed by the personal computer 20.

Configuration of the digital still camera 10:
FIG. 2 is an explanatory diagram showing a schematic configuration of the digital still camera 10. The digital still camera 10 is a camera that acquires an image as digital data by imaging light information on an image sensor (for example, CCD or CMOS). The digital still camera 10 has a function of generating not only still image data but also moving image data composed of a plurality of frame image data arranged in time series. That is, the digital still camera 10 can select a still image shooting mode and a moving image shooting mode by the user. In the following description, a case where the moving image shooting mode is selected and moving image data is generated will be described.

  As shown in FIG. 2, the digital still camera 10 collects optical information, outputs an optical signal 140 as an electric signal, A / D converts the electric signal output from the optical circuit 140, and digitizes the image data. An image data generation circuit 130 to be generated, an image processing circuit 120 for processing the generated image data, and three angular velocity sensors 161, 162, and 163 are provided. Further, the digital still camera 10 includes a control circuit 100 that controls each circuit, a controller 110 that conveys the user's intention including the imaging mode to the control circuit 100, and a memory card that stores image data in the memory card MC as an image file. A drive 150 is provided.

  The optical circuit 140 includes a plurality of pixels, an image sensor 144 that receives optical information of a captured image and outputs it as an electrical signal in pixel units, a lens 142 that collects optical information in a light receiving unit of the image sensor 144, and an image A diaphragm 146 that adjusts the amount of light received by the element 144 is provided. As shown in FIG. 2, the imaging element 144 has a plurality of pixels CP. In the present embodiment, the distance between the centers of the adjacent pixels CP (hereinafter referred to as pixel pitch) is s in both the vertical direction (y direction in FIG. 2) and the horizontal direction (x direction in FIG. 2).

  The three angular velocity sensors 161, 162, and 163 are devices that detect changes in the direction of the digital still camera 10 about the X axis, the Y axis, and the Z axis as angular velocities, respectively. Various gyroscopes are known as the angular velocity sensors 161, 162, and 163. For example, a mechanical coma gyro, a vibration gyro using vibration of a sound piece / tuning fork, an optical fiber gyro, or the like can be used.

  FIG. 3 is a schematic diagram for explaining a change in direction of the digital still camera 10 around the X axis, the Y axis, and the Z axis. The X-axis is an axis parallel to the exposure surface of the image sensor 144 and refers to the axis representing the left-right direction of the digital still camera 10, and the Y-axis is an axis parallel to the exposure surface of the image sensor 144. An axis representing the vertical direction of 10 shall be said. The Z axis is an axis representing a direction perpendicular to the exposure surface of the image sensor 144. The origin where the three axes of XYZ intersect is located at the center Cc of the exposure surface of the image sensor 144.

Hereinafter, as shown in FIG. 3, the amounts of displacement around the XYZ axes are θ _X , θ _Y , and θ _Z , respectively. The angular velocities around the XYZ axes detected by the angular velocity sensors 161, 162, and 163 are defined as V _X , V _Y , and V _Z , respectively. θ _X and V _X are angle changes (so-called pitch blur) in the captured image captured by the digital still camera 10, and θ _Z and V _Z are digital stills. This is an angle change (so-called yaw direction blur) corresponding to a lateral displacement of the subject in a captured image captured by the camera 10. Further, θ _Y and V _Y are angle changes (so-called blur in the roll direction) corresponding to a shift in the rotation direction of the subject about a straight line perpendicular to the captured image captured by the digital still camera 10. .

  Returning to FIG. 2, the description will be continued. The control circuit 100 is a well-known computer, and includes a CPU 102 that executes a control program, a RAM 104 that temporarily stores data handled by the CPU 102, calculation results, and the like, and a ROM 106 that stores a control program and the like. The control circuit 100 can function as an information generation unit M110 and an image data output unit M120 as shown in FIG. 2 by executing a control program.

The information generation unit M110 acquires the electrical signals detected by the angular velocity sensors 161, 162, and 163, and based on this, the angular velocities V _X , V _Y , and V _{Z of} the digital still camera 10 corresponding to the generation of each frame image data GD. (Hereinafter, also simply referred to as angular velocities V _X , V _Y , V _Z ). The image data output unit M120 associates the plurality of frame image data GD arranged in time series generated by the image data generation circuit 130 with the angular velocities V _X , V _Y , V _Z generated by the information generation unit M110, and generates an image file GF. To the memory card MC. Here, the plurality of frame image data and the angular velocities V _X , V _Y , and V _Z are associated with each other between the generation timing of the frame image data and the detection timings of the angular velocities V _X , V _Y , and V _Z. This means that the correspondence on the time axis can be recognized.

-Image file structure used by the image processing system:
FIG. 4 is an explanatory diagram conceptually showing the internal structure of the image file GF. The image file GF includes a main information storage area R100 that stores moving image data MD and audio data AD, and an attached information storage area R110 that stores attached information associated with the moving image data MD. In the attached information storage area R110, in addition to the angular velocities V _X , V _Y , and V _Z described above, the pixel pitch s and the focal length f are stored.

  In the present embodiment, the moving image data MD is composed of n frame image data GD1 to GDn arranged in time series. Each frame image data is composed of gradation data (hereinafter also referred to as “pixel data”) indicating a gradation value (hereinafter also referred to as “pixel value”) of each pixel in a dot matrix form. The pixel data is YCbCr data composed of Y (luminance), Cb (blue color difference), Cr (red color difference), RGB data composed of R (red), G (green), and B (blue). obtain.

As the angular velocities V _X , V _Y , and V _Z , the angular velocities V _X (1) to V _X (n), V _Y (1) to V _Y (n) corresponding to the generation of the frame image data GD1 to GDn, V _Z (1) to V _Z (n) are stored in the attached information storage area R110. That is, if the moving image data MD is moving image data of 30 frames per second, the angular velocity and the values of V _X , V _Y , and V _Z are recorded every 1/30 seconds.

-Functional configuration of the personal computer 20:
FIG. 5 is a functional block diagram of the personal computer 20 (CPU 200) according to the present embodiment. The outline of the functional configuration of the personal computer 20 (CPU 200) will be described with reference to FIG.

The image data acquisition unit M210 acquires a plurality of frame image data arranged in time series selected from the frame image data GD constituting the moving image data MD of the image file GF. The information acquisition unit M220 acquires attached information (see FIG. 4) associated with the frame image data GD. The attached information information includes information (for example, angular velocities V _X , V _Y , V _Z ) regarding the change in the orientation of the digital still camera 10.

  The misregistration correction amount calculation unit M230 calculates the misregistration correction amount using the attached information obtained from the information acquisition unit M220. The misregistration correction amount is a correction amount for correcting misregistration generated between the images represented by the plurality of frame image data acquired by the image data acquisition unit M210.

  The high-resolution image generation unit M240 corrects the above-described misregistration between the images based on the misregistration correction amount calculated by the misregistration correction amount calculation unit M230. Then, the high-resolution image generation unit M240 combines the corrected frame image data of each image to newly generate image data having a higher resolution than the frame image data.

Image processing in the personal computer 20:
Image processing executed in the personal computer 20 will be described with reference to FIGS.

  FIG. 6 is a flowchart illustrating the processing routine of the image processing according to the present embodiment. The personal computer 20 (CPU 200) activates the image processing program automatically when the memory card MC is inserted into the slot 203 or in accordance with a user instruction. The CPU 200 reads the image file GF from the memory card MC according to a user instruction, and reproduces the moving image data MD recorded in the image file GF.

  When an instruction to acquire frame image data is input by the user during reproduction of the moving image data MD, the CPU 200 selects a plurality of continuous frame image data from the frame image data GD1 to GDn constituting the moving image data MD. Is acquired (step S1). In this embodiment, it is assumed that the CPU 200 acquires frame image data for four frames continuous in time series from the timing instructed by the user. The CPU 200 temporarily stores the acquired four frame image data in the RAM 201.

  In the following description, the number of the acquired four frame image data (hereinafter also referred to as “frame number”) is a, and the frame image data of frame number a is the frame image data Fa (a = 0, 1, 2, We will call it 3). An image represented by the frame image data Fa is referred to as an image fa (a = 0, 1, 2, 3).

Subsequently, the CPU 200 acquires attached information from the image file GF (step S2). Acquired information is the angular velocities V _X , V _Y , V _Z associated with the frame image data F0 to F3, the pixel pitch s, and the interfocal distance f. Hereinafter, the angular velocities associated with the frame image data Fa, that is, the angular velocities around the XYZ axes at the time of generation of the frame image data Fa are respectively expressed as V _X a, V _Y a, V _Z a (a = 0, 1, 2, 3). ).

  When the user inputs an instruction to generate a high-resolution still image for the acquired frame image, first, the CPU 200 first causes the subject position shift that occurs between the images f0 to f3 represented by the frame image data F0 to F3. A correction amount used for correction (hereinafter simply referred to as positional deviation correction) for eliminating (hereinafter simply referred to as positional deviation) is calculated (step S3). Here, the positional deviation and the positional deviation correction will be described.

  FIG. 7 is an explanatory diagram showing a positional deviation between the image f0 represented by the frame image data F0 and the image f3 represented by the frame image data F3. FIG. 8 is an explanatory diagram showing the positional deviation correction performed on the frame image data F3 using the frame image data F0 as reference image data. In calculating the misregistration correction amount, the reference frame image data may be any of F0 to F4, but in the following description, the frame image data F0 is used as a reference. That is, it is assumed that the misalignment correction amount for the frame image data F0 is calculated for each of the other three frame image data F1 to F3.

  The positional deviation is a translational deviation in the horizontal and vertical directions of the image (hereinafter simply referred to as translational deviation) and a deviation in the rotational direction about the center of the image (hereinafter simply referred to as rotational deviation). Expressed in combination. In FIG. 7, the edge of the image f <b> 0 and the edge of the image f <b> 3 are overlapped and the center of the image represented by the image f <b> 0 is shown in order to easily show the shift amount of the image f <b> 3 from the image represented by the reference image f <b> 0. A virtual cross image X0 is additionally written at the position, and it is assumed that the cross image X0 is shifted in the same manner as the image f3, and the cross image X3 that is the shifted image is shown on the image f3. Further, in order to show the shift amount more easily, the image f0 and the cross image X0 are indicated by a thick solid line, and the image f3 and the cross image X3 are indicated by a thin broken line.

  In this embodiment, the translational displacement amount is represented by “um” in the horizontal direction, “vm” in the longitudinal direction, “δm” in the vertical direction, and the image fa (a = 1, 2, 3) with respect to the image f0. The deviation amounts are expressed as “uma”, “vma”, and “δma”. For example, as shown in FIG. 7, the image f3 has a translational shift and a rotational shift with respect to the image f0, and the shift amounts are expressed as um3, vm3, and δm3.

  Further, the correction means that the position of each pixel of the image is moved to a position where u is moved in the horizontal direction, v is moved in the vertical direction, and δ is rotated about the center of the image. This means that the coordinates of each pixel data are converted in the image data. u represents the translation correction amount in the horizontal direction, and v represents the translation correction amount in the vertical direction. Δ represents the rotation correction amount.

  When the positional deviation correction amount of the frame image data Fa (a = 1, 2, 3) with respect to the frame image data F0 is expressed as “ua”, “va”, “δa”, the positional deviation correction amount and the position The relationship of ua = −uma, va = −vma, and δa = −δma is established between the deviations. For example, the misregistration correction amounts u3, v3, and δ3 for the target frame image F3 are represented by u3 = −um3, v3 = −vm3, and δ3 = −δm3.

  As described above, for example, as shown in FIG. 8, the image f3 is changed to the base image f0 by performing correction on the frame image data F3 using the positional deviation correction amounts u3, v3, and δ3. Can be matched. At this time, when the image f3 represented by the corrected frame image data F3 (hereinafter referred to as the corrected image f3) and the image f0 are displayed so as to overlap each other, as shown in FIG. Partially matches the image f0. In order to show the correction result in an easy-to-understand manner, the same virtual cross image X0 and cross image X3 as in FIG. 7 are shown in FIG. 8, and as shown in FIG. X3 coincides with the cross image X0.

  Note that the above-mentioned “partial match” means the following. That is, as shown in FIG. 8, for example, a hatched area P1 is an area that exists only in the image f3, and no corresponding area exists in the image f0. As described above, even if the above-described correction is performed, an area that exists only in the image f0 or only in the f3 is generated due to the shift. Therefore, the image f3 completely matches the image f0. There will be no partial match.

  Returning to FIG. 6, the calculation processing of the above-described misregistration correction amounts ua, va, δa (a = 1, 2, 3) will be described. The CPU 200 calculates a misalignment correction amount using the attached information acquired in step S2 (step S3).

FIG. 9 is a schematic diagram showing the relationship between the change in the orientation of the digital still camera 10 around the X axis and the positional deviation of the subject on the image sensor 144. In FIG. 9, the orientation of the digital still camera 10 is displaced (blurred) by θ _X, so that the position of the subject with respect to the image sensor 144 and the lens lens 142 changes from the position indicated by symbol H to the position indicated by symbol H ′. This is shown schematically. It can be seen that the shift amount V of the subject on the image sensor 144 corresponding to this can be calculated from the displacement amount θ _X. The shift amount V is to be reflected in the image produced directly, since those that shift amount vm caused in the image, the misalignment correction amount described above from displacement theta _X is said to be calculated.

The calculation of the positional deviation correction amount will be specifically described. The displacement amounts around the XYZ axes in the period from the generation of the reference frame image data F0 to the generation of the frame image data Fa to be corrected are expressed as θa _X and θa. Assuming that _Y and θa _Z are as shown in FIG. 9 and the displacement amount around the X axis and Y axis corresponding to the displacement of one pixel (pixel pitch s) on the image sensor 144 is θ_pix, the positional deviation correction amount ua, va, and δa are expressed by the following formulas (1a) to (1c).

ua = θ _Y a / θ_pix (1)
va = −θ _X a / θ_pix (2)
δa = θ _Z a ... (3)

That is, the displacement amounts θ _X a, θ _Y a, and θ _Z a are angles representing blurs in the so-called pitch direction, yaw direction, and roll direction of the digital still camera 10, respectively, as described above with reference to FIG. Since it is a change, it can be seen that the correction amount for the positional deviation in the vertical direction, the horizontal direction, and the rotation direction due to each shake can be calculated.

Here, θ_pix is expressed by the following equation (4) using the interfocal distance f of the digital still camera 10 and the pixel pitch s of the image sensor 144 shown in FIG.
θ_pix = tan ⁻¹ (s / f) ≈s / f (4)

Also, the displacements around the XYZ axes θ _X a, θ _Y a, θ _Z a Is given by a value obtained by integrating the angular velocity about the XYZ axes from the generation time t0 of the frame image data F0 to the generation time ta of the frame image data Fa. Further, the displacement amounts θ _X a, θ _Y a, θ _Z a are the angular velocities V _X a, V _Y a, V _Z a (a = 1, 2, 3) and the angular velocities at the time of generating the frame image data Fa. Using the recorded time interval T, it can be approximated by the following equations (5) to (7).

  Here, in this embodiment, since the angular velocity is recorded for each frame, the time interval T is expressed by the reciprocal of the frame rate. For example, if the angular velocity is a unit (degree / sec) and the frame rate is 30 (1 / sec), the time interval T is 1/30 (sec). From the equations (1) to (7), the positional deviation correction amounts ua, va, and δa are eventually expressed by the following equations (8) to (9).

In step S2, the CPU 200 substitutes the angular velocities V _X a, V _Y a, V _Z a, the pixel pitch s, and the interfocal distance f acquired as the auxiliary information into the above formulas (8) to (9). Thus, the misregistration correction amounts ua, va, δa can be easily calculated. The calculation accuracy of the misregistration correction amount depends on the performance of the angular velocity sensors 161, 162, and 163 of the digital still camera 10. However, in order to ensure sufficient image quality in the high-resolution image data generated later, misalignment correction is required. The amount needs to be calculated with an accuracy (so-called sub-pixel level accuracy) finer than the pixel unit of the images f0 to fa. For example, the translation correction amounts ua and va are calculated in units of 1/16 pixel, and the rotation correction amount δa is calculated in units of 1/100 degrees.

  When the misregistration correction amounts are calculated for all three images fa (a = 1, 2, 3), the CPU 200 uses the calculated misregistration correction amounts to determine the position of the frame image data F1 to F3. Deviation correction is executed (step S4). As a result, the three images fa and the reference image f0 can be overlaid so as to partially match (see FIG. 8).

  Subsequently, the CPU 200 synthesizes the corrected frame image data F1 to F3 and the reference frame image data F0 to display image data representing a new high-resolution image (hereinafter referred to as high-resolution image data). Is generated (hereinafter referred to as image composition processing) (step S5). Hereinafter, the image composition processing will be briefly described with reference to FIGS. 10 and 11.

  FIG. 10 is an explanatory diagram showing, in an enlarged manner, a state in which the reference image f0 and the images f1 to f3 are arranged with the shift corrected. In FIG. 10, each pixel of an image represented by high-resolution image data generated by image synthesis processing (hereinafter referred to as a high-resolution image) is indicated by a black circle, and each pixel of the image f0 is a white quadrilateral. The pixels of the images f1 to f3 are indicated by hatched quadrilaterals. Note that the pixel density of the high-resolution image is set 1.5 times as dense as the image f0. Also, each pixel of the high resolution image is assumed to be in a position overlapping with each pixel of the image f0 every two pixels. However, the pixels of the high-resolution image do not necessarily have to be positioned so as to overlap each pixel of the image f0. For example, all the pixels of the high-resolution image can be in various positions such as being located in the middle of the pixels of the image f0. Further, the pixel density of the high-resolution image is not limited to 1.5 times in length and width, and can be freely set.

  Description will be made by paying attention to a pixel G (j) having a high-resolution image (hereinafter, the pixel G (j) is referred to as a target pixel). Here, the variable j indicates a number for distinguishing all pixels of the high resolution image. For example, the number j is set in order from the upper left pixel to the upper right pixel, and then from the left end pixel to the right end pixel in the next lower row. In each of the images f0, f1, f2, and f3, the CPU 200 pays attention to the pixel F (0), F (1), F (2), and F (3) closest to the pixel of interest G (j). Distances L0, L1, L2, and L3 with the pixel G (j) are calculated. Then, the CPU 200 determines a pixel at the closest distance (hereinafter referred to as “nearest neighbor pixel”). In the example of FIG. 10, since L3 <L1 <L0 <L2, the pixel F (3) of the image f3 is determined as the nearest pixel of the target pixel G (j). Note that the nearest pixel for the target pixel G (j) is hereinafter referred to as the nearest pixel F (3, i), assuming that it is the i-th pixel of the target frame image F3.

  The CPU 200 executes the above-described processing using all pixels constituting the high-resolution image as the target pixel, and determines the nearest pixel for each pixel.

  The CPU 200 uses the pixel value of the pixel surrounding the target pixel G (j) in the image fa (the image f3 in the example illustrated in FIG. 10) including the determined nearest pixel as the pixel value of the target pixel G (j). It is generated by a complementary process such as a bi-linear method.

  FIG. 11 is an explanatory diagram showing interpolation processing by the bi-linear method. Since the target pixel G (j) is a pixel that does not exist in any of the images f0 to f3, there is no pixel value. Therefore, the CPU 200, in addition to the nearest pixel F (3, i), three pixels F (3, i + 1), F (3, k), F (3, k +) surrounding the target pixel G (j). The region partitioned in 1) is divided into four partitions by the pixel of interest G (j), and the pixel values of the pixels at the diagonal positions are weighted and added by the area ratio, thereby adding the pixel of interest G (j ) Pixel value is generated. Here, k represents the pixel number obtained by adding the number of pixels in the horizontal direction of the frame image F3 to the i-th pixel.

  In addition to the bi-linear method, various interpolation methods such as a bi-cubic method and a nearest neighbor method can be used as the pixel value interpolation processing method.

  The CPU 200 generates pixel values by the above-described processing for all the pixels constituting the high resolution image.

As described above, according to the image processing according to the present embodiment, the misalignment correction amount used for the synthesis of the plurality of frame image data F0 to F3 acquired from the moving image data MD is set to each of the frame image data F0 to F3. Is calculated using angular velocities V _X a, V _Y a, and V _Z a (a = 1, 2, 3), which are information relating to changes in the orientation of the digital still camera 10 at the time of generation. When synthesizing multiple image data to generate high-resolution image data, it is necessary to correct the positional deviation of the subject between the images with high accuracy finer than a pixel unit in order to improve the image quality of the generated image. It is done. The subject positional deviation often includes the rotational deviation described above. In the present embodiment, in addition to the angular velocities V _X a and V _Y a corresponding to the translational deviation (vertical and lateral deviations) of the subject, the positional deviation correction is performed using the angular velocity V _Z a corresponding to the rotational deviation of the subject. Since the amount is calculated, the misregistration correction amount can be calculated with high accuracy. As a result, the image quality of the image represented by the high-resolution image data obtained by combining the frame image data F0 to F3 after the positional deviation correction can be improved.

Further, since the misregistration correction amount is calculated using the angular velocities V _X a, V _Y a, and V _Z a (a = 1, 2, 3), which are information detected by the angular velocity sensors 161 to 163, the frame image data By analyzing F0 to F3, the processing time can be reduced and the processing load can be reduced rather than calculating the displacement correction amount.

The digital still camera 10 according to this embodiment includes three angular rate sensors, the angular velocity V _X a in the moving image data (a plurality of frame image data) during generation, V _Y a, and detects the V _Z a, Since the detected angular velocity data is output in association with the generated moving image data, the above-described image processing can be performed in the external image processing apparatus, and the above-described effects can be realized.

B. Variations:
In the above embodiment, a plurality of frame image data is acquired from the moving image data generated in the moving image shooting mode of the digital still camera 10, but the acquisition mode of the plurality of image data used for generating the high resolution image data is as follows. It is not limited to this. For example, it may be a plurality of image data arranged in time series, such as moving image data captured by a digital video camera and a plurality of still image data continuously captured by a digital still camera having a continuous shooting function. The continuous shooting function generally captures multiple cuts at a high speed by storing image data in a high-speed memory (buffer memory) inside the digital still camera without transferring the data to a memory card. Refers to function.

  The digital still camera 10 according to the above embodiment includes an angular velocity sensor as a sensor that detects a change in the orientation of the digital still camera 10, but includes a sensor that detects angular acceleration and a sensor that detects the amount of angular displacement. May be. When the angular acceleration is given, the CPU 200 can calculate the angular velocity by integrating the angular acceleration, further calculate the angular displacement by integrating the angular acceleration, and when the angular displacement is given. Can be used for calculation of the positional deviation correction amount as it is.

The digital still camera 10 outputs the detected angular velocities V _X , V _Y , and V _{Z as} they are to the image file GF. However, the digital still camera 10 executes integration processing by the control circuit 100 or the like inside the digital still camera 10. Alternatively, the angular displacement may be processed and output. In such a case, the angular displacement amounts θ _X , θ _Y , and θ _Z are output to the image file GF in association with the image data. The CPU 200 can use the angular displacement amounts θ _X , θ _Y , and θ _{Z as} they are for calculating the positional deviation correction amount.

  In the image file GF in the above embodiment, the same number of angular velocities corresponding to the generation of frame image data are recorded as the number of frame image data, but the number of frame image data and the number of angular velocity data need not be the same. For example, the angular velocity may be recorded every 1/120 second with respect to moving image data of 30 frames per second (an angular velocity data four times the number of frame image data is recorded). In such a case, for example, a frame image is recorded in the image file GF by recording the frame rate (for example, 30 frames per second) and the time interval T (for example, every 1/120 second) at which the angular velocity is recorded. The correspondence on the time axis between data generation and angular velocity detection can be recognized. As the angular velocity recording pitch becomes finer (as the number of angular velocity data per recorded frame pixel data increases), the detection accuracy of the positional deviation correction amount improves.

  In the above embodiment, the digital still camera 10 is provided with an angular velocity sensor, but the digital still camera 10 may be provided with an angular velocity sensor separately. For example, an angular velocity sensor may be provided on a camera platform used when the digital still camera 10 is fixed to a tripod. In such a case, for example, the temporal change in the detected value of the angular velocity sensor is recorded in a recording medium different from the memory card MC of the digital still camera 10 in synchronization with the shooting of the moving image. It is possible to recognize the correspondence on the time axis between the generation timing of each frame image data of the image and the detection timing of the angular velocity. In this way, it is possible to obtain the same effect as the above embodiment.

  As described above, the image processing according to the present invention has been described based on the examples and the modifications. However, the embodiments of the present invention described above are for facilitating the understanding of the present invention and limit the present invention. is not. The present invention can be changed and improved without departing from the spirit and scope of the claims, and equivalents thereof are included in the present invention.

1 is an explanatory diagram illustrating an example of an image processing system including an imaging device and an image processing device (personal computer) according to an embodiment. 2 is an explanatory diagram showing a schematic configuration of a digital still camera 10. FIG. FIG. 3 is a schematic diagram for explaining a change in direction around an XYZ axis of the digital still camera. Explanatory drawing which shows notionally the internal structure of the image file GF. The functional block diagram of the personal computer 20 (CPU200) which concerns on an Example. 6 is a flowchart illustrating a processing routine of image processing according to the embodiment. Explanatory drawing which shows the position shift between the images f0 and f3. Explanatory drawing shown about the position shift correction performed with respect to the frame image data F3. FIG. 4 is a schematic diagram illustrating a relationship between a change in direction around the X axis of the digital still camera and a positional deviation of a subject on the image sensor 144; Explanatory drawing which expands and shows a mode that the image f0-f3 was correct | amended and arrange | positioned. Explanatory drawing which shows the interpolation process by a bilinear method.

Explanation of symbols

10. Digital still camera (DSC)
DESCRIPTION OF SYMBOLS 100 ... Control circuit 102 ... Central processing unit (CPU)
104: Random access memory (RAM)
106: Read only memory (ROM)
DESCRIPTION OF SYMBOLS 110 ... Controller 120 ... Image processing circuit 130 ... Image data generation circuit 140 ... Optical circuit 142 ... Lens 144 ... Imaging element 146 ... Aperture 150 ... Memory card drive 161, 162, 163 ... Angular velocity sensor 20 ... Personal computer 200 ... Central processing unit (CPU)
201: Random access memory (RAM)
202: Hard disk (HDD)
DESCRIPTION OF SYMBOLS 203 ... Card slot 204 ... Input / output terminal 25 ... Monitor 30 ... Printer 40 ... Display device 45 ... Display part MC ... Memory card M110 ... Information generation part M120 ... Image data output part M210 ... Image data acquisition part M220 ... Information acquisition part M230 ... misregistration correction amount calculation unit M240 ... high-resolution image generation unit

Claims (11)

An image processing apparatus,
Image data acquisition means for acquiring a plurality of image data arranged in time series, which is image data generated by the imaging device;
Information relating to changes in the orientation of the imaging device corresponding to the generation of each of the plurality of image data, and information relating to angular changes corresponding to deviations in the rotation direction of the subject about a straight line perpendicular to the captured image And information acquisition means for acquiring information associated with the plurality of image data,
A misregistration correction amount for correcting a misalignment of a subject between the images represented by the plurality of image data, and calculating a correction amount estimated using the acquired information on the change in the orientation A calculation means;
Based on the calculated correction amount, the positional deviation in the plurality of image data is corrected, and the corrected plurality of image data is combined to obtain high-resolution image data having a higher resolution than the plurality of image data. An image processing apparatus comprising: high-resolution image generation means for generating The image processing apparatus according to claim 1.
The image processing apparatus is configured to calculate the misregistration correction amount with an accuracy finer than a pixel unit constituting an image represented by the plurality of image data. The image processing apparatus according to claim 1 or 2,
Information regarding the change in orientation of the imaging device further includes:
Information on the angle change corresponding to the vertical displacement of the subject in the captured image;
An image processing apparatus including information related to an angle change corresponding to a lateral shift of a subject in a captured image. The image processing apparatus according to any one of claims 1 to 3,
Information on the change in orientation of the imaging device is
Information detected by a sensor provided in the imaging device, or
An image processing apparatus which is information obtained by processing information detected by the sensor. The image processing apparatus according to any one of claims 1 to 4,
Information on the change in orientation of the imaging device is
An image processing apparatus that is one of an angular displacement amount, an angular velocity, and an angular acceleration related to a change in orientation of the imaging apparatus. An image processing method comprising:
Image data generated by the imaging device, obtaining a plurality of image data arranged in time series,
Information relating to changes in the orientation of the imaging device corresponding to the generation of each of the plurality of image data, and information relating to angular changes corresponding to deviations in the rotation direction of the subject about a straight line perpendicular to the captured image And acquiring information associated with the plurality of image data,
A correction amount for correcting a positional deviation of the subject between the images represented by the plurality of image data, and calculating a correction amount estimated using the acquired information on the change in the orientation;
Based on the calculated correction amount, the positional deviation in the plurality of image data is corrected, and the corrected plurality of image data is combined to obtain high-resolution image data having a higher resolution than the plurality of image data. Image processing method for generating A computer program for executing image processing,
An image data acquisition function for acquiring a plurality of image data arranged in time series, which is image data generated by an imaging device;
Information relating to changes in the orientation of the imaging device corresponding to the generation of each of the plurality of image data, and information relating to angular changes corresponding to deviations in the rotation direction of the subject about a straight line perpendicular to the captured image And an information acquisition function for acquiring information associated with the plurality of image data,
A misregistration correction amount for correcting a subject misregistration between the images represented by the plurality of image data, and calculating a correction amount estimated using the acquired information on the change in the orientation A calculation function;
Based on the calculated correction amount, the positional deviation in the plurality of image data is corrected, and the corrected plurality of image data is combined to obtain high-resolution image data having a higher resolution than the plurality of image data. The computer program which makes a computer implement | achieve the high-resolution image generation function which produces | generates.   A computer-readable recording medium on which the computer program according to claim 7 is recorded. An imaging device comprising:
Image data generating means for generating a plurality of image data arranged in time series;
A sensor for detecting a change in orientation of the imaging device;
Using the detected change in orientation, information generating means for generating information relating to the change in orientation corresponding to the generation of each of the plurality of image data;
An image pickup apparatus comprising: image data output means for outputting the plurality of image data in association with the generated information. The imaging device according to claim 9,
Information on the change in orientation is
Angle change corresponding to the vertical displacement of the subject in the captured image,
Angle change corresponding to the horizontal displacement of the subject in the captured image,
An angle change corresponding to a shift in the rotation direction of the subject about a straight line perpendicular to the captured image;
An imaging device that is information about all or part of the information. The imaging apparatus according to claim 9 or 10, wherein:
The information relating to the change in the orientation is an imaging apparatus that is one of an angular displacement amount, an angular velocity, and an angular acceleration.

Images