JP2006033232A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology of generating a still images with high resolution while suppressing deterioration in the image quality, by detecting rapid changes in a photographic state and reflecting it to image processing. <P>SOLUTION: An image processing apparatus 100 designates a reference frame image 13 among frame images consecutive in time series for configuring a moving image MV. The image processing apparatus calculates an exposure value (EV value) of each frame image, on the basis of the reference frame image 13 and Exif information attached to frame images other than the reference frame image. The image processing apparatus discriminates that the photographic conditions of frame images 14 to 16 in existence in the vicinity of the reference frame image 13 and whose EV value is within a prescribed value (±1) is the same as / similar to that of the reference frame image 13, and selects them as processing object frame images for generating a high-resolution still image. Since frame images 10 to 12 have the EV value different by the prescribed value or larger, the image processing apparatus discriminates that the photographic conditions of the frame images 10 to 12 differs from that of the reference frame image 13 and excludes the frame images 10 to 12 from the processing object frame images. The image processing apparatus uses the selected processing object frame images, to perform sharpening processing to produce a high resolution still image 200. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus that creates a still image with higher resolution from a plurality of images.

  2. Description of the Related Art In recent years, image processing apparatuses that create high-resolution images from moving images composed of a plurality of images that are continuous in time series are becoming popular. For example, the image processing apparatus performs intermittent pixel interpolation based on a pixel shift between a reference image and images before and after the reference image, and creates a high-resolution still image.

  Recently, there is a technology to create high-resolution images while suppressing deterioration in image quality even when multiple images include subjects that move in a complex manner or move at a very high speed. It has been proposed (see Patent Document 1).

JP 2004-88616 A

  However, the conventional technology does not take into account image quality deterioration due to a sudden change in the shooting state due to a flash or zoom operation. An abrupt change in the shooting situation means that, for example, when a moving image is shot, the flash is intermittently applied from the surroundings, and the exposure value increases only for some of the images that make up the moving image. In some cases, the focal lengths of a plurality of images may be different. When such a rapid change in the shooting situation occurs during shooting, the detection accuracy of the pixel shift between the images is lowered, and the image quality of the high-resolution still image is inevitably deteriorated.

  The present invention has been made in view of the above-described problems, and detects a sudden change in the shooting situation and reflects it in image processing, thereby creating a still image with high resolution while suppressing deterioration in image quality. The purpose is to provide technology.

In order to solve at least a part of the problems described above, the present invention has the following configuration. That is,
In the image processing apparatus,
An image reading unit for reading a plurality of continuous images in time series,
A reference image designating unit for designating a reference image from the plurality of images;
An image information acquisition unit for acquiring image information capable of specifying a situation at the time of shooting each of the plurality of images;
One of the images obtained by removing the reference image from the plurality of images is selected as an attention image, and the attention image and the image based on the image information of the attention image and the image information of the reference image A determination unit that determines whether the situation at the time of shooting with the reference image is the same or similar;
A selection unit that selects, as a target image, the target image determined by the determination that the reference image and the situation at the time of shooting are the same or similar;
The gist of the present invention is to provide an image creation unit that creates a high-resolution still image having a resolution higher than the resolution of the plurality of images based on the target image and the reference image.

  By adopting such a configuration, it is possible to exclude an image that is the same as or similar to the reference image and the situation at the time of shooting from the processing target image used for the high-resolution still image creation process. Therefore, the accuracy of the sharpening process can be improved, and a high-quality high-resolution still image can be created.

In the image processing apparatus of the present invention,
The image information is a shooting condition at the time of shooting each image associated with each image of the plurality of images,
The image acquisition unit may acquire the image information.

In this way, the image processing apparatus can easily acquire the image information of each image, and therefore it is possible to reduce the calculation load for determining the same / similarity of the situation at the time of shooting.

In the image processing apparatus of the present invention,
The image information includes ISO sensitivity at the time of shooting, exposure time, and aperture value.
The determination unit calculates an exposure value of each image from the image information, and when the difference between the exposure value of the reference image and the exposure value of the target image is equal to or less than a predetermined value, the same or similar It is good also as determining with it.

Also,
The image information includes a distance to the subject at the time of shooting,
The determination unit may determine that the difference is the same or similar when the difference between the distance to the subject in the reference image and the distance to the subject in the target image is a predetermined value or less.

Also,
The image information includes a focal length at the time of shooting,
The determination unit may determine that the difference is the same or similar when a difference between the focal distance of the reference image and the focal distance of the target image is equal to or less than a predetermined value.

  In this way, it is possible to easily detect a change in the photographing state based on various information that can be acquired from the image information associated with the image. Therefore, it is possible to efficiently determine the processing target image and reduce the calculation load.

  In the image processing apparatus of the present invention, the image information may be Exif information. In this way, it is possible to easily acquire image information from an image by referring to Exif information.

In the image processing apparatus of the present invention,
Furthermore,
An image information generation unit that analyzes the plurality of images and generates the image information based on the analysis result may be provided.

  If such a configuration is adopted, the image information can be obtained by analyzing the image even when the image is not associated with image information that can specify the shooting condition.

In the image processing apparatus of the present invention,
The image information generation unit analyzes the luminance information of each of the plurality of images, generates the image information based on the luminance information,
The determination unit may determine that the difference is the same or similar when the difference between the luminance information of the reference image and the luminance information of the target image is a predetermined value or less.

  With such a configuration, it is possible to determine whether or not to select the image to be processed based on a steep change in luminance information between images, that is, a change in brightness between images.

In the image processing apparatus of the present invention,
Furthermore,
Prior to determination by the determination unit, an image distance determination unit that determines whether a time-series distance between the reference image and the target image is within a predetermined range,
The determination unit may determine whether the image of interest determined by the image distance determination unit as being within the predetermined range is the same or similar.

  In general, when a time-series distance between images that are continuous in time series increases, the time elapses and the possibility that the shooting situation is the same or similar is reduced. By taking the configuration of the present invention, it is possible to exclude a target image having a time-series distance from the reference image from the processing target image before determining the same or similar based on the image information. The processing efficiency of the resolution creation process can be improved.

  In the present invention, the various aspects described above can be applied by appropriately combining or omitting some of them. In addition to the configuration as the image processing apparatus described above, the present invention records an image processing method by the image processing apparatus, a computer program for causing the image processing apparatus to perform image processing, and the computer program recorded in a computer-readable manner. It can also be configured as a recording medium. In any configuration, the above-described aspects can be appropriately applied. As a computer-readable recording medium, various media such as a flexible disk, a CD-ROM, a DVD-ROM, a magneto-optical disk, an IC card, and a hard disk can be used.

Hereinafter, embodiments of the present invention will be described in the following order based on examples.
A. First embodiment:
A1. System overview:
A2. Function block:
A3. High-resolution still image creation process:
A3-1. Overall processing:
A3-2. Processing target frame judgment processing:
A3-3. Sharpening process:
A4. Modification of the first embodiment:
B. Second embodiment:
B1. Processing target frame judgment processing:
C. Variations:

A. First embodiment:
A1. System overview:
FIG. 1 is an explanatory diagram illustrating a system overview in the present embodiment. In this embodiment, when the image processing apparatus creates a high-resolution still image from a plurality of still images that are continuous in time series, a still image other than a still image that does not have the same or similar shooting conditions is used. A mode of creating a resolution still image will be described. In the present embodiment, the continuous image SP is composed of still images that are continuous in time series including the still images 10 to 16 and is an image that is captured in the continuous shooting mode of the digital camera. The continuous image SP corresponds to “a plurality of images continuous in time series” described in the claims of the present invention. Hereinafter, in this embodiment, each of the plurality of images is referred to as a frame.

  The image processing apparatus 100 receives the designation of the reference frame 13 from the user in the continuous image SP, and calculates the exposure value of the reference frame 13 based on the Exif information added to the reference frame 13. The exposure value is a numerical value of the exposure amount, and is a so-called EV value. “Exif” is a standard for image files for digital cameras, and “Exif information” is additional information added to an image file. In the Exif information, for example, shooting conditions such as the presence or absence of a flash when shooting with a digital camera, and information such as shooting date and time are recorded. An EV value calculation method will be described later.

  The image processing apparatus 100 is a frame other than the reference frame 13 and has a difference in EV value from the reference frame 13 within a predetermined range from a frame whose time series distance from the reference frame 13 is within a predetermined range. Are selected as target frames to be used for the resolution enhancement processing. The time series distance is a numerical value indicating how many reference frames and other frames are separated from each other in a plurality of time series continuous images. In this embodiment, a frame whose EV value difference from the reference frame 13 is within ± 1 is selected as a target frame to be used for the high resolution processing.

  As shown in the figure, the EV value of the reference frame 13 is “EV = 6”. The EV values of the frames 10 to 12 and 14 to 16 existing before and after the reference frame 13 are “EV = 10” for the frames 10 to 12 and “EV = 6” for the frames 14 to 16. The image processing apparatus 100 selects frames 14 to 16 having “EV = 6” as target frames without setting frames 10 to 12 having “EV = 10” whose EV value is “3” higher than the reference frame 13 as target frames. To do. The target frames 14 to 16 are frames to be compared with pixel deviations from the reference frame 13. In this embodiment, the reference frame 13 and the target frames 14 to 16 are collectively referred to as a processing target frame.

  The image processing apparatus 100 performs a sharpening process based on the processing target frame, and creates a high-resolution still image 200 having a higher resolution than the original image.

A2. Function block:
FIG. 2 is an explanatory diagram illustrating functional blocks of the image processing apparatus 100 in the present embodiment. The image processing apparatus 100 includes a CPU 101, an interface 102, a ROM 110, and a hard disk 120. The ROM 110 includes functional blocks that execute each function. The functional block includes an image reading unit 103, a frame specifying unit 104, a time series distance determining unit 105, an imaging condition determining unit 106, a target frame selecting unit 107, and a sharpening processing unit 108. Each functional block is controlled by the CPU. The hard disk 120 includes an image storage unit 121 and a high-resolution still image storage unit 122. The image storage unit 121 stores images before the high resolution processing such as the continuous image SP, and the high resolution still image storage unit 122 stores the high resolution still image that has been increased in resolution.

  The interface 102 has a function of notifying each functional block of an input from a device operated by a user such as a keyboard and a mouse.

  The image reading unit 103 has a function of reading a continuous image SP for performing resolution enhancement processing from an image stored in the image storage unit 121. The frame designation unit 104 designates a reference frame and a target frame from the frames constituting the read continuous image SP based on an instruction from the user notified through the interface 102. The frame designation unit 104 corresponds to the reference image designation unit of the present invention.

  The time series distance determination unit 105 has a function of determining whether or not the time series distance between the frame of interest and the reference frame is within a predetermined range. Further, when the time-series distance determination unit 105 determines that the time-series distance between the frame of interest and the reference frame is not within a predetermined range, the time-series distance determination unit 105 displays a selection screen as to whether or not to re-specify the reference frame to the user. It also has functions. The time series distance determination unit 105 corresponds to the image distance determination unit of the present invention.

  The shooting condition determination unit 106 compares the shooting conditions of the frame of interest and the reference frame to determine whether the shooting conditions between the frames are the same or similar. Specifically, the EV value is calculated from the Exif information added to each frame, and if the EV value of the frame of interest is within ± 1 compared to the EV value of the reference frame, the shooting conditions are the same or similar. Judge that there is. By using Exif information, it is possible to easily and accurately determine changes in shooting conditions. Exif information will be described with reference to FIG. The imaging condition determination unit 106 corresponds to the image information acquisition unit and determination unit of the present invention.

  FIG. 3 is an explanatory diagram illustrating Exif information in the present embodiment. FIG. 3A is an explanatory diagram illustrating the data configuration of the frame 10 in the present embodiment. The other frames have the same configuration. The frame 10 is composed of Exif information 20 and image data 21. An example of the contents of Exif information will be described with reference to FIG.

  The Exif information 20 includes, as shown in the figure, information on the reference frame 13 such as “image title” and “shooting date and time”, “color space information”, “ISO sensitivity”, “exposure time”, “subject distance”, The shooting conditions at the time of shooting the frame 10 such as “focal length” are recorded. For example, the “image title” of the frame 10 is “file05.jpg”, and the “ISO sensitivity” is “100”. The Exif information 20 can be easily referred to, and various values can be acquired. Therefore, the Exif information 20 is suitable as a shooting condition determination material.

  Returning to FIG. The target frame selection unit 107 selects a target frame determined by the shooting condition determination unit 106 as having the same or similar shooting conditions as the target frame. Further, the target frame selection unit 107 determines whether or not to continue the selection process based on whether or not the number of processing target frames has reached the required number for performing the resolution enhancement process. In this embodiment, the target frame selection unit 107 ends the selection process when the total number of reference frames and target frames is four, that is, three target frames are selected. The target frame selection unit 107 corresponds to the selection unit of the present invention.

  The sharpening processing unit 108 creates a high-resolution still image having a higher resolution than these frames based on the reference frame and the target frame selected by the target frame selection unit 107 and stores them in the high-resolution still image storage unit 122. . The sharpening processing unit 108 corresponds to the image creation unit of the present invention.

A3. High-resolution still image creation process:
A3-1. Overall processing:
FIG. 4 is a flowchart for explaining the overall processing of the high-resolution still image creation processing in this embodiment. This process is started when the user designates a reference frame.

  The image processing apparatus 100 receives a reference frame selection instruction from the user and designates the reference frame (step S10). Next, the image processing apparatus 100 performs processing for selecting a processing target frame (step S11), performs sharpening processing based on the selected processing target frame, and creates a high-resolution still image (step S12). Processing target frame selection processing and sharpening processing will be described below.

A3-2. Processing target frame selection processing:
FIG. 5 is a flowchart for explaining processing frame selection processing in the present embodiment. It is a figure explaining the detail of the process in step S11 of FIG.

  The image processing apparatus 100 calculates the EV value of the reference frame (step S20). The EV value is obtained by the following expression from “ISO sensitivity”, “exposure time”, and “aperture value” included in the Exif information. In the following formula, “ISO sensitivity” is represented as “ISO”, “exposure time” is represented as “T”, and “aperture value” is represented as “F”.

  The expression of Equation 1 is defined as “EV = 0” when “aperture value = 1.0”, “exposure time = 1 sec”, and “ISO sensitivity = 100”. Every time the exposure amount is doubled, the EV value is increased by 1. When the exposure amount is halved, the EV value is decreased by 1. For example, when the EV value is calculated by applying the formula described in Equation 1 to the Exif information 20 in FIG.

  Thus, the EV value of the frame 10 is calculated to be “10”.

  Next, the image processing apparatus 100 designates a frame of interest (step S21). In this embodiment, the frame of interest to be selected immediately after the reference frame is the frame next to the reference frame.

  Based on the time-series distance between the reference frame and the target frame, the image processing apparatus 100 determines whether the target frame is within a predetermined limit range (step S22). Details of such determination processing will be described with reference to FIG.

  FIG. 6 is a flowchart illustrating time-series distance determination processing in the present embodiment. The image processing apparatus 100 calculates the time series distance d1 between the frame of interest and the reference frame (step S30), and determines whether or not the time series distance d1 is within the limit range (step S31). The time-series distance determination process will be described with reference to FIG.

  FIG. 7 is a schematic diagram schematically showing time-series distance determination processing in the present embodiment. The continuous image SP is a continuous image having a frame rate of 30 fps, that is, 30 frames per second. In the figure, “n” of the frame f (n) indicates the order of the frame in the continuous image SP, and the frame f (n) indicates the nth frame of the continuous image SP. . This figure shows frames f (n) to f (n + 30) existing in any one second of the continuous image SP.

  The time series distance d1 between the reference frame and the other frames indicates how many the two images are separated from each other. For example, when the target frame is the frame f (n + 18), the time series distance d1 between the reference frame f (n + 15) and the target frame f (n + 18) is calculated by the following calculation.

d1 = (n + 18)-(n + 15)
= 3

  Therefore, it is determined that time series distance d1 <threshold th (= 14) and the frame of interest f (n + 18) is within the limit range.

  As the time series distance increases, the time difference between both frames also increases, and in general, there are often differences in the contents and shooting conditions of both frames. Therefore, in this embodiment, the threshold th of the time-series distance d1 is set to “14” so that the range that can be the target frame does not exceed 1 second. That is, the frames that can be the target frame are frames within the range of f (n + 1) to f (n + 29) including 14 frames before and after the reference frame f (n + 15).

  Returning to FIG. If the time-series distance d1 between the reference frame and the frame of interest is not less than or equal to the threshold th, that is, if the image processing apparatus 100 determines that it is not within the limit range (step S31: NO), whether or not to re-specify the reference frame A selection guidance screen for allowing the user to select is displayed (step S32). A screen example of the selection guidance screen will be described with reference to FIG.

  FIG. 8 is a screen example of the selection guidance screen in the present embodiment. The selection guidance screen 300 includes a guidance message display unit 301, a selection unit 302, and an OK button 303. The guidance message display unit 301 displays a message for prompting the user to select. The selection unit 302 displays radio buttons 310 to 312 and processing contents to be executed thereafter. The user selects radio buttons 310 to 312 corresponding to a desired process and presses an OK button 303. The image processing apparatus 100 receives the press of the OK button 303 and executes the selected process.

  Returning to FIG. The image processing apparatus 100 determines whether “reference frame redesignation” is selected on the selection guide screen 300 of FIG. 8 (step S33). If re-designation is selected (step S33: YES), the process returns to step S10 in FIG. If re-designation has not been selected, it is determined whether or not “clarification processing execution” has been selected on the selection guide screen 300 of FIG. 8 (step S34). When the sharpening process execution is selected (step S34: YES), the sharpening process is forcibly executed with the currently selected processing target frame (step S12). When the sharpening process execution is not selected (step S34: NO), that is, when “end without sharpening process” is selected on the selection guide screen 300, the high-resolution still image creation process itself Exit.

Returning to FIG. 6, the description will be continued. When the image processing apparatus 100 determines that the time-series distance d1 is within the limit range (step S31: YES), the image processing apparatus 100 continues the processing from step S22 in FIG.

  Returning to FIG. When it is determined that the time-series distance d1 between the reference frame and the target frame is within the limit range, the image processing apparatus 100 calculates the EV value of the target frame (step S23), and determines the EV value of the reference frame. A change amount d2 of the EV value of the frame of interest is calculated (step S24). It is determined whether or not the shooting conditions between both frames are the same or similar depending on whether or not the amount of change d2 is equal to or less than a predetermined value. When the amount of change d2 is greater than or equal to a predetermined value (step S25: NO), that is, when it is determined that the shooting conditions between the frames are not the same or similar, the next frame of interest is designated (step S28). The process is repeated from step S22. The process for designating the frame of interest will be described with reference to FIG.

  FIG. 9 is an explanatory diagram schematically showing a target frame designation process in the present embodiment. The image processing apparatus 100 designates the reference frame 13, and then exists closest to the reference frame 13, has never been a frame of interest, and has elapsed from the reference frame 13. Frame is preferentially designated as the frame of interest. That is, as shown in the figure, the reference frame 13 is designated (arrow (1)), and next, among the frames 12 and 14 that are closest to the reference frame 13 and have never been the frame of interest, The frame 14 whose time has elapsed from the reference frame 13 is designated as the frame of interest (arrow (2)). When the determination of whether or not to select the target frame 14 as the target frame is completed, the image processing apparatus 100 exists as the next target frame closest to the reference frame 13 and has never been the target frame. The frame 12 is designated (arrow (3)). The image processing apparatus 100 designates the frame of interest while making this determination.

Returning to FIG. When the amount of change d2 is equal to or smaller than the predetermined value (step S25: YES), that is, when the image processing apparatus 100 determines that the shooting conditions between the two frames are the same or similar, the image processing apparatus 100 sets the target frame as the target frame. Is selected (step S26), and it is determined whether or not the number of frames to be processed exceeds a predetermined number (step S27). If the number of frames to be processed is not the predetermined number (step S27: NO), the image processing apparatus 100 designates the next frame of interest (step S28). When the predetermined number is reached (step S27: YES), a sharpening process is executed based on the processing target frame to create a high-resolution still image (FIG. 4: step S12). Details of the sharpening process will be described below.
A3-3. Sharpening process:

  FIG. 10 is a flowchart for explaining the sharpening process in the present embodiment. This process is a process executed by the sharpening processing unit 108 of the image processing apparatus 100. In the sharpening processing of the present embodiment, four frames are used as processing target frames. In the following description of the sharpening process, among the processing target frames, the reference frame is referred to as a reference frame F0, and the processing target frames other than the reference frame F0 are referred to as target frames F1, F2, and F3.

  The image processing apparatus 100 acquires the selected processing target frame (step S40). Next, the image processing apparatus 100 calculates the shift amount of pixels existing at corresponding positions in the reference frame F0 and the target frames F1 to F3, and corrects the shift based on the shift amount. A correction amount is estimated (step S41).

  Specifically, first, the image processing apparatus 100 sequentially calculates “deviation amount” indicating how much each of the three target frames F1 to F3 is deviated from the reference frame F0 for all pixels. . The “displacement” in the present embodiment is not caused by the movement of the photographing object itself or the movement of the installation location of the video camera, but the camera work called so-called pan or the change of the orientation of the video camera like camera shake. Is due only to. That is, a shift is assumed in which all pixels are shifted by the same amount between different frames. The deviation is represented by a combination of “translational deviation” indicating horizontal and vertical deviation and “rotational deviation”. When the above-described shift amount is calculated for each pixel for the target frames F1 to F3, a correction amount for correcting the shift is estimated based on the shift amount. The correction amount is represented by a combination of a “translation correction amount” and a “rotation correction amount” in the same manner as the pixel shift amount. The manner in which the translation correction amount is estimated will be described with reference to FIG.

  FIG. 11 is a diagram schematically showing how the translation correction amount is estimated by the pattern matching method in the present embodiment. As shown in the drawing, it is assumed that the coordinates (x1, y1) of the reference frame information correspond to the coordinates (x2, y2) of the target frame information. The translation correction amount is (u, v), and the rotation correction amount is δ with the center of the frame as the origin. Since it is assumed that the focal length is not changed at the time of shooting, the following equation is used as a coordinate conversion equation on the assumption that only translation and rotation are converted.

  Since the time difference between the reference frame and the target frame is very small, u, v, and δ are minute amounts. Here, when δ is a minute amount, cos δ≈1 and sin δ≈δ, so the above equation can be replaced as follows.

  Then, u, v, and δ in the equations described in Equations 7 and 8 (or Equations 9 and 10) are estimated by the method of least squares. The estimation of the correction amount is based on a gradient method (gradient method) that estimates the pixel position in units smaller than one pixel using, for example, the luminance of each pixel between frame information.

  FIG. 12 is a diagram schematically showing a state in which the translation correction amount by the gradient method in this embodiment is estimated. As shown in the upper part of the figure, the luminance of each pixel of the reference frame information is represented as Z1 (ix, iy), and the luminance of each pixel of the target frame information is represented as Z2 (ix ′, iy ′). First, assuming that the coordinates (ix ′, iy ′) of the target frame information are between the coordinates (ix to ix + 1, iy to iy + 1) of the reference frame information, a method of obtaining the coordinates (ix ′, iy ′) by the gradient method. Will be explained.

  As shown in the middle of the figure, assuming that the position in the x-axis direction of the coordinates (ix ′, iy ′) is ix + Δx, and Px = Z1 (ix + 1, iy) −Z1 (ix, iy), Px · Δx = Z2 ( It is sufficient to obtain Δx such that ix ′, iy ′) − Z1 (ix, iy). In practice, Δx is calculated for each pixel, and the average is taken as a whole. Here, if simply expressed as Z1 = Z1 (ix, iy) and Z2 = Z2 (ix ′, iy ′), Δx that satisfies the following expression may be calculated.

  Further, as shown in the lower part of the figure, if the position in the y-axis direction of the coordinates (ix ′, iy ′) is iy + Δy, and Py = Z1 (ix, iy + 1) −Z1 (ix, iy), then Py · Δy = What is necessary is just to obtain | require (DELTA) y which becomes Z2 (ix ', iy')-Z1 (ix, iy). Here, if expressed simply by Z1 = Z1 (ix, iy) and Z2 = Z2 (ix ′, iy ′), Δy may be calculated such that the following expression is satisfied.

Therefore, in consideration of both x and y directions, Δx and Δy that minimize S ^{2 in the} following equation may be obtained by the method of least squares.

  As described above, the method for obtaining the translation correction amount assuming that the frame is translated in the x-axis direction and the y-axis direction by the gradient method has been described. Next, the rotation correction amount will be described with reference to FIG.

  FIG. 13 is a diagram schematically illustrating a pixel rotation correction amount in the present embodiment. As shown in the figure, when the distance from the origin O of the coordinates (x, y) of the reference frame information is r, and the rotation angle from the x axis is θ, r and θ are obtained by the following equations.

  Here, assuming that the translational deviation is corrected, the origins of the reference frame and the target frame are aligned, and the target frame is rotated by δ from the coordinate (x, y) to become the coordinate (x ′, y ′). The amount of movement in the x-axis direction and the amount of movement in the y-axis direction due to this rotation can be obtained by the following equations.

  Therefore, Δx and Δy in the equation described in Equation 13 are expressed by the following equations using the translation correction amounts u and v and the rotation correction amount δ.

  Substituting these into the equation described in Equation 13 yields the following equation:

That is, when the coordinates of the reference frame information are (ix, iy) and the coordinate values and gradation data (luminance values) of all the pixels of the reference frame information are substituted into the equation of Equation 20, S ² is minimized. What is necessary is just to obtain | require u, v, (delta) to carry out by the least square method.

  Returning to FIG. Next, the image processing apparatus 100 combines the frames F0 to F3 using the estimated correction amount (step S42). Specifically, the image processing apparatus 100 corrects the pixel positions of the target frames F1 to F3 using the correction amount (step S43). The pixel position correction means that the pixel position of each pixel of the target frames F1 to F3 is moved by the vertical and horizontal correction amounts based on the translation correction amount, and rotated by the rotation correction amount. Indicates replacement. As a result of such correction processing, the reference frame F0 and the target frames F1 to F3 coincide at the overlapping portion.

  Next, the image processing apparatus 100 determines the nearest pixel (step S44). FIG. 14 is a schematic diagram for explaining the nearest pixel determination process.

  FIG. 14 is a schematic diagram for explaining the nearest-neighbor pixel determination process in the present embodiment. This figure is described taking the translation correction amount as an example. This is a process executed by the image processing apparatus 100. This figure enlarges a part of the matched image of the reference frame F0 and the target frames F1 to F3, and each pixel position of the high-resolution still image 200 obtained from each of the frames F0 to F3 and the frames F0 to F3. Is shown. In the drawing, black circles represent pixels of the high-resolution still image 200, and white squares represent pixels of the reference frame F0. In addition, each pixel of the target frames F1 to F3 corrected in step S42 is represented by different hatching. In this embodiment, the high-resolution still image 200 is increased in resolution to a pixel density 1.5 times that of the reference frame F0. Accordingly, as illustrated, the distance between the pixels of the high-resolution still image 200 is 2/3 of the distance between the pixels of the reference frame F0. Among the pixels of the high-resolution still image 200 such as the pixel G (k) (k is an integer representing the pixel position) in the figure, the pixels that are every two pixels in the vertical and horizontal directions are the pixels and pixels of the reference frame F0. The positions will match.

  Description will be made by paying attention to the j-th pixel G (j) of the high-resolution still image 200. Hereinafter, the pixel G (j) is referred to as a target pixel. The image processing apparatus 100 calculates a distance L0 between the target pixel G (j) and a pixel that is closest to the target pixel G (j) in the pixels of the reference frame F0. Further, the image processing apparatus 100 calculates the distances L1, L2, and L3 with the pixels that are closest to the target pixel G (j) in the pixels of the target frames F1, F2, and F3 after the correction processing in step S42.

  Next, the image processing apparatus 100 compares the values of the distances L <b> 0 to L <b> 3 and determines a pixel that is present at a distance closest to the target pixel G (j). Hereinafter, such a pixel is referred to as a “nearest neighbor pixel”. As shown in the figure, since the pixel on the target frame F3 having the distance L3 is the pixel closest to the target pixel G (j), the image processing apparatus 100 determines this pixel as the nearest pixel. Hereinafter, it is assumed that the nearest pixel to the target pixel G (j) is the i-th pixel of the target frame F3 and is represented as the nearest pixel g (F3, i).

  The image processing apparatus 100 sequentially executes the above processing for all the pixels of the high-resolution still image 200 and determines the nearest pixel for each pixel.

  Returning to FIG. 10, the description will be continued. The image processing apparatus 100 interpolates pixel data of each pixel in the high-resolution still image 200 from pixel data of other pixels (step S45). In this embodiment, the image processing apparatus 100 uses the pixel data of the nearest pixel determined in step S43 and the pixel data in the target frame to which the nearest pixel belongs. FIG. 15 is an explanatory diagram for explaining pixel interpolation processing.

  FIG. 15 is a schematic diagram for explaining pixel interpolation processing in the present embodiment. In this figure, the process in the case where it is determined in step S43 that the nearest pixel for the target pixel G (j) is the nearest pixel g (F3, i) is illustrated.

  In addition to the nearest pixel g (F3, i), the image processing apparatus 100, together with the nearest pixel g (F3.i), a pixel g (F3, j) and a pixel g (F3) surrounding the target pixel G (j). K) and pixel g (F3, l), the pixel data of the target pixel G (j) is obtained by the bilinear method. Not only the bi-linear method but also a method such as the bi-cubic method and the nearest neighbor method may be used.

  The image processing apparatus 100 finishes the sharpening process as described above, and stores the created high-resolution still image in the high-resolution still image storage unit 122.

  According to the image processing apparatus 100 of the first embodiment described above, frames whose shooting conditions are not the same as or similar to the reference frame can be excluded from the processing target frames used for the high-resolution still image creation process. . Therefore, the accuracy of the sharpening process can be improved, and a high-quality high-resolution still image can be created.

  In this embodiment, since the EV value is used, it is possible to detect the change when the photographing condition is suddenly changed by a flash or the like. Therefore, the flashed frame can be excluded from the processing target frame, and the accuracy of the sharpening process can be improved. Using the EV value is effective, for example, when a flash is struck from the surroundings during shooting in the continuous shooting mode.

A4. Modification of the first embodiment:
FIG. 16 is an explanatory diagram for explaining a modification of the first embodiment. FIG. 16A is an explanatory diagram schematically illustrating a processing target frame selected based on a subject distance recorded in Exif information. In the figure, “SD” is an abbreviation for “Subject Distance” and represents the distance to the subject. In this modification, the subject distance is acquired from the Exif information of each frame instead of calculating the EV value in steps S20 and S23 of FIG. This can be realized by calculating the difference of “SD”. In this modification, the predetermined value in step S25 is “1”. By doing so, the image processing apparatus 100 can determine that the shooting conditions are the same or similar when the difference in subject distance between the reference frame and the target frame is 1 m or less.

  As shown in the drawing, the reference frame 13 is “SD = 5.2”, and the frames 10 to 12 and 14 are also “SD = 5.2”. The frames 15 and 16 have “SD = 10.5”. Accordingly, the frames 15 and 16 indicated by the one-dot chain line frame are determined to have different shooting conditions from the reference frame 13 and are excluded from the processing target frames. Since there are four processing target frames including the reference frame, the reference frame 13 indicated by the broken line frame and the frames 11, 12, and 14 are selected as the processing target frames.

  According to this modification, it is possible to determine the similarity of the shooting situation using the subject distance recorded in the Exif information as it is, and to reduce the processing load. Also, by using the subject distance, frames that are not suitable for sharpening processing, such as when the subject is out of focus or when the subject's movement is large, can be excluded from the processing target frames, thereby improving the accuracy of the sharpening processing. Can do.

  FIG. 16B is an explanatory diagram schematically illustrating a processing target frame selected based on the focal length recorded in the Exif information. In the figure, “FL” is an abbreviation for “Focal Length” and represents a lens focal length. In this modification, the focal length is obtained from the Exif information of each frame instead of the calculation of the EV value in steps S20 and S23 of FIG. 5 described in the embodiment, and the difference of the “EV value” is replaced in step S24. This can be realized by calculating the difference of “FL”. In this modification, the predetermined value in step S25 is “0”. That is, if the focal lengths do not match, it is determined that the shooting conditions are different. When the difference is a minute value, it may be allowed.

  As shown in the drawing, the reference frame 13 is “FL = 6.4”, and the frames 10 to 12 are also “FL = 6.4”. The frames 14 to 16 have “FL = 5.9”. Accordingly, the frames 14 to 16 in the one-dot chain line frame are determined to have different shooting conditions from the reference frame 13 and are excluded from the processing target frames. Since there are four processing target frames including the reference frame, the reference frame 13 and the frames 10 to 12 in the broken line frame are selected as the processing target frames.

  According to this modification, it is possible to determine the similarity of the shooting situation using the focal length recorded in the Exif information as it is, and the processing load can be reduced. Further, by using the focal length, it is possible to detect a case where a zoom operation is performed. When the zoom operation is performed, the accuracy of deviation detection is lowered in the sharpening process. Therefore, the precision of the sharpening process can be improved by excluding these frames from the processing target frames.

B. Second embodiment:
In the first embodiment described above, it is determined whether the shooting conditions are the same or similar based on the information recorded in the Exif information, and the processing target frame is selected. In the second embodiment, based on the luminance information obtained by analyzing the frame, it is determined whether the shooting conditions are the same or similar, and the processing target frame is selected. The system configuration in this embodiment is the same as that in the first embodiment, and the functional blocks and the sharpening processing of the image processing apparatus 100 shown in FIGS. Omitted. However, the imaging condition determination unit 106 illustrated in FIG. 2 has a function of calculating frame luminance information. That is, the image information creation unit of the present invention is configured as a part of the imaging condition determination unit 106.

B1. Shooting situation judgment processing:
FIG. 17 is a flowchart for explaining processing target frame selection processing in the present embodiment. It is a figure explaining the detail of the process of step S11 of FIG.

  When the reference frame is designated (FIG. 4: step S10), the image processing apparatus 100 calculates the average of the luminance values of the reference frame (step S50). The process for calculating the average of the luminance values is shown in FIG.

  FIG. 18 is a flowchart for explaining the calculation process of the average of the luminance values of the frames in this embodiment. This process is a process performed by the imaging condition determination unit 106 of the image processing apparatus 100. The imaging condition determination unit 106 performs color conversion processing for converting all the pixels of each frame from the RGB space to the YCbCr space (step S60). Such color conversion processing can be realized by applying the following equation.

  In the YCbCr space, Y represents a luminance value, and the Cb value and the Cr value represent color differences. The shooting condition determination unit 106 calculates the average of the luminance values of the frames by dividing the sum of the luminance values of all the pixels by the number of pixels (step S61).

  Returning to FIG. 17, the description will be continued. The image processing apparatus 100 designates a frame of interest (step S51), and determines whether or not to continue the process with the frame of interest based on the time-series distance between the reference frame and the frame of interest (step S52). When it is determined that the time series distance between the reference frame and the target frame is within the limit range, the average luminance value of the target frame is calculated (step S53), and the average luminance value of the reference frame and the target frame are calculated. An average change amount d3 of the luminance value is calculated (step S54). It is determined whether or not the shooting situation between both frames is the same or similar depending on whether or not the change amount d3 is equal to or less than a predetermined value. In this embodiment, this predetermined value is “5”. If the amount of change d3 is not less than or equal to the predetermined value (step S55: NO), that is, if it is determined that the shooting situation between both frames is not the same or similar, the next frame of interest is designated (step S58), and step S52 Repeat the process.

  When the amount of change d3 is equal to or smaller than the predetermined value (step S55: YES), that is, when the image processing apparatus 100 determines that the shooting situation between both frames is the same or similar, the image processing apparatus 100 sets the target frame as the processing target frame. Is selected (step S56), and it is determined whether or not the number of frames to be processed exceeds a predetermined number (step S57). If the predetermined number has not been reached (step S57: NO), the next frame of interest is designated (step S58). When the predetermined number is reached (step S57: YES), a high-resolution still image is created by executing a sharpening process based on the selected processing target frame (FIG. 4: step S12).

  FIG. 19 is an explanatory diagram schematically illustrating a processing target frame in the present embodiment. The moving image MV2 is composed of a plurality of frames 70 to 76 that are continuous in time series. A frame 73 with a cross hatch is a reference frame. In the figure, the average (Y) of the luminance values of each frame is shown. In this embodiment, a frame in which the difference between the luminance value of the frame and the luminance value “Y = 156” of the reference frame 73 is within 5 is selected as a processing target frame.

  Therefore, in this embodiment, the four frames selected as the processing target frames are the reference frame 73 and the frames 71, 72, and 75. Even if it is located closest to the reference frame 73 as in the case of the frame 74, the brightness value at the time of shooting changes drastically because it is away from the brightness value of the reference frame by “9”. Excluded from the frame. Since the frame whose brightness changes rapidly is not suitable for the sharpening process, the accuracy of the sharpening process can be improved by excluding such a frame from the processing target frame.

  According to the second embodiment of the present invention, it is determined whether or not an image is used for high-resolution still image creation processing based on a sharp change in luminance information between images, that is, a change in brightness between images. be able to. Therefore, the accuracy of the sharpening process can be improved. The present embodiment is also effective when additional information in which shooting conditions such as Exif information are recorded is not associated with each frame.

C. Variations:
Although various embodiments of the present invention have been described above, it is needless to say that the present invention is not limited to these embodiments and can take various configurations without departing from the spirit of the present invention.

It is explanatory drawing which illustrates the system outline | summary in 1st Example. It is explanatory drawing which illustrates the functional block of the image processing apparatus in 1st Example. It is explanatory drawing which illustrates Exif information in 1st Example. It is a flowchart explaining the whole process of the high-resolution still image creation process in 1st Example. It is a flowchart explaining the process target selection process in 1st Example. It is a flowchart explaining the time-sequential distance judgment process in 1st Example. It is the schematic diagram which showed typically the judgment processing of the time series distance in 1st Example. It is a screen example of the selection guidance screen in 1st Example. It is explanatory drawing which showed typically the attention frame designation | designated process in 1st Example. It is a flowchart explaining the sharpening process in 1st Example. It is a figure which shows typically a mode that the translation correction amount is estimated by the pattern matching method in 1st Example. It is a figure which shows typically a mode that the translation correction amount by the gradient method in 1st Example is estimated. It is a figure which shows typically the rotation correction amount of the pixel in 1st Example. It is a schematic diagram explaining the nearest pixel determination process in 1st Example. It is a schematic diagram explaining the pixel interpolation process in 1st Example. It is explanatory drawing explaining the modification of 1st Example. It is a flowchart explaining the process target frame selection process in 2nd Example. It is a flowchart explaining the calculation process of the average of the luminance value of the flame | frame in 2nd Example. It is explanatory drawing which illustrates typically the process target frame in 2nd Example.

Explanation of symbols

10 to 16, 70 to 76 ... Frame 13, 73 ... Reference frame 20 ... Exif information 21 ... Image data 100 ... Image processing apparatus 101 ... CPU
DESCRIPTION OF SYMBOLS 102 ... Interface 103 ... Image reading part 104 ... Frame designation part 105 ... Time series distance judgment part 106 ... Shooting condition judgment part 107 ... Target frame selection part 108 ... Sharpening Processing unit 120 ... Hard disk 121 ... Image storage unit 122 ... High resolution still image storage unit 200 ... High resolution still image 300 ... Selection guide screen 301 ... Guidance message display unit 302 .. Selection section 303 ... OK button 310, 311, 312 ... radio button

Claims (12)

An image processing apparatus,
An image reading unit for reading a plurality of continuous images in time series,
A reference image designating unit for designating a reference image from the plurality of images;
An image information acquisition unit for acquiring image information capable of specifying a situation at the time of shooting each of the plurality of images;
One of the images obtained by removing the reference image from the plurality of images is selected as an attention image, and the attention image and the image based on the image information of the attention image and the image information of the reference image A determination unit that determines whether the situation at the time of shooting with the reference image is the same or similar;
A selection unit that selects, as a target image, the target image determined by the determination that the reference image and the situation at the time of shooting are the same or similar;
An image processing apparatus comprising: an image creation unit that creates a high-resolution still image having a resolution higher than that of the plurality of images based on the target image and the reference image. The image processing apparatus according to claim 1,
Furthermore,
Prior to determination by the determination unit, an image distance determination unit that determines whether a time-series distance between the reference image and the target image is within a predetermined range,
The determination unit is an image processing device that determines whether or not the target image determined to be within the predetermined range by the image distance determination unit is the same or similar. The image processing apparatus according to claim 1 or 2,
The image information is a shooting condition at the time of shooting each image associated with each image of the plurality of images,
The image acquisition unit is an image processing apparatus that acquires the image information. The image processing apparatus according to claim 3,
The image information includes ISO sensitivity at the time of shooting, exposure time, and aperture value.
The determination unit calculates an exposure value of each image from the image information, and when the difference between the exposure value of the reference image and the exposure value of the target image is equal to or less than a predetermined value, the same or similar An image processing apparatus that determines that The image processing apparatus according to claim 3,
The image information includes a distance to the subject at the time of shooting,
The determination unit is an image processing apparatus that determines that the difference is the same or similar when a difference between a distance to the subject of the reference image and a distance to the subject of the attention image is a predetermined value or less. The image processing apparatus according to claim 3,
The image information includes a focal length at the time of shooting,
The determination unit is an image processing apparatus that determines that the difference is the same or similar when a difference between the focal distance of the reference image and the focal distance of the target image is a predetermined value or less. An image processing apparatus according to any one of claims 3 to 6,
The image information is an image processing apparatus that is Exif information. The image processing apparatus according to claim 1 or 2,
Furthermore,
An image processing apparatus comprising: an image information generation unit that analyzes the plurality of images and generates the image information based on the analysis result. The image processing apparatus according to claim 8, wherein
The image information generation unit analyzes the luminance information of each of the plurality of images, generates the image information based on the luminance information,
The determination unit is an image processing apparatus that determines that the difference is the same or similar when a difference between the luminance information of the reference image and the luminance information of the target image is a predetermined value or less. An image processing method performed by an image processing apparatus,
(A) reading a plurality of continuous images in time series;
(B) designating a reference image from the plurality of images;
(C) obtaining image information capable of specifying a situation at the time of photographing each of the plurality of images;
(D) selecting one of the images obtained by removing the reference image from the plurality of images as the attention image, and based on the image information of the attention image and the image information of the reference image, Determining whether the situation at the time of shooting of the image and the reference image is the same or similar;
(E) selecting the target image determined by the determination that the reference image and the situation at the time of shooting are the same or similar as a target image;
(F) An image processing method comprising: creating a high-resolution still image having a resolution higher than that of the plurality of images based on the target image and the reference image. A computer program for causing a computer to execute image processing,
The ability to read multiple images in time series,
A function of designating a reference image from the plurality of images;
A function of acquiring image information capable of specifying a situation at the time of shooting each of the plurality of images;
One of the images obtained by removing the reference image from the plurality of images is selected as an attention image, and the attention image and the image based on the image information of the attention image and the image information of the reference image A function of determining whether the situation at the time of shooting with the reference image is the same or similar;
A function for selecting, as a target image, the target image determined by the determination that the reference image and the situation at the time of shooting are the same or similar;
A computer program for causing a computer to execute a function of creating a high-resolution still image having a resolution higher than that of the plurality of images based on the target image and the reference image.   The recording medium which recorded the computer program of Claim 11 so that computer reading was possible.

Images