JP2012247816A - Image processing device, image processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that, when constituting one static image by superposing a plurality of frame static images cut out from a dynamic image, quality of the constituted static image is degraded due to an influence of uneven brightness in images of which centers of frames are bright but edges of frames get darker.SOLUTION: An image processing device calculates a distance from a target pixel in each frame static image to a reference pixel, and constitutes a static image by switching frame static images to be used for superposition for each pixel on a superposed surface according to the calculated distance. Thus, each frame static image can be superposed in a manner difficult to be influenced by uneven brightness, and it is possible to suppress degradation of quality of the constituted static image.

Description

  The present invention stores an image processing apparatus, an image processing method, a computer program, and a program for composing one large still image (mosaic image) by superimposing each frame still image cut out from a moving image. The present invention relates to a computer-readable storage medium.

  In recent years, research has been conducted on mosaicing technology that forms a single large still image from a plurality of still images that include overlapping regions. This is a technique for constructing a single still image by estimating corresponding points or regions in each still image and overlaying the images based on the estimated points and regions. This technology is currently applied to panoramic image synthesis technology for digital cameras.

  A technique for performing mosaicing based on a still image obtained by capturing a subject as a moving image and cutting out from the frame is called a video mosaicing technique. If this technology is used, a subject can be photographed while moving a photographing device with a movie photographing function little by little, and a still image including the entire subject can be constructed from them by cutting out each frame still image. it can.

  When video mosaicing is performed, there is a problem that the image quality of a still image after construction that is finally obtained varies depending on how images are superimposed in an overlapping region of each frame still image.

  In order to solve this problem, the technique disclosed in Patent Document 1 sets the priority for overlapping each frame still image, and uses the pixel value of the frame still image with the highest priority among the overlapping regions. Is configured. When determining the priority, attention is paid to the relative movement amount of each frame still image or the movement amount of an object existing in the image, and the priority is set higher as the movement amount is smaller. As a result, an image with less motion blur can be preferentially used, and the quality of the still image after the configuration can be improved.

JP 2005-309921 A

  When a single still image is composed by superimposing a plurality of frame still images cut out from a moving image, the center of the frame is bright, but it becomes darker as it goes to the edge. There is a problem that the image quality of a still image is degraded.

  An object of the present invention is to suppress degradation in image quality of a still image after configuration by superimposing still images of each frame by a technique that is not easily affected by uneven brightness when performing mosaicing processing.

An image processing apparatus that configures a plurality of frame still images cut out for each frame from a captured moving image as one still image,
Corresponding to the distance calculation means (401) for calculating the distance from the target pixel to the reference pixel in the frame still image, the projection means (402) for projecting the target pixel on the overlapping plane, and the target pixel after projection Whether or not to update the pixel value with the value of the target pixel for the pixel on the overlapping plane is determined based on the distance value from the target pixel to the reference pixel calculated by the distance calculation means (401). A pixel value update determination unit (403) for determining, a pixel value update unit (404) for updating a pixel value with a value of the target pixel for a pixel on the overlapping plane corresponding to the target pixel after projection, An image processing apparatus comprising: a distance storage unit (405) that stores a value of a distance from the target pixel to the reference pixel calculated by the distance calculation unit (401) for the target pixel.

  According to the present invention, it is possible to perform superimposition on each frame still image so as not to be affected by brightness unevenness, and it is possible to suppress degradation of the image quality of the still image after configuration.

1 is a block diagram showing an image processing system according to the present invention. The figure which shows the outline | summary of the process which the mosaicing process part 108 performs Block diagram showing mosaicing processing unit 108 Block diagram showing the image superposition unit 305 Flowchart of the entire process executed by the system in the first embodiment The figure which shows the example of the feature point extraction in Example 1 The figure which shows the calculation formula of the projection parameter in Example 1. The figure which shows the formula of the affine transformation in Example 1. The figure which shows the image superimposition in Example 1. Flowchart of overall processing of the image superimposing unit in Embodiment 1. The figure which shows the distance from the attention pixel in Example 1 to a reference | standard pixel The figure which shows the pixel on the superimposition plane corresponding to the attention pixel after projection in Example 1. FIG. The figure which shows the brightness nonuniformity in the frame still image in Example 1. FIG. The figure which shows the example which overlap | superposed the frame still image in which the brightness nonuniformity has arisen in Example 1 The figure which shows this proposal superposition method in Example 1 Flowchart of overall processing of the image superimposing unit in the second embodiment.

[Example 1]
The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing an image processing system according to the present invention.

  In FIG. 1, the image capturing apparatus 100 includes a moving image input unit 101, a display unit 102, an operation unit 103, a CPU 104, an auxiliary storage device 105, a RAM 106, a ROM 107, and a mosaicing processing unit 108.

  Hereinafter, each component will be described.

  The moving image input unit 101 includes a lens, an image sensor such as a CCD, an A / D converter, and the like, and converts an image of a subject acquired by shooting into digital moving image data.

  The display unit 102 displays information to be notified to the user.

  The operation unit 103 receives an operation by a user.

  The CPU 104 controls the entire system based on information input by the user from the operation unit 103, a program stored in the auxiliary storage device 105 or the ROM 107, and the like.

  The auxiliary storage device 105 stores the moving image data generated by the system control software and the moving image input unit 101.

  The RAM 106 is a system work memory for the CPU 104 to operate, and is also an image memory for temporarily storing moving image data.

  A ROM 107 is a boot ROM, and stores programs such as a system boot program.

  As shown in FIG. 2, the mosaicing processing unit 108 cuts out a frame still image from the input moving image, estimates the motion of the image capturing apparatus from the extracted moving image, and then projects the image on the overlapping plane to make one large image. Construct a still image.

  As shown in FIG. 3, the mosaicing processing unit 108 includes a frame cutout unit 301, a feature point extraction unit 302, a feature point tracking unit 303, a projection parameter estimation unit 304, and an image superposition unit 305. When the moving image 300 generated by the moving image input unit 101 or stored in the auxiliary storage device 105 is input, the mosaicing processing unit 108 performs processing in each of the processing units 301 to 305, and displays the composed still image 310. Generate and output.

  The frame cutout unit 301 cuts out frame still images one by one from the moving image 300.

  The feature point extraction unit 302 extracts, from the current frame still image, a feature point that has a large luminance gradient around the point and can be easily tracked between frames.

  The feature point tracking unit 303 tracks the feature points extracted by the feature point extraction unit 302 in the previous frame still image between the previous frame and the current frame still image.

  The projection parameter estimation unit 304 calculates projection parameters used when projecting the current frame still image on the overlapping plane based on the feature point tracking result by the feature point tracking unit 303. Here, the projection parameters are calculated by estimating the motion of the image capturing apparatus between the previous frame and the current frame from the result of tracking the feature points.

  The image superimposing unit 305 projects the current frame still image on the superposition plane using the projection parameter estimated by the projection parameter estimation unit 304, and superimposes it on the image already existing on the plane.

  The details of the mosaicing process and the operation of each part of the frame cutout unit 301 to the image superimposing unit 305 will be described later in detail because an example of actual processing will be described later. The Rukoto.

  As shown in FIG. 4, the image superimposing unit 305 includes a distance calculation unit 401, a projection unit 402, a pixel value update determination unit 403, a pixel value setting unit 404, a distance storage unit 405, and a pixel value storage unit 406. . When the frame superimposing image 400 generated by the frame cutout unit 301 and the projection parameter 410 generated by the projection parameter estimation unit 304 are input, the image superimposing unit 305 performs processing in each of the processing units 401 to 406 and performs still image post-configuration. An image 310 is generated and output.

  The distance calculation unit 401 calculates the distance from the target pixel in the frame still image to the reference pixel.

  The projection unit 402 projects the pixel of interest in the frame still image onto the overlapping plane using the projection parameter estimated by the projection parameter estimation unit 303.

  The pixel value update determining unit 403 determines whether or not to update the pixel value with the value of the target pixel for the pixel on the overlapping plane corresponding to the target pixel in the projected frame still image.

  The pixel value setting unit 404 sets the pixel value with the value of the target pixel for the pixel on the overlapping plane corresponding to the target pixel in the projected frame still image.

  The distance storage unit 405 stores the value of the distance from the reference pixel to the target pixel in the image calculated by the distance calculation unit 401.

  The pixel value storage unit 406 stores the pixel value of the target pixel.

  In addition, since the operation | movement of each part of the distance calculation part 401-the pixel value memory | storage part 406 handles the example which actually processes later, it will pick up again in that and will be demonstrated in detail.

  Next, FIG. 5 shows a flowchart of the entire process executed by the system from when the moving image 300 is input to the mosaicing processing unit 108 until the mosaicing process ends and the post-configuration still image 310 is generated.

  In the present embodiment, the CPU 104 reads and executes a computer program stored in the auxiliary storage device 105 (computer-readable storage medium), whereby the mosaicing processing unit 108 (the processing units 301 to 305 in FIG. It functions as each of the processing units 401 to 406) in FIG. 4, but is not limited thereto.

  For example, the mosaicing processing unit 108 (the processing units 301 to 305 in FIG. 3 and the processing units 401 to 406 in FIG. 4) may be realized by hardware such as an electronic circuit.

  In step S501 in FIG. 5, the frame cutout unit 401 cuts out frame still images from the moving image 300 one by one.

  In step S502, the feature point extraction unit 402 extracts feature points from the frame still image. In this embodiment, when extracting feature points, a technique called a Harris operator is used to extract points where the luminance gradient is large in two orthogonal directions around the points. As an example, when the Harris operator is used for an image as shown in FIG. 6, feature points as shown in the figure are extracted. At this time, the total number N of points to be extracted can be freely set. In this embodiment, N = 200. In addition, anything that can extract feature points, such as SIFT, may be used.

  In step 503, the feature point tracking unit 403 tracks the feature point extracted by the feature point extraction unit 402 in the previous frame between the previous frame and the current frame still image. In this embodiment, a feature point tracking algorithm called the LK method is used. However, any feature point tracking algorithm such as a block matching method may be used. Alternatively, feature points may be extracted from both frames, and matching corresponding points based on luminance distribution around the points may be used as a substitute for feature point tracking.

  In step S504, the projection parameter estimation unit 404 calculates projection parameters used when projecting the current frame still image on the overlapping plane based on the tracking result of the feature points in step S503. As shown in FIG. 7, the projection parameters are calculated from the coordinates of the tracking destination in the current frame and the coordinates on the overlapping plane of the feature points extracted in the previous frame. In this embodiment, the affine transformation expressed by the expression of FIG. 8 is assumed as the type of projection. In this case, the parameters are calculated based on the coordinates of the three frames on the current frame and the overlapping plane. It is assumed that a parameter that best represents the movement of all feature points may be found using a voting algorithm or the like by changing some patterns of the three feature points selected at this time. In addition, another model such as perspective transformation may be adopted as the type of projection.

  In step S505, the image superimposing unit 406 projects the current frame still image onto the superposition plane using the projection parameters estimated in step S504, and superimposes the image on the plane. As a result, the image on the overlapping plane can be updated as shown in FIG.

  If all the frames have been processed, the process ends. If there is an unprocessed frame, the process is repeated from step S501.

  This is the end of the description of FIG.

  Next, details of the processing in step S505 in FIG. 5 will be described using the flowchart in FIG.

  It is assumed that the subsequent processing is performed for every pixel in the current frame still image, and here, the processing for a certain pixel of interest will be described.

  In step S1001 of FIG. 10, the distance calculation unit 401 calculates the distance from the target pixel to the reference pixel. The distance here is a value representing how many pixels are separated in the image as shown in FIG. 11, and the unit is a pixel. In this embodiment, the reference pixel is the center of the image, but an arbitrary pixel may be selected.

  In step S1002, the projection unit 402 projects the target pixel on the overlapping plane using the projection parameter estimated in step S504.

  In step S1003, the pixel value update determination unit 403 determines whether or not to update the pixel value with the value of the target pixel for the pixel on the overlapping plane corresponding to the target pixel after projection as illustrated in FIG. . The determination is made by paying attention to the distance from the target pixel calculated in step S1001 to the reference pixel, and comparing the value with the value stored in the distance storage unit 405 in step S1005 described later. If it is determined that the distance value at the target pixel is smaller, the process proceeds to step S1004. If it is determined to be larger, the process proceeds to the end of step S1005. In the example of FIG. 12, what is stored in the pixel value storage unit 405 is the distance in the past frame A in the figure, and the value of the distance at the target pixel is smaller, so the process proceeds to step S1004.

  In step S1004, the pixel value setting unit 404 updates the pixel value with the value of the target pixel for the pixel on the overlapping plane corresponding to the target pixel after projection.

  In step S1005, the distance storage unit 405 stores the value of the distance from the target pixel to the reference pixel calculated in step S1001.

  If all the pixels are processed, the process ends. If there is an unprocessed pixel, the process is repeated from step S1001.

  Above, description of FIG. 10 is complete | finished.

  From here, the effect obtained by performing the superposition process by the above method will be described.

  As a problem when performing mosaicing processing, for each frame still image, as shown in FIG. 13, if uneven brightness occurs in the center of the frame but becomes darker as it goes to the end, brightness unevenness also occurs in the still image after configuration. Occurs, and the image quality deteriorates.

  FIG. 14 shows an example when frame still images with uneven brightness are superimposed. Here, an example in which the pixel value of one of the images is used preferentially for the overlapping region of the two frame still images is shown, but even when both pixel values are averaged, the influence of uneven brightness is still the same. Will receive.

  When overlay processing is performed using the proposed method, as shown in FIG. 15, for each pixel on the overlay plane, a frame still image in which the pixel is located closest to the center of the frame is selected for each pixel. The pixel value can be set with the value of the frame still image.

  By doing so, it is possible to suppress the influence of brightness unevenness in the frame still image and the deterioration of the image quality of the still image after construction.

[Example 2]
In Example 1, the effect of suppressing the deterioration of the image quality of the still image after the configuration was obtained by superimposing the still images of the respective frames so as not to be affected by the brightness unevenness.

  However, when noise is added to the frame still image used in the configuration, since the pixel value is set using only one frame still image for one pixel, there is a problem that it is susceptible to the influence. . In the second embodiment, processing for solving the problem is performed.

  First, the outline of the present embodiment will be described. In the present embodiment, for each pixel on the overlapping plane, the pixel value is set using a plurality of frame still images having a small distance from the target pixel to the reference pixel.

  Hereinafter, the flow of processing will be described with reference to FIG. Note that a description of the same parts as those in the first embodiment is omitted.

  In step S1603 in FIG. 16, the pixel value update determination unit 403 updates the pixel value using the value of the target pixel for the pixel corresponding to the target pixel after projection on the overlapping plane as illustrated in FIG. Determine whether or not. The determination focuses on the distance from the target pixel calculated in step S1601 to the reference pixel, and the lower value of the value and the distance corresponding to N frame still images stored in the pixel value storage unit 405 in step S1605 described later. This is done by comparing with the Nth value. If it is determined that the distance value at the target pixel is smaller, the process proceeds to step S1604. If it is determined to be larger, the process proceeds to the end point of step S1605. Note that any value may be used as the value of N.

  In step S <b> 1604, the pixel value setting unit 403 uses the frame still image corresponding to the distance to the lower N−1th distance stored in the pixel value storage unit 405 and the pixel value of the target pixel of the current frame still image. The pixel value on the overlapping plane is updated. At this time, the update value may be a simple average value of N pixel values or a weighted average value based on the distance.

  In step S1605, the distance storage unit 405 stores the value of the distance from the target pixel to the reference pixel calculated in step S1601 for the N frame still images used when updating the pixel value in step S1604. .

  In step S <b> 1606, the pixel value storage unit 406 stores pixel value values for the N frame still images used when updating the pixel values in step S <b> 1604. At this time, each pixel value is stored in correspondence with the distance value stored in step S1605.

  This is the end of the description of FIG.

  As described above, by setting the pixel value using multiple frame still images with a small distance from the target pixel to the reference pixel, it is possible to suppress the effect of noise on the frame still image become.

DESCRIPTION OF SYMBOLS 100 Image photographing device 101 Movie input part 102 Display part

Claims (7)

An image processing apparatus that configures a plurality of frame still images cut out for each frame from a captured moving image as one still image,
Corresponding to the distance calculation means (401) for calculating the distance from the target pixel to the reference pixel in the frame still image, the projection means (402) for projecting the target pixel on the overlapping plane, and the target pixel after projection Whether or not to update the pixel value with the value of the target pixel for the pixel on the overlapping plane is determined based on the distance value from the target pixel to the reference pixel calculated by the distance calculation means (401). A pixel value update determining unit (403) for determining, a pixel value setting unit (404) for updating a pixel value with a value of the target pixel for a pixel on the overlapping plane corresponding to the target pixel after projection, Distance storage means (405) for storing the value of the distance from the target pixel to the reference pixel calculated by the distance calculation means (401) for the target pixel.
An image processing apparatus.   The distance calculation means (401) calculates the distance from the reference pixel to the target pixel in the frame image before projecting the target pixel on the overlapping plane by the projection means (402). The image processing apparatus according to claim 1. The pixel value update determination means (403) is configured such that the distance value from the target pixel calculated by the distance calculation means (401) to the reference pixel is greater than the distance value stored in the distance storage means (405). 2. The image processing apparatus according to claim 1, wherein a pixel value is determined to be updated with the value of the target pixel when the value is smaller. An image processing apparatus that configures a plurality of frame still images cut out for each frame from a captured moving image as one still image,
Corresponding to the distance calculation means (401) for calculating the distance from the target pixel to the reference pixel in the frame still image, the projection means (402) for projecting the target pixel on the overlapping plane, and the target pixel after projection Whether or not the pixel value of the pixel on the overlapping plane is updated using the target pixel is determined based on the distance value from the target pixel to the reference pixel calculated by the distance calculation unit (401). A pixel value update determination unit (403) for determining, and for a pixel on the overlapping plane corresponding to the target pixel after projection, the pixel value is updated using the target pixel and a corresponding pixel of another frame still image. The pixel value setting means (404) to be performed and each pixel used when the pixel value is updated by the pixel value setting means (404) are calculated by the distance calculation means (401). Distance storage means (405) for storing the distance value from the eye pixel to the reference pixel, and distance storage means for each pixel used when the pixel value is updated by the pixel value setting means (404). Pixel value storage means (406) for storing in correspondence with the distance value stored in (405)
An image processing apparatus.   The distance calculation means (401) calculates the distance from the reference pixel to the target pixel in the frame image before projecting the target pixel on the overlapping plane by the projection means (402). The image processing apparatus according to claim 2. The pixel value update determination means (403) is configured such that the distance value from the target pixel calculated by the distance calculation means (401) to the reference pixel is the distance value stored in the distance storage means (405). The image processing apparatus according to claim 2, wherein when the pixel value is smaller than the maximum value, it is determined that the pixel value is updated using the value of the target pixel. The pixel value setting unit (404) stores the value of the distance from the target pixel and the reference pixel calculated by the distance calculation unit (401) stored in the target pixel and the distance storage unit (405). The image processing apparatus according to claim 2, wherein the pixel value is updated using all the pixels excluding the pixel having the maximum value.