JP6494388B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  As a conventional technique, there is known an image synthesis processing (HDR: High Dynamic Range imaging) technique for obtaining a high dynamic range image by synthesizing images of a plurality of frames taken with different exposure amounts. In the HDR technology, an image without so-called whiteout or blackout can be obtained by performing image composition such that image regions that are appropriately exposed in each frame image are connected. In the image synthesis process by the HDR technology, a process of generating one frame image by synthesizing a plurality of frame images in time series obtained by moving image shooting is repeated for each of the plurality of frames in time series. Therefore, it can be realized for moving images.

  However, when HDR is combined with a moving image, one frame image is combined from a plurality of frames, so the frame rate of the moving image after the combining processing is higher than the frame rate at the time of image capture. It will decline. For this reason, for example, when the subject is a moving body that is moving, there is a problem that the movement of the image of the moving body does not look smooth. On the other hand, there is known a method for smoothly showing the movement of the moving body image by synthesizing the moving body image and generating an image in which the subject image is blurred by the movement. However, with this method, for example, when the moving amount of the moving object image is large, the image becomes unnatural.

  For this reason, techniques as described in Patent Document 1 and Patent Document 2 have been proposed. Patent Document 1 discloses a technique for calculating a motion vector of an image to be synthesized in a synthesis process using HDR, and not synthesizing the pixel when the motion vector is larger than a certain threshold value. . In Patent Document 2, for example, when the imaging sensitivity is low, the number of synthesized images is small, or when a person is detected, a threshold value for determining whether to perform HDR synthesis processing is reduced. Technology is disclosed.

JP-A-10-243288 JP 2012-29014 A

  In the technique described in Patent Document 1, a composition process is performed on a moving object image with a small amount of movement, while a composition process is not performed on a moving object image with a large amount of movement. For this reason, generation | occurrence | production of the unnatural depiction by the synthetic | combination process of a moving body image with a large movement amount is avoided. However, if the moving body image that is to be combined because of the small amount of movement is a conspicuous image, for example, the moving body image after the combining process may be depicted as an unnatural image. There is. Further, in the technique described in Patent Document 2, since the way of synthesis is changed according to various conditions, the occurrence of unnatural depiction is extremely reduced. However, even in the case of the technique of Patent Document 2, when the moving body image to be combined is, for example, a conspicuous image, the combined moving body image is depicted as an unnatural image. May end up. That is, with these techniques, there is a possibility that image quality deterioration may occur such that the moving body image after the synthesis processing is depicted as an unnatural image.

  The present invention has been made in view of such a problem, and an image processing apparatus and an image processing method capable of reducing image quality deterioration such that a moving body image after synthesis processing is depicted as an unnatural image. And to provide a program.

  In the image processing apparatus according to the present invention, the first frame image and the second frame image are captured from at least two frame images of the first frame image and the second frame image obtained by moving image capturing. A moving body detecting unit that detects a subject image whose subject position has changed between them as a moving body image; from the first frame image and the second frame image, photographing the first frame image; and Movement amount detection means for obtaining the movement amount of the moving body image during the shooting of the frame image, area detection means for obtaining the area of the moving body image, the first frame image and the second frame image The first frame image and the second frame image are combined by the combining unit based on the moving amount and the area of the moving object image. Of, and having a control means for controlling the mixing ratio of the moving object image of the first of said moving object image and the second frame image of the frame image.

  According to the present invention, it is possible to reduce image quality deterioration that causes a moving body image after combining processing to be depicted as an unnatural image when a moving body image is combined.

It is a figure which shows schematic structure of an image processing apparatus. It is a figure which shows the structural example of the HDR process part of 1st Embodiment. It is a figure explaining the frame order of a moving image. It is a figure which shows the structural example of the image development part. It is a figure explaining the gamma curve for suitable frames and under frames. It is a figure for demonstrating the relationship between a brightness | luminance synthetic | combination ratio and a brightness | luminance. It is a figure for demonstrating motion vector detection. It is a figure explaining the relationship between the motion of a camera and the movement amount of a background image. It is a figure for demonstrating the example of calculation of a motion vector. It is a figure used for description of the motion vector of a moving body image. It is a figure explaining the motion vector for every block of a moving body image. It is a figure used for description of a process in order to determine a moving body image. It is a figure for demonstrating the image range to which a moving subject synthetic | combination ratio is applied. It is a figure for demonstrating a synthetic | combination amount and an evaluation value. It is a figure for demonstrating the use coefficient and average brightness | luminance of a suitable frame image. It is a figure which shows the structural example of the HDR process part of 2nd Embodiment. It is a figure for demonstrating the mobile body image of the same area. It is a figure which shows the structural example of the HDR process part of 3rd Embodiment. It is a figure for demonstrating a brightness | luminance difference synthetic | combination ratio. It is a figure which shows schematic structure of the information processing apparatus of 4th Embodiment. It is a figure for demonstrating the storage area of a memory. It is a schematic flowchart of the image processing of 4th Embodiment. It is a detailed flowchart of the image processing of 4th Embodiment.

  The image processing apparatus of the present embodiment is a digital camera or a digital video camera as an example, and in particular, a moving image having a wide dynamic range by performing, for example, HDR synthesizing on a plurality of frames of a digitized moving image signal. Is generated. In this embodiment, when HDR composition processing is performed on a moving subject (hereinafter referred to as a moving object), HDR composition processing is performed while maintaining the smoothness of the appearance of a moving image. Enables generation of moving images with a wide dynamic range. Specifically, in the present embodiment, when shooting a moving image, a first frame (hereinafter referred to as “appropriate frame”) with proper exposure and a second frame (hereinafter referred to as “under”) with underexposure. "Frame")) Two frame images of the image are taken alternately. Further, in the present embodiment, when HDR image synthesis is performed using two frame images captured alternately, the movement amount, area, and luminance of the moving body image are obtained as described later. In this embodiment, the composition ratio of the moving object image (hereinafter referred to as the moving object composition ratio) is controlled based on the moving amount, area, and luminance of the moving object image. Thereby, in this embodiment, even if the moving body image is a particularly conspicuous image, for example, when the moving amount of the moving body image is large or the area of the moving body image is large, It is possible to reduce image quality deterioration that causes the moving body image to appear unnatural.

<First Embodiment>
FIG. 1 is a diagram illustrating a configuration example of a camera as an example of an image processing apparatus according to the first embodiment. The camera shown in FIG. 1 includes an optical system 1, an imaging device 2, a signal processing unit 3, an HDR processing unit 4, an adjustment processing unit 5, an encoding processing unit 6, an output unit 7, and a UI unit 8. And an angular velocity sensor 9 and a bus 10.

  In FIG. 1, light passing through an optical system 1 having an iris, a lens, and the like is photoelectrically converted by an image sensor 2 constituted by a CMOS or a CCD. An image signal acquired by photoelectric conversion in the image sensor 2 is sent to the signal processing unit 3. The image sensor 2 is a so-called Bayer array imaging device. The UI unit 8 includes a key, a switch, and the like for a user interface, and a setting information generation unit that generates setting information for performing various settings for each unit of the camera based on an instruction from the user via the key and the like. Have. Based on the user's instruction, the setting information generating unit selects, for example, a moving image mode or a still image mode, setting information for controlling each part of the camera according to the selected mode, setting of the HDR mode, and its HDR Setting information for controlling each part of the camera in the mode is generated. Also, the setting information generation unit of the UI unit 8 generates ISO sensitivity settings, shutter speeds, exposure amount settings, and various setting information for shooting according to these settings. Various setting information generated by the UI unit 8 is transmitted via the bus 10 to the optical system 1, the image sensor 2, the signal processing unit 3, the HDR processing unit 4, the adjustment processing unit 5, the encoding processing unit 6, and the output unit. Sent to 7 mag.

  Hereinafter, an overview of each process in the camera of the present embodiment when the moving image mode and the HDR mode are set and moving image shooting and HDR image synthesis are performed will be described. When the moving image mode and the HDR mode are selected according to an instruction from the user, the UI unit 8 performs processing for alternately capturing images of appropriate frames and underframes and synthesizing one frame image from both frame images. Each setting information is generated. These pieces of setting information are sent to the optical system 1, the image sensor 2, the signal processing unit 3, the HDR processing unit 4 and the like as described above. In this case, the optical system 1 and the image sensor 2 alternately capture appropriate frame images and underframe images, and send the appropriate frame and underframe image signals to the signal processing unit 3.

  The signal processing unit 3 performs various processes such as A / D conversion and gain control on the appropriate frame and underframe image signals supplied from the imaging device 2, and sends the processed image data to the HDR processing unit 4. Output. Further, the angular velocity sensor 9 detects the angular velocity of the camera and outputs the detected angular velocity value to the HDR processing unit 4 via the bus 10.

  The HDR processing unit 4 uses the appropriate frame and underframe image data supplied from the signal processing unit 3 and the angular velocity value data supplied from the angular velocity sensor 9 to perform HDR image synthesis processing as described later. Run. Then, the HDR processing unit 4 outputs the combined image signal to the adjustment processing unit 5. The configuration of the HDR processing unit 4 and the composition process will be described in detail later.

  The adjustment processing unit 5 performs various adjustment processing such as luminance gain adjustment and resizing on the image data after the synthesis processing by the HDR processing unit 4, and the image data after the adjustment processing to the encoding processing unit 6. send. The encoding processing unit 6 encodes the image data after the adjustment processing in the adjustment processing unit 5 and sends the data after the encoding processing to the output unit 7. The output unit 7 outputs the encoded image data to, for example, HDMI (High Definition Multimedia Interface) (HDMI is a registered trademark), saves it in a storage medium such as a memory card, and displays a so-called rear liquid crystal of the camera Output processing such as displaying on the device.

  FIG. 2 is a diagram illustrating a configuration example of the HDR processing unit 4 of the camera of FIG. 2 includes an input terminal 401, 402, 403, a developing unit 404, 405, a luminance composition ratio calculation unit 406, a motion vector calculation unit 407, a background separation unit 408, and an alignment unit. 409. The HDR processing unit 4 includes an area calculation unit 410, a movement amount calculation unit 411, a luminance calculation unit 412, a moving body synthesis ratio calculation unit 413, a synthesis unit 414, and an output terminal 415. .

  In FIG. 2, the appropriate frame image data supplied from the signal processing unit 3 is input to the input terminal 402, and the underframe image data supplied from the signal processing unit 3 is input to the input terminal 403. Appropriate frame image data input to the input terminal 402 is sent to the developing unit 404, and underframe image data input to the input terminal 403 is sent to the developing unit 405. Further, the angular velocity value data detected by the angular velocity sensor 9 of FIG. 1 is input to the input terminal 401 and sent to the background separation unit 408.

  The development unit 404 performs development processing on the image data of the appropriate frame, and outputs the image data of the appropriate frame after the development processing to the luminance composition ratio calculation unit 406 and the motion vector calculation unit 407. The development processing refers to a raw image (RAW image) captured by the image sensor 2 that is debayered to be converted into a signal composed of luminance and color difference, and noise and optical signals are removed from the luminance and color difference signals. This is a process for performing proper distortion correction and image optimization. The development unit 405 performs development processing on the underframe image data, and outputs the underframe image data after the development processing to the luminance composition ratio calculation unit 406 and the motion vector calculation unit 407. Details of the developing units 404 and 405 will be described later.

  The luminance composition ratio calculation unit 406 calculates the luminance composition ratio between the two frame images of the appropriate frame after the development processing and the underframe after the development processing based on the luminance of the underframe image after the development processing. Then, the luminance synthesis ratio calculation unit 406 sends information on the calculated luminance synthesis ratio to the synthesis unit 414. Details of the luminance synthesis ratio calculation processing by the luminance synthesis ratio calculation unit 406 will be described later.

  The motion vector calculation unit 407 is an example of a moving body detection unit, and uses a motion vector between two frame images of an appropriate frame after the development process and an under frame after the development process with reference to an image of the appropriate frame after the development process. Is calculated. That is, the motion vector calculation unit 407 calculates a motion vector for detecting, as a moving body image, a subject image whose subject position has changed while an appropriate frame image and an underframe image have been captured. Then, the motion vector calculation unit 407 includes the area calculation unit 410, the movement amount calculation unit 411, and the luminance calculation unit based on the motion vector information, the image data of the appropriate frame after the development process, and the image data of the under frame after the development process. 412 and the background separation unit 408. Details of the motion vector calculation processing by the motion vector calculation unit 407 will be described later.

  Based on the image data, the angular velocity value, and the motion vector of the appropriate frame after the development process and the under frame after the development process, the background separation unit 408 extracts the background image and the moving body image from the appropriate frame image and the under frame image. To separate. Then, the background separation unit 408 sends the image data of the appropriate frame after the development process and the underframe after the development process, and the motion vector information corresponding to the separated background image and moving body image to the alignment unit 409. . Details of the background separation processing in the background separation unit 408 will be described later.

  The alignment unit 409 performs alignment processing for aligning the position of an appropriate frame image after development processing with respect to the underframe image after development processing based on the motion vector of the background image. Then, the alignment unit 409 converts the underframe image data after the development process, the image data of the appropriate frame after the alignment process, and the motion vector information of the moving body image into the area calculation unit 410, the movement amount calculation unit 411, This is sent to the luminance calculation unit 412. Also, the alignment unit 409 sends the image data of the appropriate frame after the alignment process and the image data of the under frame after the development process to the synthesis unit 414. Details of the alignment processing by the alignment unit 409 will be described later.

  The area calculation unit 410 is an example of an area detection unit, and based on the underframe image data after the development process, the image data of the appropriate frame after the alignment process, and the motion vector of the mobile object image, Calculate the area. Then, the area calculation unit 410 sends information on the area of the moving object image to the moving object composition ratio calculation unit 413. Details of the area calculation processing by the area calculation unit 410 will be described later.

  The movement amount calculation unit 411 is an example of a movement amount detection unit, and is based on the underframe image data after the development process, the image data of the appropriate frame after the alignment process, and the motion vector of the moving body image. The amount of image movement is calculated. In other words, the movement amount calculation unit 411 obtains the movement amount of the moving body image during the appropriate frame image shooting and the under image shooting. Then, the movement amount calculation unit 411 sends the movement amount information of the moving object image, the underframe image data after the development process, and the image data of the appropriate frame after the alignment process to the moving object composition ratio calculation unit 413. . Details of the movement amount calculation processing by the movement amount calculation unit 411 will be described later.

  The luminance calculation unit 412 is an example of luminance detection means, and based on the underframe image data after the development process, the appropriate frame image data after the alignment process, and the motion vector of the mobile object image, Calculate the brightness of the image. Then, the luminance calculation unit 412 sends the luminance information of the moving object image to the moving object composition ratio calculation unit 413. Details of the luminance calculation processing by the luminance calculation unit 412 will be described later.

  The moving object composition ratio calculation unit 413 applies the underframe image data after the development process, the image data of the appropriate frame after the alignment process, the area of the moving object image, the moving amount, and the luminance to each moving object image. A synthesis ratio (mobile body synthesis ratio) is calculated. That is, the moving body composition ratio calculation unit 413 controls the composition ratio of the moving body image of the appropriate frame image and the moving body image of the under frame image based on the moving amount, area, and luminance of the moving body image. It is an example. Then, the mobile body composition ratio calculation unit 413 sends information on the mobile body composition ratio to the composition unit 414. Details of the mobile body composition ratio calculation processing by the mobile body composition ratio calculation unit 413 will be described later.

  Based on the luminance composition ratio, the image data of the appropriate frame after the alignment process, the image data of the underframe after the development process, and the information of each moving object composition ratio, the combining unit 414 The image and the underframe image after development processing are combined. Then, the frame image data after being synthesized by the synthesis unit 414 is sent to the adjustment processing unit 5 in FIG. 1 via the output terminal 415. Details of the image composition processing by the composition unit 414 will be described later.

  Hereinafter, the synthesizing process in the HDR processing unit 4 will be described in more detail. Image data of an appropriate frame is sequentially input to the input terminal 402 for each frame. Also, underframe image data is sequentially input to the input terminal 403 for each frame. That is, in the imaging device 2 at this time, as shown in FIG. 3, imaging of the appropriate frame and the under frame is performed alternately like the appropriate frame 351r, the under frame 351u, the appropriate frame 352r, the under frame 352u,. It has been broken. The image data of the appropriate frames 351r, 352r, 353r, 354r,... Are sequentially input to the input terminal 402, and the image data of the under frames 351u, 352u, 353u, 354u,. Has been. In this embodiment, in order to simplify the explanation, the exposure amount difference between the appropriate frame image and the under frame image is set to a difference corresponding to, for example, two steps of ISO sensitivity, and images are taken in the order of the appropriate frame and the under frame. Is assumed to have been made. Of course, the difference in the exposure amount is an example. In FIG. 3, the combination of the appropriate frame and the under frame is a combination of two frames such as the appropriate frame 351r and the under frame 351u, the appropriate frame 352r and the under frame 352u,. A combination may be used.

  The developing unit 404 performs a developing process for image data of an appropriate frame, and the developing unit 405 performs a developing process for image data of an under frame. 4A shows a configuration example of the developing unit 404, and FIG. 4B shows a configuration example of the developing unit 405. The developing unit 404 illustrated in FIG. 4A includes a white balance unit 4041, an NR processing unit 4042, a color interpolation unit 4043, a matrix conversion unit 4044, a gamma conversion unit 4045, and a color adjustment unit 4046. 4B includes a white balance unit 4051, an NR processing unit 4052, a color interpolation unit 4053, a matrix conversion unit 4054, a gamma conversion unit 4055, and a color adjustment unit 4056. The development unit 405 illustrated in FIG. The developing unit 404 in FIG. 4 (a) and the developing unit 405 in FIG. 4 (b) have almost the same configuration. Therefore, the same configuration is applied to the developing unit in FIG. 4 (a). Only the configuration example 404 will be described.

  The white balance unit 4041 performs processing for adjusting so-called white balance. Specifically, the white balance unit 4041 applies a gain to each of the R, G, and B color signals so that the values of the R, G, and B color signals of the image that should be white are the same. Perform white balance adjustment processing. The NR processing unit 4042 performs noise reduction processing for reducing noise caused by a sensor (imaging device 2) or the like that is not derived from a subject or a background image in the input image. The color interpolation unit 4043 generates a color image in which R, G, and B color signals are aligned in all the pixels by interpolating the R, G, and B color mosaic images from the Bayer-array image sensor 2. The color image generated by the color interpolation unit 4043 is converted from R, G, B to Y, U, V by the matrix conversion unit 4044, and further, tone correction is performed by the gamma conversion unit 4045. Converted to a typical color image. Thereafter, the color adjustment unit 4046 performs image correction such as saturation enhancement, hue correction, and edge enhancement on the color image as color adjustment processing for improving the appearance of the image.

  Here, when image composition by HDR is performed, image data captured at different exposures are used, and thus it is necessary to apply a gain to the image data in advance to make the luminance level uniform. Since the gain value needs to be set to a value that does not cause so-called whiteout or blackout, for example, a value corresponding to a gamma curve as shown in FIG. 5 is used instead of a uniform gain value. In FIG. 5, a gamma curve 501 indicated by a solid line in the drawing indicates an example of a gamma curve for an appropriate frame, and a gamma curve 502 indicated by a dotted line in the drawing indicates an example of a gamma curve for an underframe. The gamma conversion unit 4045 performs a gamma conversion process using the gamma curve 501 in FIG. 5 for an image of an appropriate frame, and the gamma conversion unit 4055 performs a gamma conversion by the gamma curve 502 in FIG. 5 for an underframe image. Process. As can be seen from the gamma characteristics shown in FIG. 5, a gain larger than that of an appropriate frame image is applied to an underframe image, and therefore noise in the underframe image after development processing increases. There is concern. Therefore, the NR processing unit 4052 of the under-frame developing unit 405 performs stronger noise reduction (NR) processing than the NR processing unit 4042 of the appropriate frame developing unit 404. As a result, the noise amounts of the appropriate frame image after the development processing and the underframe image after the development processing are aligned. By this processing, the uncomfortable feeling of the frame image after the HDR image composition processing due to the difference between the appropriate frame image and the underframe image is reduced. Specific methods for noise reduction include various methods such as general noise reduction methods such as smoothing processing at an appropriate kernel size, and noise reduction methods using known filters such as ε filters and edge-preserving bilateral filters. Can be considered. Since these noise reduction methods are known, their description is omitted here. As to which of these noise reduction methods is used, an appropriate method may be selected as appropriate in consideration of balance with resources such as camera processing speed and memory.

  In this embodiment, the configuration of each developing unit 404, 405 is the same as an example having two developing units 404, 405. Therefore, only one developing unit is used, and parameters used for development are set appropriately. You may make it switch according to the image input of a flame | frame and an under frame.

  Returning to FIG. 2, the luminance composition ratio calculation unit 406 calculates the luminance composition ratio for each image region in accordance with the luminance of each image region of the underframe image. Each image region when the luminance composition ratio calculation unit 406 calculates the luminance composition ratio is an image region obtained by dividing a frame image into a plurality of images. For example, an image region corresponding to a block image described later when detecting a motion vector However, it may be a different image area. In the present embodiment, in order to simplify the description, it is assumed that each image region when the luminance composition ratio calculation unit 406 calculates the luminance composition ratio is an image region corresponding to a block image described later.

  FIG. 6 is an example of a graph showing the relationship between the luminance and the luminance synthesis ratio, which is referred to when the luminance synthesis ratio calculation unit 406 calculates the luminance synthesis ratio according to the luminance of the image area of the underframe. When an HDR composite image is generated as in the present embodiment, as shown in FIG. 6, the luminance composition ratio calculation unit 406 uses an image region of an appropriate frame for an image region darker than the luminance composition threshold Y1. On the other hand, the luminance composition ratio calculation unit 406 uses an underframe image region for an image region brighter than the luminance composition threshold Y2. In addition, the luminance composition ratio calculation unit 406 calculates the luminance composition ratio between the image area of the appropriate frame and the image area of the under frame for the image area having the brightness of the luminance composition threshold values Y1 to Y2 according to the luminance of the image area. Change gradually. That is, with respect to the image area having the brightness of the brightness synthesis threshold Y1 to Y2, the brightness synthesis ratio calculation unit 406 increases the brightness of the image area of the appropriate frame as the brightness of the image area is closer to the brightness synthesis threshold Y1. Generate a composite ratio. Conversely, the luminance composition ratio calculation unit 406 generates a luminance composition ratio in which the ratio of the underframe image area increases as the brightness of the image area approaches the luminance composition threshold Y2. Thereby, in the image area having the brightness of the luminance synthesis threshold values Y1 to Y2, the switching of the brightness of the image by the synthesis of the image area of the appropriate frame and the image area of the under frame becomes smooth.

  The motion vector calculation unit 407 in FIG. 2 calculates the motion vector of the moving object image between the appropriate frame after the development process and the underframe image after the development process. First, the motion vector calculation unit 407 divides an appropriate frame image after development processing and an underframe image after development processing into a plurality of block images, and performs edge detection processing for each block image. As an edge detection method, for example, a method of creating an image with a large amount of blur by applying a low-pass filter to the input image, and detecting an edge by subtracting an image with a large amount of blur from the input image Can be used. As another edge detection method, for example, a method of detecting an edge using a known differential filter, a pre-wit filter, or the like may be used. According to these edge detections, it is possible to improve the motion vector detection accuracy by detecting only the edge of the subject image, not the noise caused by the sensor. Next, as described above, the motion vector calculation unit 407 performs motion vector detection processing on the block image in which edge detection is possible in the appropriate frame image after development processing and the underframe image after development processing. As described above, the motion vector calculation unit 407 improves motion vector detection accuracy by performing motion vector detection only on a block image for which edge detection has been performed.

  Hereinafter, the details of the motion vector detection process in the motion vector calculation unit 407 will be described with reference to FIGS. 7A and 7B. Note that P1 to P9 in FIGS. 7A and 7B represent pixel values, respectively. First, the motion vector calculation unit 407 sets the position reference image for motion vector detection as a block image 701 whose edge is detected in an appropriate frame image after development processing as shown in FIG. In addition, the motion vector calculation unit 407 sets an image portion including a plurality of block images whose edges are detected in the underframe image after the development processing as a motion vector detection target image 702 for the position reference image (block image 701). . Next, the motion vector calculation unit 407 calculates the absolute value of the difference between each pixel in the block image 701 that is the position reference image and each pixel in the block image included in the motion vector detection target image 702. And the sum of the absolute differences of these pixels is obtained. The motion vector calculation unit 407 sets the processing for obtaining the sum of absolute difference values for the block image 701 of the position reference image while sequentially shifting the coordinate position for each pixel in the motion vector detection target image 702. Perform on block images. Then, the motion vector calculation unit 407 obtains a block image 703 having the smallest sum obtained by sequentially shifting the coordinate position for each pixel as described above in the motion vector detection target image 702. In FIG. 7B, when the coordinates of the upper left corner of the motion vector detection target image 702 are (x0, y0), the coordinates of the upper left corner of the block image 703 with the smallest sum are (x1, y2). An example is shown. That is, the motion vector calculation unit 407 converts the block image 703 corresponding to the coordinates (x1, y2) in the motion vector detection target image 702 into the block image 701 corresponding to the block reference image 701 in FIG. Detect as. Then, the motion vector calculation unit 407 calculates a motion vector from the coordinates of the block image 701 of the position reference image in the appropriate frame image and the coordinates of the block image 703 in the motion vector detection target image 702 in the underframe image. Ask. That is, the motion vector calculation unit 407 calculates, as a motion vector, the direction and distance represented by the difference between the coordinates of the block image 703 in the underframe image with respect to the coordinates of the block image 701 in the appropriate frame image.

  The background separation unit 408 in FIG. 2 is based on the motion vector information calculated by the motion vector calculation unit 407 as described above and the angular velocity information supplied from the angular velocity sensor 9 via the input terminal 401. A moving body image and a background image are separated. Hereinafter, the details of the separation process performed by the background separation unit 408 will be described with reference to FIGS. 8 and 9.

  FIG. 8 is a diagram schematically illustrating how the background image moves on the imaging surface of the imaging element 2 of the camera when the camera is panned from the camera position 1101a to the camera position 1101b. Although the image pickup device 2 is actually built in the camera, the camera and the image pickup device 2 are shown separately in FIG. 8 for easy understanding of the description. As shown in FIG. 8, when the camera 110 is panned from the position 1101 a to the position 1101 b and a stationary subject 1103 is photographed, the subject image formed on the imaging surface of the image sensor 2 is from the image position 1104. It moves to the image position 1105. Here, the stationary object 1103 is the background, and the movement amount 1106 of the background image on the imaging surface of the image sensor 2 is determined by the panning camera movement angle 1107 and the focal length f (mm) of the optical system 1. And can be calculated from Further, the moving angle 1107 is the angular velocity ω (angle / second) of the camera detected by the angular velocity sensor 9, the frame rate fps (frame / second), and the number of images n between the frame images sampled for calculating the moving amount. It can be calculated using (number of frames). Then, the amount of movement of the background image 1106 is converted into the number of pixels by using the pixel pitch pp (mm / number of pixels) to calculate how much the background image has moved on the imaging surface of the imaging device 2. Can do. The background separation unit 408 calculates the amount of movement of the background image on the imaging surface of the imaging device 2 using Expression (1).

  FIG. 9 is a diagram for explaining how the motion vector is used when the background separation unit 408 separates the moving body image and the background image. The horizontal axis in FIG. 9 indicates the magnitude of the motion vector, and the vertical axis indicates the quantity of motion vectors having the same magnitude. The horizontal axis in FIG. 9 distinguishes the direction of the motion vector with “0” as a boundary. Further, the movement amount of the background image (movement amount 1106 in FIG. 8) calculated by Expression (1) is shown as the movement amount 1201 in FIG. In the example of FIG. 9, there are vector groups 1211 to 1214 as a group of motion vectors having the same size, and the vector group 1214 is a motion vector corresponding to the movement amount 1201 of the background image calculated by the equation (1). It is a group. On the other hand, the vector group 1211 to 1213 is a motion vector group having a different size from the vector group 1214 corresponding to the movement amount 1201 of the background image. That is, each motion vector of the vector groups 1211 to 1213 is considered to be a motion vector of an image portion different from the vector group 1214 corresponding to the movement amount 1201 of the background image, in other words, each motion vector of the moving body image. Accordingly, the background separation unit 408 determines the motion vector supplied from the motion vector calculation unit 407 based on the relationship between the motion vector and the movement amount of the background image, and the group 1203 corresponding to the moving body image and the group 1203 corresponding to the background image. And can be separated.

  The alignment unit 409 in FIG. 2 calculates a so-called affine coefficient using the motion vector included in the group 1203 corresponding to the background image separated by the background separation unit 408. An affine coefficient is a matrix used for affine transformation combining linear transformation and parallel movement, and can be expressed by the following equation (2). In addition, since Formula (2) is a well-known formula, detailed description is abbreviate | omitted.

  In equation (2), the coordinates of the image before correction by affine transformation are (x, y), and the coordinates of the image after correction by affine transformation are (x ′, y ′), and 3 in this equation (2) A x3 matrix is an affine coefficient. The alignment unit 409 calculates an affine coefficient using the motion vector calculated from each block image by the motion vector calculation unit 407 described above. Then, the alignment unit 409 sets the pre-correction image as an appropriate frame image after development processing, calculates an affine coefficient for the under frame image after development processing, and uses the affine coefficient to affine the appropriate frame image. Perform conversion. Thereby, the alignment unit 409 can align the background image of the appropriate frame image with the background image of the underframe image.

  The area calculation unit 410 in FIG. 2 uses the moving vector of the group 1202 corresponding to the moving body, the appropriate frame image after the background image is aligned, and the underframe image after the development processing, as the moving body. Calculate the area of the image. The details of the area calculation process performed by the area calculation unit 410 will be described below with reference to FIGS. 10, 11, and 12 (a) to 12 (d).

  FIG. 10 is a diagram showing motion vector groups 1211 to 1213 included in the group 1202 corresponding to the moving subject shown in FIG. The horizontal axis in FIG. 10 indicates the magnitude of the motion vector, and the vertical axis indicates the number of motion vectors having the same magnitude. Here, for example, when the motion vector group 1211 is taken as an example, a plurality of motion vectors having the same size in the motion vector group 1211 are each motion of a plurality of block images constituting a certain moving body image. It is highly likely that it is a vector. On the other hand, even if the motion vectors in the motion vector group 1211 have the same size, there is a possibility that they are motion vectors of different moving body images. Therefore, when there are a plurality of motion vectors having the same size, it is necessary to determine whether or not each of the motion vectors is a motion vector of the same moving object image.

  FIGS. 11 and 12 (a) to 12 (d) correspond to each block image constituting the moving body image when different moving body images exist on a certain frame image. An arrangement example of each motion vector is shown. In other words, each small square area in the drawing represents a motion vector block (hereinafter referred to as an MV block 1100) corresponding to the block image. In addition, areas in which a plurality of MV blocks 1100 with hatched lines are gathered (hereinafter referred to as MV areas 1101, 1102, and 1103) are areas in which MV blocks corresponding to different moving body images are gathered. Suppose there is. The motion vectors of the MV blocks in the MV areas 1101 to 1103 are assumed to have the same size. That is, since the motion vectors of the MV blocks in the MV areas 1101 to 1103 have the same size, it is assumed that they are included in one motion vector group 1211 in FIG.

  12A to 12D show a process in which the area calculation unit 410 classifies the motion vector of each MV block 1100 shown in FIG. 11 into one of the MV areas 1101, 1102, and 1103, respectively. It is a figure for demonstrating. First, as shown in FIG. 12B, the area calculation unit 410 sets a certain MV block as a target MV block A1 when determining which MV area is included. Next, as shown in FIG. 12B, the area calculation unit 410 has the motion vectors of the eight MV blocks adjacent to the target MV block A1 substantially the same as the motion vectors of the target MV block A1 (direction). It is determined whether or not the motion vectors are substantially the same in size. In addition, as shown by the arrows in FIG. 12 (b), the eight directions are upward, downward, left, right, diagonally upper left, diagonally upper right, diagonally lower left, and diagonally lower right with respect to the target MV block A1. That's it. When there is an adjacent MV block having a motion vector substantially the same as the motion vector of the target MV block A1, the area calculation unit 410 determines that the adjacent MV block is an MV block A1 included in the same MV area as the target MV block A1. judge. Furthermore, as described above, the area calculation unit 410 sets an adjacent MV block that is determined to be an MV block having substantially the same motion vector as the target MV block A1 as a new target MV block A1, as shown in FIG. The same processing as described above is repeated as shown in FIG. Thereby, the area calculation part 410 can determine with the MV area which consists of each adjacent MV block of the substantially same motion vector as object MV block A1 corresponding to the same mobile body image. FIG. 12D shows an example after the MV areas 1101 to 1103 in FIG. 12A are classified for each motion vector of the MV block A1, the MV block A2, and the MV block A3. Then, the area calculation unit 410 obtains the number of MV blocks A1, A2, and A3 in each MV area 1101, 1102, and 1103 classified for each motion vector of the MV blocks A1, A2, and A3 as described above. . The MV blocks A1, A2, and A3 in the MV areas 1101, 1102, and 1103 correspond to the block images described above. Therefore, the area calculation unit 410 can obtain the area of the moving object image corresponding to each MV area 1101 to 1103 from the number of each MV block A1, A2, A3, that is, the number of block images N_A1, N_A2, N_A3. .

  The movement amount calculation unit 411 in FIG. 2 calculates the movement amount of the moving body image based on the motion vector of the moving body image, the appropriate frame image after the alignment process, and the underframe image after the development process. Hereinafter, the motion vectors of the MV blocks A1 to A3 in the MV areas 1101 to 1103 shown in FIG. The movement amount calculation unit 411 includes the average value M_A1 of the motion vectors of each MV block A1, the average value M_A2 of the motion vectors of each MV block A2, and the size of the motion vector of each MV block A3. An average value M_A3 is calculated. Then, the movement amount calculation unit 411 uses the average value M_A1 of the motion vectors of the MV block A1 as the movement amount of the moving body image corresponding to the MV area 1101. Similarly, the movement amount calculation unit 411 uses the average value M_A2 of the MV block A2 as the movement amount of the moving object image corresponding to the MV area 1102, and sets the average value M_A3 of the MV block A3 as the moving object image corresponding to the MV area 1103. The amount of movement.

  The luminance calculation unit 412 in FIG. 2 calculates the luminance of the moving object image based on the motion vector of the moving object image, the appropriate frame image after the alignment process, and the underframe image after the development process. Hereinafter, the luminance of the block image corresponding to each MV block A1 to A3 of each MV area 1101 to 1103 shown in FIG. The luminance calculation unit 412 calculates the average luminance K_A1 of the block image corresponding to each MV block A1, the average luminance K_A2 of the block image corresponding to each MV block A2, and the block image corresponding to each MV block A3. An average luminance value K_A3 is calculated. Then, the luminance calculation unit 412 sets the average value K_A1 of each luminance of each block image corresponding to each MV block A1 as the luminance of the moving object image corresponding to the MV area 1101. In addition, the luminance calculation unit 412 sets the average value K_A2 of each luminance of each block image corresponding to each MV block A2 as the luminance of the moving body image corresponding to the MV area 1102. Similarly, the luminance calculation unit 412 sets the average value K_A3 of each luminance of each block image corresponding to each MV block A3 as the luminance of the moving object image corresponding to the MV area 1103.

  The moving body composition ratio calculation unit 413 in FIG. 2 determines the moving body based on the appropriate frame image after the alignment process, the underframe image after the development process, and the area, moving amount, and luminance information of the moving body image. Calculate the composite ratio. FIG. 13 shows a range on the image to which the moving object synthesis ratio is applied to the moving object image. In the case of the example in FIG. 13, the range on the image to which the moving object synthesis ratio is applied to the moving object image is image ranges 1301, 1302, and 1303 indicated by shading in the drawing. The image range 1301 linearly represents the block image before movement corresponding to the motion vector, the block image after movement, and both the block images before and after the movement in each block image of the moving body image of an appropriate frame after the alignment process. It consists of regions generated by connecting. The image ranges 1302 and 1303 are generated in the same manner as the image range 1301. The moving body composition ratio calculation unit 413 sets these image ranges 1301 to 1303 as ranges on the image to which the moving body composition ratio is applied.

  Next, the mobile body composition ratio calculation unit 413 calculates an evaluation value for calculating the mobile body composition ratio. In the present embodiment, the evaluation value is a value for evaluating the moving amount and the area size of the moving object image. Taking the above-mentioned MV blocks A1, A2, A3 as an example, the moving body composition ratio calculation unit 413 uses the above-described number N_A1, N_A2, N_A3 and the average values M_A2, M_A3, M_A3, respectively, according to Equation (3). Evaluation values E_A1, E_A2, and E_A3 are calculated.

E_A1 = N_A1 × M_A1
E_A2 = N_A2 × M_A2 (3)
E_A3 = N_A3 × M_A3

  FIG. 14 shows a moving body of the evaluation value E, the appropriate frame image after the alignment process, and the underframe image after the development process when the evaluation value obtained by generalizing the evaluation values E_A1, E_A2, and E_A3 is “E”. It is a figure which shows the relationship with the synthetic | combination amount T in an image. In the graph of FIG. 14, the horizontal axis indicates the evaluation value E, and the vertical axis indicates the composite amount T. “E1” is set as the first evaluation threshold, and [E2] is set as the second evaluation threshold. Yes. According to the graph of FIG. 14, when the evaluation value E is in the range from “0” to “E1”, that is, when the evaluation value E is less than or equal to the first evaluation threshold (E1 or less), the composition amount T is the first composition. The quantity is fixed at “0.5”. When the evaluation value E is equal to or greater than “E2”, that is, equal to or greater than the second evaluation threshold, the composite amount T is constant at a second composite amount that is smaller than “0.5” and larger than “0”. On the other hand, when the evaluation value E is in the range from “E1” to “E2”, the composite amount T decreases as the evaluation value E increases. That is, in the example of FIG. 14, in the range from “E1” to “E2”, the amount of synthesis T in the moving body image of the appropriate frame image and the moving body image of the underframe image decreases as the evaluation value E increases. Is set to However, the value of the synthesis amount T calculated here may be the following first and second synthesis ratio patterns.

First ratio pattern → appropriate frame image: T, underframe image: 1-T
Second ratio pattern → appropriate frame image: 1-T, underframe image: T

  That is, how to use the composite amount T differs between the first ratio pattern and the second ratio pattern depending on which of the appropriate frame image and the underframe image is used as a reference. In the case of the present embodiment, the moving body composition ratio calculation unit 413 moves based on the moving body image of the appropriate frame image based on the luminance of the moving body image of the appropriate frame image and the luminance of the moving body image of the underframe image. Determine body synthesis ratio. Specifically, the moving body composition ratio calculation unit 413 obtains the use coefficient q of the appropriate frame image from the luminance of the moving body image of the appropriate frame image and the luminance of the moving body image of the underframe image, and uses the use coefficient q. The final moving object composition ratio is calculated based on the composition amount T of each block image. Hereinafter, description will be made using, for example, the synthesis amounts T_A1, T_A2, and T_A3 corresponding to the MV blocks A1, A2, and A3 and the use coefficients q obtained by generalizing the use coefficients q_A1, q_A2, and q_A3 of the appropriate frame images.

  FIG. 15 shows the average luminance K obtained by generalizing the average luminance magnitudes K_A1, K_A2, and K_A3 of the block images corresponding to the MV blocks A1, A2, and A3 of the moving object image, and the use coefficient q of the appropriate frame image. It is a graph which shows a relationship. In the graph of FIG. 15, the horizontal axis indicates the average luminance K of the moving body image, the vertical axis indicates the use coefficient q of the appropriate frame image, “K1” as the first luminance threshold, and “K2” as the second luminance threshold. Is set. According to the graph of FIG. 15, when the average luminance K is in the range of “0” to “K1”, that is, the first luminance threshold or less (“K1” or less), the ratio based on the appropriate frame image is large. Made. That is, when the average luminance K is equal to or lower than “K1”, the use coefficient q of the appropriate frame image is set to “1”. Further, when the average luminance K is “K2” or more, that is, the second luminance threshold or more, the ratio based on the underframe image is increased. That is, when the average luminance K is “K2” or more, the use coefficient q of the appropriate frame image is set to “0”. On the other hand, when the average luminance K is in the range of “K1” to “K2”, the ratio between the appropriate frame image and the underframe image is gradually changed. That is, in the range from “K1” to “K2”, the usage coefficient q is set such that the proportion of the moving body image of the appropriate frame image gradually increases as the average luminance K increases. Then, the moving body composition ratio calculation unit 413 obtains the use coefficient q based on the graph of FIG. 15, and obtains the final appropriate frame image by the following equation (4) using the use coefficient and the composition amount T. Is calculated.

    D = T × (1-q) + (1-T) × q (4)

  The combining unit 414 in FIG. 2 combines the appropriate frame image after the alignment processing and the underframe image after the development processing according to the moving object combining ratio D for the moving object image, while the background image. On the other hand, the synthesis is performed according to the luminance synthesis ratio. That is, when the luminance composition ratio is “J”, the composition unit 414 calculates a composite image for each pixel by the following equation (5).

Background image: Underframe image × J + appropriate frame image × (1-J)
Mobile object image: underframe image × (1-D) + appropriate frame image × D
.... Formula (5)

  As described above, according to the present embodiment, it is possible to suppress the deterioration of the smoothness of the moving image due to the moving body image being combined and the frame rate being lowered by the HDR image combining processing for the moving image. Yes. Further, in the present embodiment, by controlling the moving object composition ratio based on the area, moving amount, and brightness of the moving object image, it is possible to prevent unnatural image composition from being performed, and to optimize the moving image. HDR images can be obtained. In particular, in the present embodiment, when the moving body image is a conspicuous image, for example, when the moving amount of the moving body image is large or the area of the moving body image is large, the moving body image is obtained by the HDR synthesizing process. It is possible to reduce image quality deterioration that is rendered as an unnatural image. That is, according to the present embodiment, it is possible to realize a process that suppresses the deterioration of the smoothness of the moving image while maintaining the effect of the image composition process by HDR, and further, the depiction that the moving body image looks unnatural. Can be prevented.

<Second Embodiment>
Next, an image processing apparatus according to the second embodiment will be described below. As in the first embodiment, the image processing apparatus according to the second embodiment performs HDR composition processing on a moving body image of appropriate frames and underframes that are alternately captured by moving image shooting. Based on the moving amount, the area, and the luminance, the moving body synthesis ratio is controlled. Furthermore, in the second embodiment, the areas of the moving body images are compared among the respective moving body images, and the moving body synthesis ratios of the moving body images having substantially the same area are made closer (approximately the same ratio). And adjust the moving object composition ratio after the adjustment. In the second embodiment, since the moving object images having the same area are combined with the moving object combining ratios close to each other, it is possible to generate an HDR combined image with less unnatural appearance.

  The configuration of a camera, which is an example of an image processing apparatus according to the second embodiment, is the same as that in FIG. FIG. 16 shows a configuration example of the HDR processing unit 4 in the second embodiment. Note that the HDR processing unit 4 shown in FIG. 16 is mostly the same as the HDR processing unit 4 of the first embodiment, and therefore differs from the HDR processing unit 4 of the first embodiment in the following description. Only the part will be described.

  In the HDR processing unit 4 of the second embodiment shown in FIG. 16, the information on the area of the moving object image calculated by the area calculating unit 410 is sent to the moving object composition ratio calculating unit 413 and the same area correcting unit 416. It is done. The same area correction unit 416 corrects the moving object composition ratio based on the moving object composition ratio that is the output of the moving object composition ratio calculation unit 413 and the area of the moving object image that is the output of the area calculation unit 410, The corrected mobile body composition ratio is sent to the composition unit 414.

  Hereinafter, the correction process of the moving body composition ratio performed by the same area correction unit 416 will be described in more detail. FIG. 17 is a diagram illustrating an example when a plurality of moving body images having the same area exist in the frame image. In FIG. 17, image ranges 1701, 1702, and 1703 indicated by shading in the figure are image ranges to which the moving object composition ratio is applied to the moving object image obtained based on the motion vector as described above. Show. In FIG. 17, “A4, A5, A6” representing MV blocks corresponding to the motion vectors of the respective moving body images are also drawn. The area of the moving object image corresponding to each of the image ranges 1701, 1702, and 1703 is obtained from the number of MV blocks in the image range. Here, the number of MV blocks in the moving body image in the image range 1701 is N_A4, the number of MV blocks in the moving body image in the image range 1702 is N_A5, and the number of MV blocks in the moving body image in the image range 1703 is N_A5. In the example of FIG. 17, it is assumed that the numbers N_A4, N_A5, and N_A6 of the MV blocks are the same (N_A4 = N_A5 = N_A6).

  At this time, the same area correction unit 416 is supplied with the moving body synthesis ratios D_A4, D_A5, and D_A6 of the appropriate frames calculated by the moving body synthesis ratio calculation unit 413 corresponding to the MV blocks A4, A5, and A6. When the number N_A4, N_A5, N_A6 of these MV blocks is smaller than the threshold value L and the number thereof is equal (N_A4 = N_A5 = N_A6), the same area correction unit 416 performs the corrected movement by the following equation (6). The body synthesis ratio D2 is calculated. That is, the same area correction unit 416 calculates the average composition ratio of the mobile object composition ratios D_A4, D_A5, and D_A6 of the appropriate frame as the corrected mobile object composition ratio D2.

  D2 = (D_A4 + D_A5 + D_A6) / 3 Formula (6)

  The same area correction unit 416 sends the corrected moving body composition ratio D2 calculated by the above processing to the composition unit 414. The combining unit 414 uses the corrected moving body combining ratio D2 for each moving body image corresponding to the image ranges 1701, 1702, and 1703, and uses the corrected moving body combining ratio D2 in the same manner as in the first embodiment described above. The moving body image is synthesized.

  According to the second embodiment described above, the same effect as that of the first embodiment can be obtained. Furthermore, in the second embodiment, correction is performed so that the moving object composition ratio is approximately the same for moving object images in which the area of the moving object image is smaller than the threshold value L and the area is substantially equal. I have to. As a result, according to the second embodiment, the moving body images having the same area are combined with each other at the moving body combining ratios close to each other, so that it is possible to generate an HDR combined image with less unnatural appearance. .

<Third Embodiment>
Next, an image processing apparatus according to the third embodiment will be described below. As in the first embodiment, the image processing apparatus according to the third embodiment performs HDR composition processing on a moving body image of appropriate frames and underframes alternately captured by moving image shooting. Based on the moving amount, the area, and the luminance, the moving body synthesis ratio is controlled. Furthermore, in the third embodiment, the moving object composition ratio is controlled based on not only the motion vector information but also the difference value between the appropriate frame image and the underframe image. That is, in the third embodiment, it is possible to calculate in detail the moving object composition ratio with respect to the moving object image, based on the movement of the pixel unit in the moving object image of the appropriate frame image and the under frame image.

  The configuration of a camera, which is an example of an image processing apparatus according to the third embodiment, is the same as that in FIG. FIG. 18 shows a configuration example of the HDR processing unit 4 in the third embodiment. Note that the HDR processing unit 4 shown in FIG. 18 is mostly the same as the HDR processing unit 4 of the first embodiment, and therefore differs from the HDR processing unit 4 of the first embodiment in the following description. Only the part will be described.

  In the HDR processing unit 4 illustrated in FIG. 18, the luminance difference combination ratio calculation unit 417 is an example of a luminance difference detection unit, and the image data of an appropriate frame after the alignment process, which is an output of the alignment unit 409, and the development process The subsequent underframe image data is input. The luminance difference composition ratio calculation unit 417 calculates a luminance difference composition ratio based on the image data of the appropriate frame after the alignment process and the image data of the under frame after the development process. Information on the luminance difference synthesis ratio calculated by the luminance difference synthesis ratio calculation unit 417 is sent to the moving object synthesis ratio calculation unit 413. The moving body composition ratio calculation unit 413 corrects the above-described moving body composition ratio D based on the luminance difference composition ratio and calculates a corrected moving body composition ratio D3.

  Hereinafter, the luminance difference composition ratio calculation unit 417 and the moving object composition ratio calculation unit 413 according to the third embodiment will be described in more detail. The luminance difference composition ratio calculation unit 417 calculates the absolute difference value P of the luminance of each pixel of the appropriate frame image after the alignment processing and the underframe image after the development processing. FIG. 19 is a graph showing the relationship between the absolute difference value P and the luminance difference composition ratio PK. In the graph of FIG. 19, the horizontal axis represents the difference absolute value P, the vertical axis represents the luminance difference composition ratio PK, “P1” is set as the first difference threshold, and “P2” is set as the second difference threshold. Yes. According to the example of FIG. 19, when the difference absolute value P is in the range of 0 to P1, that is, the first difference threshold value or less (P1 or less), the luminance difference composition ratio PK is the first ratio of zero (0). Made as a percentage. When the difference absolute value P is equal to or greater than P2, that is, equal to or greater than the second difference threshold, the luminance difference composition ratio PK is set to 100%, which is the second ratio. On the other hand, when the difference absolute value P is in the range of P1 to P2, the luminance difference composition ratio PK is gradually increased from zero percent to 100 percent. That is, when the difference absolute value P is in the range of P1 to P2, a luminance difference composition ratio PK that changes from zero percent to 100 percent as the difference absolute value P increases is set for each pixel. The moving body composition ratio calculation unit 413 uses the luminance difference composition ratio PK calculated based on the graph of FIG. 19 and the moving body composition ratio D calculated as in the first embodiment, for each pixel. The corrected mobile body composition ratio D3 is calculated by the equation (7).

  D3 = PK × D (7)

  The mobile body composition ratio calculation unit 413 sends the corrected mobile body composition ratio D3 calculated as described above to the composition unit 414. The combining unit 414 uses the corrected moving body combining ratio D3 to combine the appropriate frame and underframe moving body images on a pixel-by-pixel basis as in the case of the first embodiment described above.

  According to the third embodiment described above, the same effect as in the case of the first embodiment can be obtained. Furthermore, in the third embodiment, since the moving object combining ratio is calculated in detail based on the movement of the moving object image in units of pixels, it is possible to generate a natural combined image with very little visual discomfort. Become.

<Fourth Embodiment>
The image processing apparatuses in the first, second, and third embodiments described above can also be realized by an information processing apparatus such as a personal computer and a computer program executed by the information processing apparatus. Hereinafter, as a fourth embodiment, a case where a personal computer executes a program and realizes the above-described image composition processing and the like will be described.

  FIG. 20 is a diagram illustrating a schematic configuration of an information processing apparatus according to the fourth embodiment. In FIG. 20, a CPU 300 is a central processing unit that controls the entire information processing apparatus according to the present embodiment and performs various processes. The memory 301 includes a ROM that stores a so-called BIOS and a boot program, and a RAM that the CPU 300 uses as a work area. The instruction input unit 303 includes, for example, a pointing device such as a keyboard and a mouse, various switches, and the like. The external storage device 304 is, for example, a hard disk device or the like, and executes an operating system (OS) that executes various controls of the information processing apparatus according to the present embodiment and various processes described in the first to third embodiments. A computer program to be stored is stored. The external storage device 304 is also used as a storage area necessary for various calculations. The storage device 305 is a device for accessing a portable storage medium (for example, a BD-ROM or a DVD-ROM disc) that stores moving image data, for example. A bus 302 is a bus through which image data is exchanged between the computer and an external interface. The digital camera 306 is a camera for capturing a moving image or a still image, and communicates with the information processing apparatus of the fourth embodiment by wire or wireless. The digital camera 306 includes the optical system 1, the image sensor 2, and the angular velocity sensor 9 in FIG. 1, and transmits captured image data and information on the angular velocity detected by the angular velocity sensor 9 to the information processing apparatus. The display 307 is a display device for displaying a moving image, a still image, an image after combining processing by HDR in the present embodiment, and the like. The communication line 309 includes, for example, a LAN, a public line, a wireless line, and a broadcast radio wave. The communication interface 308 transmits and receives image data and the like via the communication line 309.

  The information processing apparatus shown in FIG. 20 will be described in more detail. First, when power is supplied in response to a power-on instruction from the user via the instruction input unit 303, the CPU 300 loads the OS of the external storage device 304 into the memory 301 according to a boot program stored in the ROM of the memory 301. (RAM). Then, in accordance with an instruction from the user via the instruction input unit 303, the CPU 300 loads an application program from the external storage device 304 to the memory 301. As a result, the information processing apparatus can execute each function of the image processing apparatus described in the above embodiments.

  FIG. 21 is a diagram illustrating an example of a storage area of the memory 301 when the application program is loaded into the memory 301. That is, storage areas 2101 to 2106 are prepared in the memory 301 at this time. A storage area 2101 is an area where an OS (operating system) for controlling the entire information processing apparatus by the CPU 300 and controlling the execution of various software is developed. The storage area 2102 is an area where a video processing software program for the CPU 300 to execute HDR image composition processing is developed. In the storage area 2103, an image input software program for allowing the CPU 300 to control the camera 306 to alternately capture images of appropriate frames and underframes and input (capture) those frame images frame by frame is developed. It is an area. The storage area 2104 is an area where a video display software program for executing processing for generating image data to be displayed on the display 307 by the CPU 300 is developed. A storage area 2105 is an image area in which image data is stored. The storage area 2106 is a working area for storing various parameters, for example.

  FIG. 22 is a flowchart when the CPU 300 executes the application program of this embodiment and performs the image processing of each embodiment described above. In FIG. 22, the CPU 300 first initializes each unit as the processing of step S1. After step S1, CPU 300 advances the process to step S2.

  In step S2, CPU 300 determines whether or not to end the application program. The end determination in step S <b> 2 is performed based on whether or not the user inputs an end instruction via the instruction input unit 303. That is, if it is determined in step S2 that an end instruction has been input from the user via the instruction input unit 303, the CPU 300 ends the process of this flowchart. On the other hand, if the CPU 300 determines in step S2 that an end instruction has not been input from the user via the instruction input unit 303, the process proceeds to step S3.

  In step S <b> 3, the CPU 300 stores the frame-unit image data captured by the camera 306 in the image area in the storage area 2105 of the memory 301. After step S3, CPU 300 advances the process to step S4. In step S4, the CPU 300 executes various types of image processing such as HDR image composition processing as described in the above embodiment. After step S4, the CPU 300 returns the process to step S2.

  FIG. 23 is a flowchart showing in more detail the image processing in step S4 of FIG. First, as a process of step S401, the CPU 300 stores, in the memory 301, parameters such as two frames of image data of an appropriate frame and an underframe that are continuous in time among the image data stored in the storage device 305, and angular velocity. Remember me. After step S401, the CPU 300 advances the process to step S402.

  In step S402, the CPU 300 develops the two frames of image data stored in the memory 301 so as to match the respective brightness as described above. After step S402, CPU 300 advances the process to step S403. In step S403, the CPU 300 calculates a luminance synthesis ratio with respect to the underframe. After step S403, the CPU 300 advances the process to step S404.

  In step S404, the CPU 300 calculates a motion vector between the appropriate frame image and the underframe image. After step S404, CPU 300 advances the process to step S405. In step S405, the CPU 300 separates the moving body image and the background image using the motion vector and the angular velocity. After step S405, CPU 300 advances the process to step S406.

  In step S406, the CPU 300 aligns the background image of the appropriate frame with the background image of the underframe using the motion vector. After step S406, CPU 300 advances the process to step S407.

  In step S407, the CPU 300 calculates the area of the moving object image using the motion vector. After step S407, CPU 300 advances the process to step S408. In step S408, the CPU 300 calculates the moving amount of the moving body image using the motion vector. After step S408, CPU 300 advances the process to step S409. In step S409, the CPU 300 calculates the luminance of the moving object image using the motion vector. After step S409, CPU 300 advances the process to step S410.

  In step S410, the CPU 300 calculates a moving body composition ratio based on the area, moving amount, and luminance of the moving body image. After step S410, CPU 300 advances the process to step S411. In step S <b> 411, the CPU 300 uses the luminance composition ratio and the moving object composition ratio to perform composition processing of the appropriate frame and underframe images, and stores the frame image data after the composition processing in the memory 301.

  As described above, according to the fourth embodiment, in the information processing apparatus such as a personal computer, it is possible to realize an image composition process equivalent to each of the first to third embodiments described above. The computer program is normally stored in a computer readable device, and can be executed by setting it in a reading device of the computer and copying or installing it in an internal memory or the like. Therefore, it goes without saying that such a computer-readable storage medium falls within the category of this embodiment.

  In addition, a program that realizes one or more functions of the above-described embodiments may be supplied to a system or an apparatus via a network or a storage medium. One or more processors in the computer of the system or apparatus read and execute the program. In addition, it can be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  In the above-described embodiment, HDR image synthesis processing is taken as an example, but in addition, for example, a plurality of frame images with different focus distances in the focus direction, or a plurality of colors with different colors. The present invention can also be applied to various image composition processing such as frame composition processing.

  DESCRIPTION OF SYMBOLS 1 Optical system, 2 Image pick-up element, 3 Signal processing part, 4 HDR processing part, 5 Adjustment processing part, 6 Coding process part, 7 Output part, 8 UI part, 9 Angular velocity sensor, 404,405 Development part, 406 Luminance composition Ratio calculation unit, 407 Motion vector calculation unit, 408 Background separation unit, 409 Position adjustment unit, 410 Area calculation unit, 411 Movement amount calculation unit, 412 Luminance calculation unit, 413 Mobile body composition ratio calculation unit, 414 Composition unit, 416 Identical Area correction unit, 417 Luminance difference composite ratio calculation unit

Claims (13)

A subject whose subject position has changed between the shooting of the first frame image and the shooting of the second frame image from at least two frame images of the first frame image and the second frame image by moving image shooting Moving body detecting means for detecting an image as a moving body image;
From the first frame image and the second frame image, a moving amount detection means for obtaining a moving amount of the moving body image during the shooting of the first frame image and the shooting of the second frame image. When,
Area detecting means for determining the area of the moving object image;
Combining means for combining the first frame image and the second frame image;
The moving body image of the first frame image when the first frame image and the second frame image are combined by the combining unit based on the moving amount and the area of the moving body image. And a control means for controlling a synthesis ratio of the second frame image and the moving body image.   The first frame image is a frame image taken with a first exposure amount, and the second frame image is a frame taken with a second exposure amount that is smaller than the first exposure amount. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image.   The control means obtains an evaluation value representing the amount of movement and the size of the area of the moving body image, and the amount of synthesis of the moving body image of the first frame image decreases as the evaluation value increases. The image processing apparatus according to claim 2, wherein the composition ratio is controlled to be The control means includes
When the evaluation value is equal to or less than a first evaluation threshold, the synthesis ratio is controlled so that the synthesis amount is constant at the first synthesis amount;
When the evaluation value is greater than or equal to a second evaluation threshold greater than the first evaluation threshold, the synthesis ratio is controlled so that the synthesis amount is constant at a second synthesis amount smaller than the first synthesis amount. And
When the evaluation value is between the first evaluation threshold and the second evaluation threshold, the amount of synthesis decreases from the first synthesis amount to the second synthesis amount as the evaluation value increases. The image processing apparatus according to claim 3, wherein the composition ratio is controlled to be Brightness detecting means for detecting the brightness of the moving body image of the first frame image and the brightness of the moving body image of the second frame image;
The control means is configured to determine the first frame image based on the luminance of the moving body image of the first frame image detected by the luminance detecting means and the luminance of the moving body image of the second frame image. The image processing apparatus according to claim 3, wherein the composition ratio based on the moving body image is determined. The luminance detection means obtains an average luminance of the moving body image of the first frame image and the moving body image of the second frame image;
The control means includes
If the average luminance is less than or equal to a first luminance threshold, set a coefficient to increase the proportion of the moving body image of the first frame image,
When the average luminance is equal to or higher than a second luminance threshold higher than the first luminance threshold, the coefficient for increasing the ratio of the moving body image in the second frame image is set.
When the average luminance is between the first luminance threshold and the second luminance threshold, the coefficient that increases the ratio of the moving body image in the first frame image as the average luminance decreases. Set,
The image processing apparatus according to claim 5, wherein the composition ratio based on the moving body image of the first frame image is determined based on the coefficient and the composition amount.   The control means includes the moving body image of the first frame image, the moving body image of the second frame image, the moving body image of the first frame image, and the second frame image. The range of an image formed by linearly connecting the moving body images is set as a range of a moving body image to be combined by the combining ratio. An image processing apparatus according to 1. The moving body detecting means detects a plurality of moving body images,
The area detecting means detects each area of the plurality of moving body images,
The control means controls the composition ratio for each of the plurality of moving body images, and the composition ratio of each of the plurality of moving body images having substantially the same area is substantially the same. It controls so that it may become. The image processing apparatus of any one of Claims 1-7 characterized by the above-mentioned.   The control means obtains an average composition ratio from the composition ratio of each mobile object image having substantially the same area, and uses the average composition ratio as the composition ratio of each mobile object image having substantially the same area. The image processing apparatus according to claim 8. A luminance difference detecting means for obtaining a luminance difference in pixel units between the moving body image of the first frame image and the moving body image of the second frame image;
The image processing apparatus according to claim 2, wherein the control unit controls the synthesis ratio in units of pixels of the moving body image based on a luminance difference in units of pixels. . The brightness difference detecting means includes
If the luminance difference in pixel units is less than or equal to the first difference threshold, a luminance difference synthesis ratio that is the first ratio is set in pixel units,
If the luminance difference in pixel units is equal to or greater than a second difference threshold value that is greater than the first difference threshold value, a luminance difference composition ratio that is a second ratio that is greater than the first ratio is set in pixel units;
When the luminance difference of the pixel unit is between the first difference threshold and the second difference threshold, the ratio is increased from the first ratio to the second difference as the luminance difference of the pixel unit increases. Set the luminance difference composition ratio that changes to the ratio in pixel units,
The said control means correct | amends the said composition ratio on the basis of the said moving body image of the said 1st frame image by a pixel unit by the brightness | luminance difference composition ratio of the said pixel unit. Image processing device. The moving body detecting means is configured to capture the first frame image and the second frame image from at least two frame images of the first frame image and the second frame image obtained by moving image shooting. Detecting a subject image whose subject position has changed as a moving body image;
The moving amount detection means moves the moving body image from the first frame image and the second frame image during the shooting of the first frame image and the shooting of the second frame image. A step of seeking
An area detecting means for determining an area of the moving object image;
Combining means for combining the first frame image and the second frame image;
Based on the amount of movement and the area of the moving body image, the control unit is configured to display the first frame image when the first frame image and the second frame image are combined by the combining unit. And a step of controlling a composition ratio of the moving body image and the moving body image of the second frame image.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-11.

Images