JP7157526B2 - IMAGE PROCESSING DEVICE, IMAGING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

2. Description of the Related Art In recent years, many imaging apparatuses such as digital cameras and digital video cameras that synthesize a plurality of images and record the synthesized image have been commercialized. Among these imaging apparatuses, there is an imaging apparatus that has a function of synthesizing a plurality of images shot at different timings to generate a synthetic image with reduced random noise. As a result, the user can obtain a composite image with reduced random noise compared to a single image that is not composited.

However, when a subject moves while a plurality of images are being captured, the moved subject may become a multiple image in the composite image. In order to suppress the occurrence of multiple images, a technique has been disclosed that prohibits compositing processing for an area in which movement of an object is detected. Japanese Unexamined Patent Application Publication No. 2002-100003 discloses a technique for detecting a moving object region based on the absolute value of the inter-frame difference between a plurality of images that have been reduced at a reduction ratio determined according to the amount of camera shake. Further, in Patent Document 2, a reduced image is selected according to the positional deviation remaining after alignment of a plurality of images, and a moving object region is detected based on the absolute value of the inter-frame difference of the selected plurality of reduced images. A technique for doing so is disclosed.

In the moving object region detection based on the inter-frame difference absolute value, for example, in the case of an image shot with high sensitivity, the inter-frame difference absolute value increases due to random noise. Therefore, it becomes difficult to distinguish between a moving object and random noise, and there is a possibility that an area with large random noise is erroneously detected as a moving object area.

Therefore, in Patent Document 1, by changing the image reduction ratio based on the amount of camera shake, the shift in the position of a still subject due to camera shake and the effects of random noise are reduced, and the detection accuracy of the moving subject area is improved. . In addition, in Patent Document 2, by selecting a reduced image according to the positional deviation remaining after aligning a plurality of images, the positional deviation of a still object due to camera shake and the effects of random noise are reduced, and the movement of the subject is reduced. The detection accuracy of the subject area has been improved.

JP 2013-62741 A Japanese Patent No. 5398612

However, when a moving object area is detected based on the absolute value of the inter-frame difference between reduced images, by enlarging the moving object area detected between the reduced images to the original resolution, even the still area around the moving object is detected as the moving object area. There are times when it is done. In addition, by reducing the image, the amplitude of the pixel value of a small subject becomes low, so detection of a small moving subject may fail.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for improving detection accuracy of a moving object region based on differences in pixel values between images.

In order to solve the above problems, the present invention provides a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from the first reference image. reduction means for generating a second reference image and a second reference image by reducing each of the reference images by a first magnification; and pixel values of the first reference image and pixel values of the first reference image and A first moving object likelihood is obtained based on the difference between the pixel values of the first reference image, and a second moving object likelihood is obtained based on the difference between the pixel values of the second reference image and the second reference image. Based on at least one of the first acquisition means for acquiring the moving object likelihood of the first reference image, the first reference image, the second reference image, and the second reference image, a second obtaining means for obtaining an edge degree at each position in the imaging range; and combining the first moving object likelihood and the second moving object likelihood based on the edge degree for each position in the imaging range. determining means for determining a ratio; and generating a synthesized moving object likelihood by synthesizing the first moving object likelihood and the second moving object likelihood according to the determined synthesis ratio for each position in the imaging range. and synthesizing means for combining images.

In addition, other features of the present invention will be further clarified by the description in the attached drawings and the following detailed description.

According to the present invention, it is possible to improve the detection accuracy of a moving object area based on the pixel value difference between images.

FIG. 2 is a block diagram showing the configuration of the imaging device 100; FIG. 3 is a diagram showing the configuration of a composite image generation unit 200 included in an image processing unit 107; 4 is a flowchart of processing executed by a composite image generation unit 200; 4A and 4B are diagrams for explaining alignment processing; FIG. FIG. 10 is a diagram showing the relationship between the composite moving object likelihood and the composite ratio w of the reference image; FIG. 4 is a diagram showing the configuration of a moving object region detection unit 202 according to the first embodiment; 4 is a flowchart of synthetic moving object likelihood calculation processing executed by a moving object region detection unit 202 according to the first embodiment; The figure which shows a moving object likelihood calculation curve. The figure which shows an edge degree calculation curve. The figure which shows the calculation curve of the synthetic|combination ratio of moving object likelihood. FIG. 4 is a conceptual diagram of a standard image and a reference image of the same size (before reduction). (A) (B) Conceptual diagram of reduced image data, (C) (D) Conceptual diagram of moving object likelihood, (E) (F) Conceptual diagram of edge degree, (G) Combining ratio of moving object likelihood Conceptual diagram, (H) Conceptual diagram of synthetic moving object likelihood. FIG. 10 is a diagram showing a curve for calculating a composite ratio of moving object likelihoods during high-sensitivity imaging; Schematic diagram showing the motion likelihood at three different resolutions and the final composite motion likelihood. 10 is a flowchart of synthetic moving object likelihood calculation processing executed by a moving object region detection unit 202 according to the second embodiment; The figure which shows the structure of the moving body area|region detection part 202 which concerns on 2nd Embodiment. FIG. 4 is a diagram showing SAD between a standard image and a reference image in each layer; FIG. 10 is a diagram for explaining a process of determining a synthesis ratio of SADs in x1/4 hierarchy; FIG. 11 is a diagram for explaining a process of determining a synthesis ratio of SADs in the x1/1 layer; The figure explaining the setting method of a synthetic|combination ratio conversion curve. FIG. 5 is a diagram for explaining a specific example of processing for changing a synthesis ratio conversion curve according to ISO sensitivity; FIG. 16 is a view for explaining details of SAD difference calculation processing in S1504 of FIG. 15;

Embodiments of the present invention will be described below with reference to the accompanying drawings. It should be noted that the technical scope of the present invention is determined by the claims and is not limited by the following individual embodiments. Also, not all combinations of features described in the embodiments are essential to the present invention. It is also possible to combine features described in separate embodiments as appropriate.

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of an imaging device 100, which is an example of an image processing device. In FIG. 1, the control unit 101 is, for example, a CPU, reads an operation program of each block provided in the imaging apparatus 100 from a ROM 102 described below, develops it in a RAM 103 described below, and executes it, thereby executing each block included in the imaging apparatus 100. controls the behavior of The ROM 102 is an electrically erasable/recordable non-volatile memory, and stores an operation program for each block included in the imaging apparatus 100 as well as parameters required for the operation of each block. A RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data output during operation of each block included in the imaging apparatus 100 .

The optical system 104 includes a lens group including a zoom lens and a focus lens, and forms a subject image on an imaging unit 105, which will be described later. The imaging unit 105 is, for example, an imaging device such as a CCD or CMOS sensor. Output. The A/D converter 106 converts the input analog image signal into a digital image signal and outputs the obtained digital image signal (image data) to the RAM 103 . Also, the A/D conversion unit 106 amplifies the analog image signal or the digital image signal based on the amplification factor (sensitivity information) determined by the control unit 101 .

The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the RAM 103 . The image processing unit 107 also includes a composite image generation unit 200, which will be described later, and combines a plurality of image data stored in the RAM 103 to generate a composite image.

A recording unit 108 is a detachable memory card or the like. Image data processed by the image processing unit 107 is recorded as a recorded image in the recording unit 108 via the RAM 103 . A display unit 109 is a display device such as an LCD, and displays image data recorded in the RAM 103 and the recording unit 108, a user interface for receiving instructions from the user, and the like.

Next, the operation of the image processing unit 107 will be described in detail. In the present embodiment, an example of generating a composite image with reduced random noise by combining two images based on a composite moving object likelihood, which will be described later, will be described.

The configuration of the composite image generation unit 200 included in the image processing unit 107 will be described with reference to FIG. A composite image generation unit 200 composites two images stored in the RAM 103 to generate a composite image. The composite image generation unit 200 includes an alignment unit 201 , a moving object area detection unit 202 and an image composition unit 203 .

Next, referring to FIG. 3, processing executed by the synthetic image generation unit 200 will be described. In S<b>301 , the control unit 101 selects a reference image that serves as a reference for synthesis and a reference image that is to be synthesized with respect to the reference image from a plurality of images stored in the RAM 103 . For example, the control unit 101 selects the first image captured immediately after the shutter of the imaging device 100 is pressed as the reference image, and selects the second and subsequent images as the reference image. The composite image generation unit 200 acquires the selected standard image and reference image from the RAM 103 .

In S302, the alignment unit 201 detects a motion vector between the standard image and the reference image, and geometrically transforms the reference image based on the motion vector, thereby aligning the position of the reference image with the standard image.

Alignment processing will now be described with reference to FIG. 4A shows the reference image, FIG. 4B shows the reference image, and FIG. 4C shows the aligned reference image. The three images shown in FIGS. 4(A) to 4(C) are different in that the positions of the subjects are different as described below, but they all include a predetermined shooting range.

The reference image in FIG. 4B is taken at a time different from that of the reference image in FIG. 4A, and is out of position with the reference image in FIG. 4A due to camera shake or the like. The alignment unit 201 corrects such a positional deviation by alignment processing. To align the positions, first, the alignment unit 201 calculates a motion vector indicating global motion between the base image and the reference image. A motion vector calculation method includes, for example, a block matching method. Next, the alignment unit 201 calculates a geometric transformation coefficient A as shown in Equation (1), which is a coefficient for geometrically transforming the reference image, based on the calculated motion vector.

Alignment unit 201 generates the aligned reference image shown in FIG. The reference image is I (x coordinate, y coordinate), and the aligned reference image is I'(x' coordinate, y' coordinate).

Through such alignment processing, positions of still subjects (eg, buildings and trees) in the standard image and the reference image can be aligned as in the aligned reference image in FIG. 4C.

Returning to FIG. 3 , in S303, the moving object region detection unit 202 detects a moving object region based on the reference image and the aligned reference image (hereinafter also simply referred to as “reference image”), and calculates the synthetic moving object likelihood. do. In the present embodiment, the moving object area is represented by multi-valued data called moving object likelihood. Details of the moving object region detection unit 202 and details of the synthetic moving object likelihood calculation process will be described later.

In S304, the image synthesizer 203 synthesizes the reference image and the reference image according to the synthesis ratio w of the reference image determined based on the synthetic moving object likelihood, as shown in Equation (3), to obtain a synthesized image. to generate
P = w × Pbase + (1-w) × Pref (3)
Pbase represents the pixel value of the base image, Pref represents the pixel value of the reference image, w represents the synthesis ratio of the base image, and P represents the pixel value of the synthesized image.

Here, a method for determining the synthesis ratio w of the reference image will be described with reference to FIG. FIG. 5 is a diagram showing the relationship between the composite moving object likelihood and the composite ratio w of the reference image. In the example of FIG. 5, the larger the synthetic moving object likelihood, the larger the synthesis ratio w of the reference image. In the example of FIG. 5, for a moving object region with a high synthetic moving object likelihood, the synthesis ratio of the reference image is set to 100% (w=1), for example. Therefore, it is possible to substantially prohibit the synthesis processing of moving object regions and suppress the occurrence of multiple images. That is, when the composite moving object likelihood is high, the pixel values of the reference image are used as the pixel values of the composite image. On the other hand, for a still region with a low synthesized moving object likelihood, the synthesis ratio of the reference image is set to 50% (w=0.5), for example. Therefore, the standard image and the reference image can be evenly synthesized, and random noise can be reduced. As can be understood from FIG. 5, when the synthesized moving object likelihood is small, the difference between the pixel value synthesis ratio of the reference image and the pixel value synthesis ratio of the reference image becomes small.

Next, referring to FIG. 6, the configuration of moving object region detection section 202 will be described. The moving object region detection unit 202 includes moving object likelihood calculation units 600, 601, 602, edge level calculation units 610, 611, enlargement processing units 620, 621, 622, 623, synthesis ratio calculation units 630, 631, and a moving object likelihood synthesis unit. 640 , 641 and a reduction processing unit 650 . A moving object area detection unit 202 calculates a synthetic moving object likelihood based on the inter-frame difference absolute value between the standard image and the reference image.

Next, synthetic moving object likelihood calculation processing executed by the moving object region detection unit 202 will be described with reference to FIG. In S701, the moving object region detection unit 202 acquires a standard image and a reference image. Examples of the base image and the reference image are shown in FIGS. 4(A) and 4(C). FIG. 4A is a diagram showing a reference image, and FIG. 4C is a diagram showing a reference image (registered reference image). Since the base image and the reference image are shot at different times, the positions of the subject that moved during the shooting are different. 4A and 4C, the person 410 in the reference image of FIG. 4A moves to the position of the person 420 in FIG. The person 411 has moved to the position of the person 421 in the reference image of FIG. 4(C). The moving object area detection unit 202 detects such a moving area as a moving object area.

In S702, the reduction processing unit 650 generates a reduced standard image and reference image by performing reduction processing on the standard image and the reference image. Specifically, the reduction processing unit 650 reduces the number of vertical and horizontal pixels by a factor of 1/4 (hereinafter also referred to as a “1/4 size reference image”) and a reference image (hereinafter referred to as a “1/4 size standard image”). (also called a "size reference image"). Further, the reduction processing unit 650 performs reduction processing on the standard image and the reference image, thereby reducing the number of vertical and horizontal pixels to a standard image (hereinafter, also referred to as a “1/16 size standard image”). ) and a reference image (hereinafter also referred to as a “1/16 size reference image”) are also generated. Note that the standard image and the reference image before reduction are hereinafter also referred to as a "same size standard image" and a "same size reference image", respectively.

Details of the reduction process will be described with reference to FIGS. 11, 12A, and 12B. FIG. 11 is a conceptual diagram of a standard image and a reference image of the same size (before reduction). To simplify the explanation, various types of processing will be explained based on the base image 1100 and the reference image 1101 shown in FIG. In FIG. 11 , a dashed line 1102 indicates an example of pixel values of the reference image 1100 and a solid line 1103 indicates an example of pixel values of the reference image 1101 . A reduction processing unit 650 reduces the standard image 1100 and the reference image 1101 by smoothing processing and pixel thinning processing. Available reduction methods include, for example, bilinear and bicubic reduction methods.

FIG. 12A shows 1/16 size image data, and FIG. 12B shows 1/4 size image data. A dashed line 1200 in FIG. 12A indicates an example of pixel values of a 1/16 size reference image, and a solid line 1201 in FIG. 12A indicates an example of pixel values of a 1/16 size reference image. A dashed line 1202 in FIG. 12B indicates an example of pixel values of a 1/4 size reference image, and a solid line 1203 in FIG. 12B indicates an example of pixel values of a 1/4 size reference image. 12A and 12B, the horizontal coordinates are shown enlarged so as to correspond to the horizontal coordinates before reduction shown in FIG. This also applies to FIGS. 12(C) to 12(H), which will be described later.

A pixel value 1200 of the 1/4 size reference image shown in FIG. 12B is smoothed by reduction processing more than a pixel value 1102 of the reference image 1100 shown in FIG. Also, the pixel value 1200 of the 1/16 size reference image shown in FIG. 12(A) is further smoothed by reduction processing than the pixel value 1202 of the 1/4 size reference image shown in FIG. 12(B). .

Returning to FIG. 7, in S703, the moving object likelihood calculation units 600, 601, and 602 calculate, for each pixel, the inter-frame difference absolute value between the standard image and the reference image with the corresponding resolution (size). A moving object likelihood is calculated based on the absolute difference value. In other words, the moving object likelihood calculation units 600, 601, and 602 calculate the inter-frame difference absolute value for each position in a predetermined shooting range included in the standard image and the reference image, and calculate the moving object likelihood. Specifically, the moving object likelihood calculation unit 600 calculates the inter-frame difference absolute value between the same-size reference image and the same-size reference image for each pixel, and calculates the moving object likelihood as shown in FIG. A normal size moving object likelihood is calculated based on the curve. Further, the moving object likelihood calculation unit 601 calculates the inter-frame difference absolute value between the 1/4 size reference image and the 1/4 size reference image for each pixel, and calculates the moving object likelihood calculation curve as shown in FIG. A 1/4 size moving object likelihood is calculated based on. Further, the moving object likelihood calculation unit 602 calculates the inter-frame difference absolute value between the 1/16 size reference image and the 1/16 size reference image for each pixel, and calculates the moving object likelihood calculation curve as shown in FIG. 1/16 size moving object likelihood is calculated based on.

Examples of 1/16 size moving object likelihood and 1/4 size moving object likelihood are shown in FIGS. The 1/16 size moving object likelihood shown in FIG. 12(C) is the moving object likelihood calculated based on the 1/16 size image data shown in FIG. 12(A). The 1/4 size moving object likelihood shown in FIG. 12(D) is the moving object likelihood calculated based on the 1/4 size image data shown in FIG. 12(B).

The reason for changing the shape of the moving object likelihood calculation curve according to the resolution (size) of the image data as shown in FIGS. This is to reduce The inter-frame difference absolute value between the standard image and the reference image increases not only due to movement of the subject but also due to random noise. 1/4 size image data and 1/16 size image data have less random noise than normal size image data due to smoothing processing during reduction processing. Therefore, the thresholds TH1 and TH2 are set to smaller values for moving object likelihood calculation curves for lower resolution image data in which random noise is further reduced. On the other hand, the thresholds TH1 and TH2 are set to larger values for moving object likelihood calculation curves for higher resolution image data with more random noise.

Specifically, as shown in FIG. 8A, in the case of the same-size moving object likelihood calculation curve, the thresholds TH1 and TH2 are set to relatively large values so as not to erroneously detect random noise as motion. In addition, as shown in FIG. 8C, in the case of a 1/16 size moving object likelihood calculation curve, random noise is reduced. a relatively small value.

In this way, by changing the threshold value of the moving object likelihood calculation curve according to the resolution of the image data for calculating the absolute value of the inter-frame difference, erroneous detection of motion due to random noise can be reduced, and the motion of the subject can be accurately detected. It becomes possible to

Returning to FIG. 7, in S704, the edge degree calculation units 610 and 611 calculate the edge degree for each pixel based on the edge strength of the standard image and the reference image with corresponding resolutions (sizes). That is, the edge degree calculation units 610 and 611 calculate the edge degree for each position in a predetermined shooting range included in the standard image and the reference image.

A method for calculating the edge degree will be described in detail below. However, the edge degree calculation method of this embodiment is not limited to the specific examples shown below. Any calculation method can be employed as long as it can calculate a value that serves as an index indicating the edge state (existence of edge, intensity of contrast, etc.) for each position in the imaging range.

First, a method for calculating edge strength will be described. Edge strength is calculated by, for example, Sobel filtering. Sobel filter processing will be described. First, the vertical edge strength Sv is calculated by multiplying the nine pixel values above, below, to the left and right of the pixel of interest by the coefficients shown in Equation (4), respectively, and summing the multiplication results. Similarly, the horizontal edge strength Sh is calculated by multiplying the nine pixel values above, below, to the left, and to the right of the pixel of interest by the coefficients shown in Equation (5), respectively, and summing the multiplication results. Then, as shown in equation (6), the edge strength E is calculated from the square root of the sum of the squares of the vertical edge strength Sv and the horizontal edge strength Sh.

The edge degree calculation unit 610 calculates the edge strength for each pixel of the 1/4 size reference image and the 1/4 size reference image, and calculates the edge strength of the 1/4 size reference image and the 1/4 size reference image for each pixel. , whichever is greater. The edginess calculator 610 then calculates the edginess from the selected edge strength based on the edginess calculation curve shown in FIG. This gives a 1/4 size edge degree. Further, the edge degree calculation unit 611 calculates the edge strength for each pixel of the 1/16 size reference image and the 1/16 size reference image, and calculates the edge strength of the 1/16 size reference image and the 1/16 size reference image for each pixel. Select the larger edge strength of the reference image. Then, the edge degree calculation unit 611 calculates the edge degree from the selected edge strength based on the edge degree calculation curve shown in FIG. This gives an edge degree of 1/16 size.

Examples of 1/16 size edge degree and 1/4 size edge degree are shown in FIGS. 12(E) and 12(F), respectively. The 1/16 size edge degree shown in FIG. 12(E) is an edge degree calculated based on the 1/16 size image data shown in FIG. 12(A). The 1/4 size edge degree shown in FIG. 12(F) is the edge degree calculated based on the 1/4 size image data shown in FIG. 12(B).

Note that in the above description, the edge level calculation units 610 and 611 select the larger one of the edge level of the reference image and the edge level of the reference image, and calculate the edge level from the selected edge level based on the edge level calculation curve. calculated. However, the edge degree calculation method is not limited to this. For example, the edge level calculation units 610 and 611 may obtain the average value of the edge strengths of the standard image and the reference image for each pixel, and calculate the edge level from the average value of the edge strengths based on the edge level calculation curve. Alternatively, the edge degree calculation units 610 and 611 may calculate only the edge strength of the reference image for each pixel, and calculate the edge degree from the edge strength of the reference image based on the edge degree calculation curve.

Returning to FIG. 7, in S705, the moving object region detection unit 202 uses the enlargement processing units 620 and 622 and the synthesis ratio calculation units 630 and 631 for each pixel (that is, for each position in the predetermined shooting range), A 1/4 size synthesis ratio and a 1/4 size synthesis ratio are calculated based on the edge degree. Details of the processing of S705 will be described later.

In step S706, the moving object region detection unit 202 uses the enlargement processing units 621 and 623 and the moving object likelihood combining units 640 and 641 to perform pixel-by-pixel (that is, each position in the predetermined shooting range) sequentially from the smallest size. Synthesize moving object likelihoods. This synthesis is performed according to the 1/4 size synthesis ratio and the 1/4 size synthesis ratio.

Here, details of the processing of S705 and S706 will be described with reference to FIGS. 10 and 12. FIG. First, processing related to synthesis of the 1/16 size moving object likelihood and the 1/4 size moving object likelihood will be described. The enlargement processing unit 622 enlarges the 1/16 size edge degree to 4 times vertically and horizontally. By this enlargement, the 1/16 size edge degree has a resolution equivalent to 1/4 size. A synthesis ratio calculation unit 631 calculates a 1/4 size synthesis ratio based on the enlarged 1/16 size edge degree. The 1/4 size synthesis ratio has a resolution equivalent to 1/4 size, like the enlarged 1/16 size edge degree.

Here, FIG. 10B shows an example of a curve for calculating the 1/4 size synthesis ratio. The synthesis ratio calculator 631 calculates a 1/4 size synthesis ratio from the enlarged 1/16 size edge degree based on the 1/4 size synthesis ratio calculation curve. As shown in FIG. 10B, the curve for calculating the 1/4 size synthesis ratio is designed such that the higher the edge degree, the higher the synthesis ratio of the 1/4 size moving object likelihood. As a result, the synthesis ratio of the 1/16 size moving object likelihood is high in flat portions with a small edge degree, and the synthesis ratio of the 1/4 size moving object likelihood is high in an edge portion with a high edge degree. The 1/16 size image data has less random noise than the 1/4 size image data due to smoothing processing during reduction processing. Therefore, the 1/16 size moving object likelihood is less likely to erroneously detect random noise as motion than the 1/4 size moving object likelihood. Therefore, in a flat portion with a small edge degree, by increasing the synthesis ratio of 1/16 size moving object likelihoods that are less likely to be adversely affected by random noise, it is possible to generate synthetic moving object likelihoods that are less likely to be adversely affected by random noise. .

An example of the 1/4 size synthesis ratio is shown in FIG. 12(G). The 1/4 size composite ratio shown in FIG. 12(G) is a composite ratio calculated based on the 1/16 size edge degree shown in FIG. 12(E). The 1/4 size synthesis ratio indicates 0% in areas where the 1/16 size edge degree is small. Therefore, in a flat portion with a small edge degree, the synthesis ratio of the 1/16 size moving object likelihood is 100%, and the adverse effects of random noise can be suppressed. On the other hand, since the outline of the moving subject (the boundary between the moving subject and the background) has a high degree of edge, the combination ratio of the 1/4 size moving subject likelihood is high. Therefore, the 1/4 size composite moving object likelihood is a value that more accurately corresponds to the contour of the moving subject than the 1/16 size moving object likelihood that is simply expanded vertically and horizontally by four times.

The enlargement processing unit 623 enlarges the 1/16 size moving object likelihood four times vertically and horizontally. As a result of this enlargement, the 1/16 size moving object likelihood has a resolution corresponding to 1/4 size. Based on the 1/4 size synthesis ratio, the moving object likelihood synthesizing unit 641 synthesizes the 1/4 size moving object likelihood and the expanded 1/16 size moving object likelihood to generate a 1/4 size synthetic moving object likelihood. do. This synthesis is performed according to Equation (7) below.
MC4 = wm4 x M4 + (1 - wm4) x M16 (7)
Here, M4 represents the 1/4 size moving object likelihood, M16 represents the expanded 1/16 size moving object likelihood, wm4 represents the 1/4 size combining ratio, and MC4 represents the 1/4 size combining moving object likelihood. represents degrees.

FIG. 12(H) shows an example of the 1/4 size synthetic moving object likelihood. The 1/4 size synthetic moving object likelihood shown in FIG. 12(H) is based on the 1/4 size synthetic ratio shown in FIG. ) and the 1/4 size moving object likelihood shown in FIG. 12(D). The 1/4 size synthetic moving object likelihood shown in FIG. 12(H) has a sharper contrast near the edge than the 1/16 size moving object likelihood shown in FIG. 12(C). This is because the 1/16 size edge rate (see FIG. 12(E)) increases near the contour of the moving subject, resulting in a higher 1/4 size synthesis ratio (see FIG. 12(G)). .

Next, processing related to synthesis of the 1/4 size moving object likelihood and the 1/4 size synthetic moving object likelihood will be described. The enlargement processing unit 620 enlarges the 1/4 size edge degree vertically and horizontally four times. Due to this enlargement, the 1/4 size edge degree has a resolution equivalent to the full size. Composite ratio calculation section 630 calculates the same size composite ratio based on the enlarged 1/4 size edge degree. The same size compositing ratio has a resolution equivalent to the same size as the enlarged 1/4 size edge degree.

Here, FIG. 10A shows an example of a curve for calculating the same-size synthesis ratio. Composite ratio calculation section 630 calculates the same size composite ratio from the enlarged 1/4 size edge degree based on the calculation curve for the same size composite ratio. As shown in FIG. 10A, the curve for calculating the same-size synthesis ratio is designed so that the synthesis ratio of the same-size moving object likelihood becomes high when the edge degree is within a medium predetermined range. . There are two reasons for such a design.

The first reason is to suppress sensitive detection of a minute positional shift of a high-contrast edge and movement of a subject, which increases the degree of edge, as movement. In the same-size standard image and reference image, if the position of the high-contrast edge deviates even slightly, the moving object likelihood increases. If the synthetic moving object likelihood is high for such an area and the synthetic processing for such an area is not performed in S304 of FIG. 3, noise reduction is not performed. As a result, high-contrast edges that are slightly misaligned become noisy in the finally obtained synthesized image. Therefore, as shown in FIG. 10A, for the high-contrast edge region, instead of using the same-size moving object likelihood, the moving object likelihood of a lower resolution (size) is used to increase the synthesis ratio of the same-size moving object likelihood. The composite ratio calculation curve is designed.

The second reason is to detect motion of a subject with a low edge degree. Subjects with a low edge degree include, for example, subjects with low-contrast edges, subjects with thin lines, subjects with small dots, and the like. In the low-contrast edge region, even if the subject moves, the inter-frame difference absolute value is relatively small. Since the absolute value of the inter-frame difference is further reduced by performing the reduction processing, there is a possibility that the 1/4 size moving object likelihood and the 1/16 size moving object likelihood do not include such motion of the subject. Therefore, it is necessary to determine the motion of a subject with a low edge degree based on the same-size moving object likelihood. Therefore, as shown in FIG. 10A, the synthetic ratio of the same-size moving object likelihood is set high for a low-contrast edge region within a predetermined range (second range) with a medium edge degree. ing. On the other hand, for flat portions within a predetermined range (first range) where the edge degree is small, the composite ratio of the 1/4 size composite moving object likelihood based on the 1/4 size moving object likelihood and the 1/16 size moving object likelihood is set high. Similarly, for high-contrast edges within a predetermined range (third range) where the edge degree is large, the 1/4 size synthetic moving object likelihood based on the 1/4 size moving object likelihood and the 1/16 size moving object likelihood The composition ratio is set high. As a result, it is possible to generate the same-size synthetic moving object likelihood that is less likely to be adversely affected by random noise.

Note that the combining ratio calculator 630 may use the calculation curve shown in FIG. 10B instead of the calculation curve shown in FIG. 10A. Similarly, the combining ratio calculator 631 may use the calculation curve shown in FIG. 10(A) instead of the calculation curve shown in FIG. 10(B).

The enlargement processing unit 621 enlarges the 1/4 size synthetic moving object likelihood four times vertically and horizontally. By this enlargement, the 1/4 size moving object likelihood has a resolution corresponding to the same size. A moving object likelihood synthesizing unit 640 synthesizes the same size moving object likelihood and the enlarged 1/4 size synthetic moving object likelihood based on the synthetic ratio of the same size moving object likelihood, and obtains the same size synthetic moving object likelihood. to generate This synthesis is performed according to Equation (8) below.
MC1 = wm1 x M1 + (1 - wm1) x MC4' (8)
Here, M1 represents the same-size moving object likelihood, MC4′ represents the enlarged 1/4-size combined moving-object likelihood, wm1 represents the combining ratio of the same-size moving object likelihood, and MC1 represents the same-size moving object likelihood. represents the synthetic moving object likelihood.

The same-size synthetic moving object likelihood thus obtained is used to determine the synthetic ratio w of the reference image in S304 of FIG.

By the way, in the above description, the final synthetic moving object likelihood (same size synthetic moving object likelihood degree) shall be calculated. However, the combination of image data resolutions for calculating the composite moving object likelihood is not limited to this. For example, the magnification (reduction rate) is not limited to 1/4, and may be 1/2. Also, the types of resolutions are not limited to three types. For example, the synthetic moving object likelihood may be calculated based on image data with four or more types of resolutions, or image data with two different types of resolutions.

Further, in the above description, only the curve for calculating the same-size synthesis ratio is applied to a low-contrast edge region, which is a predetermined range with an intermediate degree of edge, as shown in FIG. 10A. It was assumed that it was designed to increase the composite ratio of likelihoods. However, the design of the synthesis ratio calculation curve is not limited to this. For example, when calculating the synthetic moving object likelihood, when synthesizing moving object likelihoods with a predetermined number of resolutions from higher resolution images, a synthesis ratio calculation curve as shown in FIG. 10A may be used. As a specific example, consider a case where the magnification (reduction rate) of the next smallest image data after the normal size is 1/2, and the composite moving object likelihood is calculated based on image data with five different resolutions. In this case, when calculating the 1/2 size synthetic moving object likelihood and the 1/2 size synthetic moving object likelihood, a synthesis ratio calculation curve as shown in FIG. 10A may be used. Also, when calculating the synthetic moving object likelihood of other magnifications (for example, 1/4 size and 1/8 size), a synthetic ratio calculation curve as shown in FIG. 10B may be used.

Further, the combining ratio calculation curve used by the combining ratio calculation units 630 and 631 may be changed according to the sensitivity information at the time of shooting. FIG. 13 shows an example of a curve for calculating a synthesis ratio when sensitivity is high. When the sensitivity is high, the degree of random noise in the standard image and the reference image increases (increases in amplitude), so the possibility of erroneously detecting random noise as motion increases. Therefore, the composite ratio of the moving object likelihood calculated based on lower resolution image data, which is less susceptible to random noise, is increased. FIG. 13B shows an example of a curve for calculating the 1/4 size composition ratio used by the composition ratio calculator 631 when the standard image and the reference image are shot at high sensitivity. Compared to FIG. 10B, the upper limit of the 1/4 size synthesis ratio is limited to 50%. As a result, even when the edge degree is large, the synthesis ratio of the 1/16 size moving object likelihood is 50%. Therefore, the composite ratio of the moving object likelihood calculated based on the lower resolution image data is increased. FIG. 13A shows an example of a curve for calculating the same-size composite ratio used by the composite ratio calculator 630 when the standard image and the reference image are captured at high sensitivity. Compared to FIG. 10A, the upper limit of the same size composition ratio is limited to 50%. As a result, even in the case of low-contrast edges, the synthesis ratio of the 1/4 size moving object likelihood is 50%. Therefore, the composite ratio of the moving object likelihood calculated based on the lower resolution image data is increased. Note that if the reference image and the reference image have different sensitivities when photographed, the combining ratio calculation units 630 and 631 may change the calculation curve based on the sensitivity when photographing at least one of the standard image and the reference image. .

Further, in the above description, the edge degree calculated with image data of lower resolution is expanded to calculate the composite ratio of the moving object likelihood, but the calculation method of the composite ratio is not limited to this. For example, in FIG. 6, the synthesis ratio calculation unit 631 calculates the edge degree obtained by expanding the 1/16 size edge degree calculated by the edge degree calculation unit 611 by four times vertically and horizontally, and the 1/4 size edge degree calculated by the edge degree calculation unit 610. A composite ratio may be calculated based on both degrees. In this case, for example, the synthesis ratio calculation unit 631 calculates the synthesis ratio based on the average of the edge degree obtained by enlarging the 1/16 size edge degree by four times vertically and horizontally and the 1/4 size edge degree, or based on whichever is larger. Calculate By calculating the synthesis ratio by such a method, the degree of edge near the edge increases, so the synthesis ratio of the 1/4 size moving object likelihood increases near the edge. Therefore, it is possible to calculate the synthetic moving object likelihood along the contour of the moving object more accurately. Further generalizing, the edge degree calculation unit 610 calculates the composite ratio calculation unit 630 can be used to calculate the edge measure. Further, the edge degree calculation unit 611 calculates the composite ratio calculation unit 631 based on at least one of the 1/4 size reference image, the 1/4 size reference image, the 1/16 size reference image, and the 1/16 size reference image. can calculate the edge measure used by .

Further, in the above description, the same edge degree calculation curve (see FIG. 9) is used when calculating the edge degree regardless of the type of image data from which the edge strength is calculated. However, the edge degree calculation curve may change according to various conditions. For example, similar to the moving object likelihood calculation curve, the edge degree calculation curve may be changed according to the resolution of the image data from which the edge strength was calculated. For example, as with the moving object likelihood calculation curve in FIG. 8, the lower the resolution of the edge degree calculation curve, in which the random noise is reduced, the smaller the threshold value of the edge degree calculation curve. On the other hand, the higher the resolution of the edge degree calculation curve in which the random noise is not reduced, the larger the threshold of the edge degree calculation curve. In this way, by changing the edge degree calculation curve according to the resolution of the image data from which the edge strength is calculated, it is possible to prevent random noise from being erroneously detected as an edge.

Also, in the above explanation, the same-size synthetic moving object likelihood is used when synthesizing the standard image and the reference image to reduce random noise, but the usage of the same-size synthetic moving object likelihood is It is not limited to this. For example, the same size synthetic moving object likelihood may be used for an HDR compositing function that expands the dynamic range by compositing multiple images taken with different exposures. In this case, by matching the brightness levels of the standard image and the reference image captured at different exposures and inputting them to the moving object region detection unit 202, the same size combined moving object likelihood can be calculated.

As described above, according to the first embodiment, the imaging device 100 generates reduced standard images and reference images from standard images and reference images. Then, the imaging device 100 generates a first moving object likelihood based on the difference between the base image and the reference image, and generates a second moving object likelihood based on the difference between the reduced base image and the reference image. Then, the imaging device 100 synthesizes the first moving object likelihood and the second moving object likelihood according to a synthesis ratio determined based on the edge degree of each position in the imaging range. This makes it possible to improve the detection accuracy of a moving object region based on the pixel value difference between images.

[Second embodiment]
1st Embodiment demonstrated the structure which synthesize|combines moving object likelihood according to the compositing ratio determined based on edge degree. On the other hand, in the second embodiment, a configuration will be described in which moving object likelihoods are synthesized according to a synthesis ratio determined based on the difference between the moving object likelihoods of the upper layer and the moving object likelihoods of the lower layer. The term "hierarchy" used herein means a specific magnification (reduction ratio) for the standard image and the reference image.

In the second embodiment, the basic configuration of an imaging device 100 is the same as in the first embodiment (see FIGS. 1 and 2). However, although the detailed configuration of the moving object region detection unit 202 in FIG. 2 was the configuration shown in FIG. 6 in the first embodiment, it is the configuration shown in FIG. 16 in the second embodiment. Also, the processing executed in S303 of FIG. 3 was the processing shown in FIG. 7 in the first embodiment, but the processing shown in FIG. 15 is executed in the second embodiment. In addition, the image synthesizing process in S304 of FIG. 3 is performed according to the synthetic moving object likelihood generated by the process shown in FIG. 7 in the first embodiment, but in the second embodiment it is generated by the process shown in FIG. It is performed according to the synthetic moving object likelihood that is calculated. Differences from the first embodiment will be mainly described below.

First, with reference to FIGS. 4 and 14, an outline of hierarchical moving body detection will be described. FIG. 14 is a schematic diagram showing the motion likelihood at three different resolutions and the final composite motion likelihood. In the present embodiment, the imaging apparatus 100 uses the sum of absolute differences (SAD) between the standard image and the reference image as the moving object likelihood based on the difference between the pixel values of the standard image and the reference image. and However, the imaging device 100 may use the same moving object likelihood as in the first embodiment. That is, the imaging apparatus 100 calculates the inter-frame difference absolute value between the base image and the reference image for each pixel, and calculates the moving object likelihood according to the moving object likelihood calculation curves shown in FIGS. You may

The imaging apparatus 100 calculates the SAD of each layer for the reference image in FIG. 4A and the aligned reference image in FIG. 4-hierarchy SAD” and “x1/1-hierarchy (same size hierarchy) SAD”). Then, the imaging device 100 generates a synthesized SAD (synthesized moving object likelihood) by synthesizing the SADs of each layer (see “Synthetic SAD” in FIG. 14). In FIG. 14, white areas represent areas where differences occur between frames and are detected as moving object areas. In addition, "x1/16 layer", "x1/4 layer", and "x1/1 layer (same size layer)" are respectively 1/16 size, 1/4 size, and 1/4 size described in the first embodiment. Corresponds to the same size.

In FIG. 14, positions 1401-1404 represent the same background region positions, and positions 1411-1414 represent the same person region positions. Looking at the positions 1401 to 1403, it can be seen that more differences are detected in the background area in higher resolution layers. This is because the difference between the images is detected even if the subject does not move because noise is added to the image data. Conversely, in a layer with a low resolution, resistance to noise is strong, and noise is less likely to be detected as differential information. However, as the resolution becomes coarser, a motion-occurring region is likely to be detected as being larger than it actually is. For example, at position 1411, a larger moving object area than at position 1413 is detected. At positions 1402 and 1412 corresponding to the intermediate resolution, the noise resistance is somewhat high, and the outline of the moving object area is relatively along the outline of the moving subject. That is, the intermediate resolution hierarchy has characteristics between the high resolution hierarchy and the low resolution hierarchy. By synthesizing the SADs of these different hierarchies, it is possible to obtain information as shown in "composite SAD" in FIG. As a result, it is possible to detect a moving object area along the contour of a moving subject with high noise resistance.

Next, synthetic moving object likelihood calculation processing executed by the moving object region detection unit 202 will be described with reference to FIGS. 15 and 16. FIG. In S1501, the moving object region detection unit 202 acquires a standard image and a reference image, as in S701 of FIG.

In S1502, the reduction processing unit 1650 generates a reduced base image and reference image by performing reduction processing on the base image and the reference image. Specifically, the reduction processing unit 1650 generates a 1/4 size reference image, a 1/4 size reference image, a 1/16 size reference image, and a 1/16 size reference image, as in S702 of FIG. .

In S1503, the SAD calculators 1611, 1612, and 1613 calculate the SAD between the base image and the reference image with corresponding resolutions (sizes). For example, the SAD calculators 1611, 1612, and 1613 calculate the sum of absolute differences (SAD) including pixels around the pixel of interest (for example, 3×3 pixels around the pixel of interest).

In S<b>1504 , the moving object region detection unit 202 uses the enlargement processing units 1621 and 1622 and the SAD calculation units 1612 and 1613 to calculate the SAD difference between layers. Specifically, the enlargement processing units 1621 and 1622 enlarge the SAD calculated by the SAD calculation units 1612 and 1613 vertically and horizontally four times. As a result of this enlargement, the SAD calculated by the SAD calculation unit 1612 has a resolution equivalent to the same size as the SAD calculation unit 1613, and the SAD calculated by the SAD calculation unit 1613 has a resolution equivalent to 1/4 size. Next, the synthesis ratio calculators 1631 and 1632 calculate the difference in SAD between hierarchies. Specifically, the synthesis ratio calculator 1631 calculates the difference between the SAD calculated by the SAD calculator 1611 and the SAD calculated by the SAD calculator 1612 and expanded by the expansion processor 1621 . The synthesis ratio calculator 1632 also calculates the difference between the SAD calculated by the SAD calculator 1612 and the SAD calculated by the SAD calculator 1613 and expanded by the expansion processor 1622 . Details of the calculation process in S1504 will be described later with reference to FIG.

In S1505, the synthesis ratio calculation units 1631 and 1632 calculate the synthesis ratio by performing threshold determination based on the SAD difference calculated in S1504.

In S1506, the moving object region detection unit 202 uses the enlargement processing units 1623 and 1624 and the SAD synthesis units 1641 and 1642 to synthesize SADs in order from the lower hierarchy. Specifically, the enlargement processing unit 1623 enlarges the SAD calculated by the SAD calculation unit 1613 vertically and horizontally four times. Due to this enlargement, the SAD calculated by the SAD calculator 1613 has a resolution corresponding to 1/4 size. Then, the SAD synthesizing unit 1642 synthesizes the SAD calculated by the SAD calculating unit 1612 and the SAD calculated by the SAD calculating unit 1613 and enlarged by the enlarging processing unit 1623 to generate a 1/4 size synthetic SAD. do. This synthesis is performed according to the synthesis ratio calculated by the synthesis ratio calculator 1632 . Next, the enlargement processing unit 1624 enlarges the 1/4 size synthesized SAD generated by the SAD synthesis unit 1642 four times vertically and horizontally. Due to this enlargement, the 1/4 size composite SAD has a resolution equivalent to the same size. Then, the SAD synthesizing unit 1641 synthesizes the SAD calculated by the SAD calculating unit 1611 and the ¼ size synthetic SAD enlarged by the enlargement processing unit 1624 to generate the same size synthetic SAD. This synthesis is performed according to the synthesis ratio calculated by the synthesis ratio calculator 1631 . The same-size composite SAD thus generated is used for the composite processing in S304 of FIG.

The SAD synthesis processing according to the above synthesis ratio can be executed, for example, according to the following equation (9).
Upper layer SAD×synthesis ratio+lower layer SAD×(1−synthesis ratio) (9)
Here, 0≦synthesis ratio≦1. In addition, the “upper layer SAD” means the SAD of the image data layer with the higher resolution among the two layers to be synthesized, and the “lower layer SAD” is the layer of the image data with the lower resolution among the two layers to be synthesized. of SAD. However, when SAD synthesis is performed between a lower hierarchy and a further lower hierarchy, the "lower hierarchy SAD" corresponds to the synthesized SAD of the lower hierarchy.

The combination ratio in equation (9) is determined so as to increase as the difference in SAD between layers increases. Therefore, according to the synthesizing process of Equation (9), the greater the difference in SAD between hierarchies, the greater the weight of the upper hierarchical SAD in the resulting synthesized SAD.

Next, operations of the combining ratio calculation units 1631 and 1632 will be described in detail with reference to FIGS. 17, 18A, and 18B. As for the pixel values of the standard image and the reference image of each layer, the pixel values shown in FIGS. 11 and 12(A) to (B) are used as in the first embodiment. FIG. 17(A) shows the SAD 1713 between the base image and the reference image in the x1/1 hierarchy. FIG. 17(B) shows the SAD 1723 between the base image and the reference image in the x1/4 hierarchy. FIG. 17(C) shows the SAD 1733 between the base image and the reference image in the x1/16 layer.

The synthesis ratio calculation unit 1632 calculates a synthesis ratio for synthesizing the x1/16 layer SAD and the x1/4 layer SAD. The synthesis ratio at this time is calculated based on the difference between the SAD of the x1/16 layer and the SAD of the x1/4 layer. As described above, the x1/16 layer SAD is enlarged vertically and horizontally by the enlargement processing unit 1622 four times. As shown in FIG. 18A, the synthesis ratio calculator 1632 calculates a difference 1811 between the SAD 1733 of the x1/16 layer and the SAD 1723 of the x1/4 layer. Strictly speaking, the difference 1811 is a difference absolute value, but for the sake of simplification, it is written as "difference". In a region where the SAD difference 1811 is large, there is a possibility that a moving object region wider than it actually is detected in the lower hierarchy (x1/16 hierarchy). Therefore, the synthesis ratio calculation unit 1632 performs control so that the synthesis ratio of the upper hierarchy (x1/4 hierarchy) is increased in the region where the SAD difference 1811 is large. This can be accomplished by using a composite ratio conversion curve 1821. FIG. Composite ratio conversion curve 1821 has a lower threshold 1822 and an upper threshold 1823 for the SAD difference, with values between 0% and 100% between these two thresholds. The synthesis ratio calculator 1632 converts the SAD difference 1811 into a synthesis ratio 1831 according to the synthesis ratio conversion curve 1821 .

Similarly, the synthesis ratio calculation unit 1631 calculates a synthesis ratio for synthesizing the x1/4 layer SAD and the x1/1 layer SAD. The synthesis ratio at this time is calculated based on the difference between the SAD of the x1/4 layer and the SAD of the x1/1 layer. As described above, the x1/4 layer SAD is enlarged vertically and horizontally four times by the enlargement processing unit 1621 . As shown in FIG. 18B, the synthesis ratio calculator 1631 calculates a difference 1851 between the SAD 1723 of the x1/4 layer and the SAD 1713 of the x1/1 layer. Strictly speaking, the difference 1851 is a difference absolute value, but for the sake of simplification, it is written as "difference". In a region where the SAD difference 1851 is large, there is a possibility that a moving object region wider than it actually is detected in the lower hierarchy (x1/4 hierarchy). Therefore, the synthesis ratio calculation unit 1631 performs control so that the synthesis ratio of the upper hierarchy (x1/1 hierarchy) is increased in the region where the SAD difference 1851 is large. This can be accomplished by using a composite ratio conversion curve 1861. FIG. Composite ratio conversion curve 1861 has a lower threshold 1862 and an upper threshold 1863 for the SAD difference, with values between 0% and 100% between these two thresholds. The synthesis ratio calculator 1631 converts the SAD difference 1851 into a synthesis ratio 1871 according to the synthesis ratio conversion curve 1861 .

As will be described later with reference to FIG. 21, the SAD differences 1811 and 1851 are calculated including the SADs of peripheral pixels (pixels at peripheral positions) of the pixel of interest. Therefore, in FIG. 18A, the SAD difference 1811 at a particular horizontal coordinate does not necessarily match the difference between the x1/4 layer SAD 1723 and the x1/16 layer SAD 1733 at that coordinate. Similarly, in FIG. 18B, the SAD difference 1851 at a particular horizontal coordinate does not necessarily match the difference between the x1/1 layer SAD 1713 and the x1/4 layer SAD 1723 at that coordinate.

Using the SAD calculated based on the low-resolution standard image and the reference image detects the area outside the outline of the moving object as a moving object area. As described above, by increasing the composition ratio of the SAD of the upper hierarchy for the region where the difference between the SAD of the upper hierarchy and the SAD of the lower hierarchy is large, the moving object region fitted to the contour of the moving subject can be detected with high accuracy. It becomes possible.

The synthesis ratio calculators 1631 and 1632 may use synthesis ratio conversion curves of the same shape, or may use synthesis ratio conversion curves of different shapes as shown in FIGS. 18A and 18B. Looking at the SAD of each resolution, the area with large SAD (moving object area) tends to spread around the edge of the moving object in the layer with lower resolution, and the area with large SAD (moving object area) tends to spread around the edge of the moving object in the layer with higher resolution. It tends to fit the edge without spreading around. In view of this tendency, the composition ratio conversion curve used by the composition ratio calculation unit 1632 is set so that the composition ratio of the upper hierarchy is higher than the composition ratio conversion curve used by the composition ratio calculation unit 1631 .

A method of setting the synthesis ratio conversion curve will be described with reference to FIG. FIG. 19 shows three patterns of synthetic ratio conversion curves. Reference numeral 1901 denotes a synthesis ratio conversion curve at the time of initial setting; reference numeral 1902, a synthesis ratio conversion curve configured to increase the synthesis ratio of the upper hierarchy; 2 shows a composite ratio transformation curve configured in FIG. When it is desired to increase the composition ratio of the x1/1 layer, the composition ratio calculation unit 1631 changes the shape of the composition ratio conversion curve from the shape indicated by reference numeral 1901 to the shape indicated by reference numeral 1902 . To achieve this, for example, the synthesis ratio calculation unit 1631 sets the lower limit threshold 1822 to a value lower than the lower limit threshold 1862 and sets the upper limit threshold 1823 to a value lower than the upper limit threshold 1863 . As a result, even when a small SAD difference occurs, the synthesis ratio calculation unit 1632 can perform control so that the weight of the upper layer is higher than that of the synthesis ratio calculation unit 1631 . This makes it possible to use the optimum synthesis ratio conversion curve for each resolution.

Also, the shape of the synthesis ratio conversion curve may be changed according to the ISO sensitivity. When the sensitivity is low, noise decreases, so the synthesis ratio calculation units 1631 and 1632 set the lower limit threshold and the upper limit threshold to smaller values. Conversely, when the sensitivity is high, noise increases, so the combination ratio calculation units 1631 and 1632 set the lower limit threshold and the upper limit threshold to larger values.

A specific example of processing for changing the synthesis ratio conversion curve according to the ISO sensitivity will be described with reference to FIG. FIG. 20(A) shows parameters that define the initial setting of the synthesis ratio conversion curve. FIG. 20B shows items that require threshold adjustment when this parameter is changed. Lowering the shooting sensitivity from ISO 800 to ISO 400 improves resistance to noise. Therefore, as indicated by reference numeral 2001, the synthesis ratio calculation units 1631 and 1632 set the lower limit threshold value and the upper threshold value of the synthesis ratio conversion curve to small values (the shape indicated by reference numeral 1901 in FIG. 19 is changed to the shape indicated by reference numeral 1902). ). Conversely, if the shooting sensitivity is increased from ISO800 to ISO1600, the resistance to noise will decrease. Therefore, as indicated by reference numeral 2002, the synthesis ratio calculation units 1631 and 1632 set large values for the lower limit threshold value and the upper threshold value of the synthesis ratio conversion curve (the shape indicated by reference numeral 1901 in FIG. 19 is changed to the shape indicated by reference numeral 1903). ). Note that if the reference image and the reference image have different sensitivities when photographed, the composition ratio calculation units 1631 and 1632 change the composition ratio conversion curve based on the sensitivity when photographing at least one of the standard image and the reference image. good too.

Next, details of the SAD difference calculation processing in S1504 of FIG. 15 will be described with reference to FIG. FIGS. 21(A) and 21(B) are conceptual diagrams of a base image and a reference image including moving object regions. As described above, the synthesis ratio calculators 1631 and 1632 calculate the difference between the upper layer SAD and the enlarged lower layer SAD. At this time, the synthesis ratio calculation units 1631 and 1632 calculate the SAD difference by considering not only the SAD of the pixel of interest but also the SAD of the surrounding pixels of the pixel of interest. In FIG. 21, to simplify the explanation, the SAD difference is represented by a binary value (1 in areas where the difference is large and 0 in areas where the difference is small). Although the difference between the x1/4 layer SAD and the x1/16 layer SAD will be described below, the same applies to the difference between the x1/1 layer SAD and the x1/4 layer SAD.

Since the SAD of the higher hierarchy (x1/4 hierarchy) has a relatively small detection range for moving object regions, the texture of the inside of a person tends to be small. As a result, the SAD value tends to be small, so detection failure of the moving object region tends to occur. For example, SAD 2121 shown in FIGS. 21C and 21D is calculated as the difference between the areas 2111 and 2113 . On the contrary, in the lower hierarchy (x1/16 hierarchy), the detection range of the moving object area is relatively large, so the SAD tends to increase. For example, SAD 2122 shown in FIGS. 21C and 21D is calculated as the difference between the areas 2101 and 2103 . After that, when calculating the difference between the SAD of the x1/16th layer (after enlargement) and the SAD of the x1/4th layer for the pixel of interest, a difference 2131 shown in FIG. 21C is obtained. According to the difference 2131, the difference in SAD between hierarchies is large. Therefore, if the combining ratio is determined based on the difference 2131, the weight of the SAD in the x1/4 hierarchy is increased. As a result, there is a possibility that detection omission of the moving object region may occur in a portion of the moving object region where the inter-frame difference is small, such as inside a moving object.

Therefore, as shown in FIG. 21D, the synthesis ratio calculation unit 1632 calculates the difference between the SAD of the x1/4 layer and the SAD of the x1/16 layer, including the SAD of the pixels surrounding the pixel of interest. Specifically, the synthesis ratio calculation unit 1632 calculates the average value of the SADs of the pixel of interest and the surrounding pixels for each of the SAD of the x1/4 layer and the SAD of the x1/16 layer, and calculates the absolute value of the difference between the average values. Calculate As a result, a difference 2132 shown in FIG. 21(D) is obtained. A difference 2132 indicates that there is no difference in SAD between tiers. Therefore, if the combining ratio is determined based on the difference 2132, the weight of the SAD in the x1/16 hierarchy is increased. As a result, detection omission of the moving object region is suppressed.

Note that the synthesis ratio calculation unit 1632 performs a process of applying a median filter or a low-pass filter to the SAD of each layer instead of referring to the SADs of the pixels surrounding the pixel of interest, and then calculates the difference in SAD between layers. may

As described above, according to the second embodiment, the imaging device 100 generates reduced standard images and reference images from the standard images and reference images. Then, the imaging device 100 generates a first moving object likelihood based on the difference between the base image and the reference image, and generates a second moving object likelihood based on the difference between the reduced base image and the reference image. Then, the imaging device 100 synthesizes the first moving object likelihood and the second moving object likelihood according to a synthesis ratio determined based on the difference between the first moving object likelihood and the second moving object likelihood. This makes it possible to improve the detection accuracy of a moving object region based on the pixel value difference between images.

[Other embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Reference Signs List 100: Imaging device 202: Moving object region detection unit 600, 601, 602: Moving object likelihood calculation unit 610, 611: Edge level calculation unit 630, 631: Synthesis ratio calculation unit 640, 641: Moving object likelihood synthesis section 650 ... reduction processing section

Claims (17)

reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; reduction means for generating a second reference image and a second reference image by
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; a first obtaining means for obtaining a second moving object likelihood based on a difference between a value and a pixel value of the second reference image;
a second step of obtaining an edge degree at each position of the imaging range based on at least one of the first reference image, the first reference image, the second reference image, and the second reference image; a means for obtaining the
determining means for determining a synthesis ratio of the first moving object likelihood and the second moving object likelihood based on the edge degree for each position in the imaging range;
synthesizing means for generating a synthesized moving object likelihood by synthesizing the first moving object likelihood and the second moving object likelihood according to the determined synthesis ratio for each position in the imaging range;
An image processing device comprising: When the edge degree is in a second range that is larger than the first range and smaller than the third range, the determining means determines that when the edge degree is in the first range or the third range, the 2. The image processing apparatus according to claim 1, wherein a synthesis ratio of said first moving object likelihood and said second moving object likelihood is determined such that a synthesis ratio of said first moving object likelihood is large. . wherein the determination means determines the synthesis ratio of the first moving object likelihood and the second moving object likelihood such that the synthesis ratio of the first moving object likelihood increases as the edge degree increases. 2. The image processing apparatus according to claim 1. The determining means determines the first moving object likelihood and the second moving object likelihood based on the degree of random noise in at least one of the first reference image and the first reference image in addition to the edge degree. 4. The image processing apparatus according to any one of claims 1 to 3, wherein a combination ratio of degrees is determined. The determining means determines the sensitivity of at least one of the first standard image and the first reference image at the time of shooting as the degree of random noise in at least one of the first standard image and the first reference image. 5. The image processing apparatus according to claim 4, characterized in that: reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; to generate a second standard image and a second reference image, and reduce the first standard image and the first reference image by a second magnification smaller than the first magnification, thereby reducing the third a reduction means for generating a reference image and a third reference image of
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; obtaining a second moving object likelihood based on the difference between the value and the pixel value of the second reference image, and based on the difference between the pixel value of the third reference image and the pixel value of the third reference image a first acquisition means for acquiring a third moving object likelihood;
obtaining a first edge degree for each position in the imaging range based on at least one of the first reference image, the first reference image, the second reference image, and the second reference image; and a second edge degree at each position in the photographing range based on at least one of the second reference image, the second reference image, the third reference image, and the third reference image a second acquisition means for acquiring
determining means for determining a first synthesis ratio of the second moving object likelihood and the third moving object likelihood based on the second edge degree for each position in the imaging range;
combining the second moving object likelihood and the third moving object likelihood according to the determined first combination ratio for each position in the imaging range to generate a first combined moving object likelihood; means and
with
The determining means determines a second synthesis ratio of the first moving object likelihood and the first synthetic moving object likelihood based on the first edge degree for each position in the imaging range,
The synthesizing means synthesizes the first moving object likelihood and the first synthetic moving object likelihood according to the determined second synthesis ratio for each position in the imaging range, thereby producing a second synthetic moving object. An image processing device characterized by generating a likelihood. reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; reduction means for generating a second reference image and a second reference image by
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; acquisition means for acquiring a second moving object likelihood based on the difference between the value and the pixel value of the second reference image;
Determination of determining a composite ratio of the first moving object likelihood and the second moving object likelihood based on a difference between the first moving object likelihood and the second moving object likelihood for each position in the imaging range. means and
synthesizing means for generating a synthesized moving object likelihood by synthesizing the first moving object likelihood and the second moving object likelihood according to the determined synthesis ratio for each position in the imaging range;
An image processing device comprising: The determining means determines the first moving object likelihood and the second moving object likelihood so that the larger the difference absolute value between the first moving object likelihood and the second moving object likelihood, the larger the synthetic ratio of the first moving object likelihood. 8. The image processing apparatus according to claim 7, wherein a combining ratio of the second moving object likelihood and the second moving object likelihood is determined. The determining means, based on the difference between the first moving object likelihood and the second moving object likelihood and the sensitivity at the time of shooting of at least one of the first reference image and the first reference image, 9. The image processing apparatus according to claim 7, wherein a composite ratio of said first moving object likelihood and said second moving object likelihood is determined. The determination means determines the first moving object likelihood and the 10. The image processing apparatus according to any one of claims 7 to 9, wherein a difference between second moving object likelihoods is calculated. reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; to generate a second standard image and a second reference image, and reduce the first standard image and the first reference image by a second magnification smaller than the first magnification, thereby reducing the third a reduction means for generating a reference image and a third reference image of
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; obtaining a second moving object likelihood based on the difference between the value and the pixel value of the second reference image, and based on the difference between the pixel value of the third reference image and the pixel value of the third reference image Acquisition means for acquiring a third moving object likelihood;
A first synthetic ratio of the second moving object likelihood and the third moving object likelihood is calculated based on a difference between the second moving object likelihood and the third moving object likelihood for each position in the imaging range. a determining means for determining;
combining the second moving object likelihood and the third moving object likelihood according to the determined first combination ratio for each position in the imaging range to generate a first combined moving object likelihood; means and
with
The determining means determines a second synthetic ratio of the first moving object likelihood and the first combined moving object likelihood based on a difference between the first moving object likelihood and the second moving object likelihood. ,
The synthesizing means synthesizes the first moving object likelihood and the first synthetic moving object likelihood according to the determined second synthesis ratio for each position in the imaging range, thereby producing a second synthetic moving object. An image processing device characterized by generating a likelihood. An image processing device according to any one of claims 1 to 11;
imaging means for generating the first standard image and the first reference image;
An imaging device comprising: An image processing method executed by an image processing device,
reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; a reduction step to thereby generate a second reference image and a second reference image;
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; a first obtaining step of obtaining a second moving object likelihood based on a difference between a value and a pixel value of the second reference image;
a second step of obtaining an edge degree at each position of the imaging range based on at least one of the first reference image, the first reference image, the second reference image, and the second reference image; a step of obtaining
a determination step of determining a synthesis ratio of the first moving object likelihood and the second moving object likelihood based on the edge degree for each position in the imaging range;
a synthesizing step of synthesizing the first moving object likelihood and the second moving object likelihood according to the determined synthesis ratio for each position in the imaging range to generate a synthesized moving object likelihood;
An image processing method comprising: An image processing method executed by an image processing device,
reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; to generate a second standard image and a second reference image, and reduce the first standard image and the first reference image by a second magnification smaller than the first magnification, thereby reducing the third a reduction step of generating a reference image and a third reference image of
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; obtaining a second moving object likelihood based on the difference between the value and the pixel value of the second reference image, and based on the difference between the pixel value of the third reference image and the pixel value of the third reference image A first acquisition step of acquiring a third moving object likelihood;
obtaining a first edge degree for each position in the imaging range based on at least one of the first reference image, the first reference image, the second reference image, and the second reference image; and a second edge degree at each position in the photographing range based on at least one of the second reference image, the second reference image, the third reference image, and the third reference image a second obtaining step of obtaining
a determination step of determining a first synthesis ratio of the second moving object likelihood and the third moving object likelihood based on the second edge degree for each position in the imaging range;
combining the second moving object likelihood and the third moving object likelihood according to the determined first combination ratio for each position in the imaging range to generate a first combined moving object likelihood; process and
with
In the determination step, for each position in the imaging range, a second synthetic ratio of the first moving object likelihood and the first combined moving object likelihood is determined based on the first edge degree;
In the synthesizing step, the first moving object likelihood and the first synthesized moving object likelihood are synthesized in accordance with the determined second synthesis ratio for each position in the imaging range to obtain a second synthetic moving object. An image processing method characterized by generating a likelihood. An image processing method executed by an image processing device,
reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; a reduction step to thereby generate a second reference image and a second reference image;
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; an obtaining step of obtaining a second moving object likelihood based on the difference between the value and the pixel value of the second reference image;
Determination of determining a composite ratio of the first moving object likelihood and the second moving object likelihood based on a difference between the first moving object likelihood and the second moving object likelihood for each position in the imaging range. process and
a synthesizing step of synthesizing the first moving object likelihood and the second moving object likelihood according to the determined synthesis ratio for each position in the imaging range to generate a synthesized moving object likelihood;
An image processing method comprising: An image processing method executed by an image processing device,
reducing each of a first reference image including a predetermined imaging range and a first reference image including the predetermined imaging range captured at a time different from that of the first reference image by a first magnification; to generate a second standard image and a second reference image, and reduce the first standard image and the first reference image by a second magnification smaller than the first magnification, thereby reducing the third a reduction step of generating a reference image and a third reference image of
obtaining a first moving object likelihood based on a difference between a pixel value of the first reference image and a pixel value of the first reference image for each position in the imaging range; obtaining a second moving object likelihood based on the difference between the value and the pixel value of the second reference image, and based on the difference between the pixel value of the third reference image and the pixel value of the third reference image an acquisition step of acquiring a third moving object likelihood;
A first synthetic ratio of the second moving object likelihood and the third moving object likelihood is calculated based on a difference between the second moving object likelihood and the third moving object likelihood for each position in the imaging range. a decision step of deciding;
combining the second moving object likelihood and the third moving object likelihood according to the determined first combination ratio for each position in the imaging range to generate a first combined moving object likelihood; process and
with
In the determining step, a second synthetic ratio of the first moving object likelihood and the first combined moving object likelihood is determined based on a difference between the first moving object likelihood and the second moving object likelihood. ,
In the synthesizing step, the first moving object likelihood and the first synthesized moving object likelihood are synthesized in accordance with the determined second synthesis ratio for each position in the imaging range to obtain a second synthetic moving object. An image processing method characterized by generating a likelihood. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 11.

Images