JP7025237B2 - Image processing equipment and its control method and program - Google Patents

Description

The present invention relates to an image processing device for calculating the reliability of a motion vector, a control method thereof, and a program.

In order to detect the movement amount of the pixel of interest between multiple input images as a motion vector, a technology is known that generates multiple hierarchical images with different resolutions from the input images and detects the motion vector using the generated hierarchical images. Has been done. By using the value of the motion vector detected in the low-resolution image among the plurality of hierarchical images as the initial value in the next layer, the amount of calculation for detecting the motion vector can be reduced.

In addition, as a technique using a plurality of hierarchical images, edges are detected in a plurality of hierarchical images, and the edge detection results for each layer are weighted and synthesized according to the control information of the image pickup device and the motion vector obtained from the images. A technique for generating a value is known (Patent Document 1). By weighting and synthesizing the edge detection results for each layer in this way, it is possible to improve the accuracy of edge detection regardless of the shooting state of the camera such as camera shake and motion shake.

Japanese Unexamined Patent Publication No. 2013-165352

By the way, the detection accuracy of the detected motion vector may generally decrease depending on the shooting conditions and the like, and when the detected motion vector is used for processing later, the accuracy of the processing is the accuracy of the motion vector. Will depend on. On the other hand, when detecting a motion vector, if a reliability for determining the correctness of the detected motion vector is generated, it is possible to process the motion vector in consideration of the reliability in the subsequent processing. Become. Further, although the above-mentioned Patent Document 1 discloses a technique for improving the accuracy of edge detection regardless of the shooting state of the camera, it does not consider a technique for accurately obtaining the reliability for detecting a motion vector.

The present invention has been made in view of the above problems, and an object thereof is to realize a technique capable of accurately obtaining the reliability of a motion vector.

In order to solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, it is determined to determine the weight for each layer of the plurality of layered images according to the image acquisition means for acquiring a plurality of layered images having different resolutions for the input image and the photographing parameters for photographing the input image. Reliability calculation for calculating the reliability of each of the plurality of motion vectors for a plurality of motion vectors obtained between the means and the images of the same layer of the plurality of layered images acquired from input images having different frames. It has a means and a reliability synthesis means for synthesizing the reliability for each of the plurality of motion vectors for each layer, which is calculated by the reliability calculation means by using the weight for each layer. The degree calculation means is characterized in that it does not calculate the reliability for each of the plurality of motion vectors for a layer having a weight lower than a predetermined threshold among the weights for each layer .

According to the present invention, it is possible to accurately obtain the reliability of the motion vector.

A block diagram showing a functional configuration example of a digital camera as an example of the image processing apparatus according to the first embodiment. A flowchart showing a series of operations of the reliability information generation process according to the first embodiment. The figure explaining the template matching which concerns on Embodiment 1. The figure which shows typically the reliability map which concerns on Embodiment 1. A flowchart showing a series of operations of the reliability determination process according to the first embodiment. The figure explaining the structural example of the weighting information which concerns on Embodiment 1. Block diagram showing a functional configuration example of the digital camera according to the second embodiment A flowchart showing a series of operations of the reliability information generation process according to the second embodiment. A flowchart showing a series of operations of the reliability determination process according to the second embodiment. A flowchart showing a series of operations of the reliability synthesis process according to the second embodiment.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, as an example of the image processing device, an example of using a digital camera capable of calculating the motion vector of the captured image signal will be described. However, this embodiment is applicable not only to digital cameras but also to other devices capable of calculating motion vectors. These devices include, for example, mobile phones including smartphones, game consoles, tablet terminals, clock-type and eyeglass-type information terminals, medical devices, surveillance systems, in-vehicle systems, robots that acquire and process images, and the like. You can do it.

(Digital camera configuration)
FIG. 1 is a block diagram showing a functional configuration example of a digital camera 100 as an example of the image processing device of the present embodiment. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. You may. It may also be realized by a combination of software and hardware. Therefore, in the following description, the same hardware can be realized as the main body even if different functional blocks are described as the main body of operation.

The photographing optical system 101 includes a lens group including, for example, a zoom lens, a focus lens, and the like, and forms a subject image on the image pickup element 102. The image pickup device 102 has a configuration in which a plurality of pixels having a photoelectric conversion element are arranged two-dimensionally. The image pickup element 102 photoelectrically converts the subject optical image imaged by the photographing optical system 101 with each pixel, and outputs an analog image signal. The image pickup device 102 may be an image pickup device such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The development processing unit 103 forms an image signal from an analog signal output from the image pickup device 102. The development processing unit 103 includes an A / D conversion unit (not shown), an auto gain control unit (AGC), and an auto white balance unit. The A / D conversion unit converts an analog image signal into analog and digital, and outputs a digital signal in pixel units. The auto gain control unit and the auto white balance unit further perform gain correction and white balance processing on the digital image signal, and output developed image data (also simply referred to as an image). The image pickup device 102 and the development processing unit 103 constitute an image pickup system for acquiring an image.

The memory 104 includes a volatile memory such as SDRAM. The memory 104 temporarily stores various data such as moving image data, still image data, and audio data, or shooting parameters of the digital camera 100.

The multi-resolution image generation unit 105 performs resizing processing on the image input from the development processing unit 103. The motion vector detection unit 106 detects a motion vector between an image input from the memory 104 and an image input from the multi-resolution image generation unit 104. The reliability calculation unit 107 calculates the reliability for the detected motion vector by using the image used for detecting the motion vector. The shooting parameter acquisition unit 108 acquires the shooting parameters used by the digital camera 100 at the time of image acquisition. The reliability determination unit 109 weights the reliability calculated by the reliability calculation unit 107 based on the values of the imaging parameters acquired by the imaging parameter acquisition unit 108. The reliability synthesis unit 110 synthesizes the reliability calculated at a plurality of resolutions based on the weighting determined by the reliability determination unit 109.

The control unit 111 includes, for example, a CPU (or MPU), expands and executes a program stored in the recording unit 112 in the memory 104, controls each block of the digital camera 100, and transfers data between the blocks. To control. Further, the control unit 111 controls each block of the digital camera 100 in response to an operation signal from the operation unit 113 that receives an operation from the user. Further, the control unit 111 replaces any or all of the above-mentioned multi-resolution image generation unit 105, motion vector detection unit 106, reliability calculation unit 107, reliability determination unit 109, and reliability synthesis unit 110, which will be described later. May be done.

The recording unit 112 includes a recording medium for recording a captured image, and is composed of, for example, a semiconductor memory, a magnetic disk, or the like. The recording unit 112 stores a program for the CPU in the control unit 111, a set value for operation, and the like.

The operation unit 113 includes switches for inputting various operations related to shooting, such as a power button, a still image recording button, and a button for instructing to start and stop moving image recording. Further, the operation unit 113 has a menu display button, a decision button, other cursor keys, a pointing device, a touch panel, and the like, and when a user operates these keys or buttons, an operation signal is transmitted to the control unit 111. The touch panel detects, for example, contact, proximity, or pressure and accepts an input operation from the user.

The display unit 114 includes, for example, a display device such as a liquid crystal display, an organic EL display, or electronic paper, and displays a photographed image or an image read from the recording unit 112. In addition, a menu screen for the user to operate the digital camera 100 is displayed.

(A series of operations related to the reliability information generation process)
Next, with reference to FIG. 2, a series of operations related to the reliability information generation process according to the present embodiment will be described. It should be noted that this process is started when, for example, a button or the like included in the operation unit 113 detects a start instruction for moving image shooting. Further, this processing is realized by the control unit 111 expanding and executing the program recorded in the recording unit 112 in the working area of the memory 104, and controlling each unit such as the multi-resolution image generation unit 105.

In step S201, the development processing unit 103 acquires the input images sequentially input in the moving image in response to the instruction of the control unit 111. Specifically, first, the image sensor 102 images a subject image formed by the photographing optical system 101, and sequentially outputs an analog signal according to the subject brightness. That is, each input image is sequentially output at a predetermined frame rate. After that, the development processing unit 103 converts the analog signal into a digital signal by the above-mentioned A / D conversion unit, and performs level correction and white level correction on the digital signal by the AGC and AWB in the development processing unit 103. The development processing unit 103 outputs an image that has undergone level correction or the like.

In step S202, the multi-resolution image generation unit 105 resizes the image input from the development processing unit 103 in response to an instruction from the control unit 111, and a plurality of hierarchical images (multi-resolution images) having different resolutions in stages. To generate. As an example of a multi-resolution image, for example, the input image is resized at a reduction ratio of "1/2 to the nth power". Here, n is an integer of 1 or more, and a reduced image having a different reduction ratio is generated stepwise, such as 1/2 times, 1/4 times, 1/8 times, etc., based on the resolution of the input image. I will do it. The reduction ratio mentioned here is an example, and other methods may be used as long as it is possible to generate reduced images having different resolutions step by step. However, it should be noted that when the reduction ratio is small (for example, the difference from the input image is reduced to a small size), there is no difference in the motion vector and the reliability property in the subsequent processing. Further, if the number of images to be generated is too large, the processing time of the subsequent processing increases, and the minimum resolution image may become too small and the texture information may be lost. Therefore, it is advisable to use effective values obtained by experiments or the like in advance for the reduction ratio and the number of sheets. Further, for example, when performing a hierarchical search, the resolution and the number of layers may be determined depending on how much movement is desired to be detected.

Further, the resizing method used for generating the multi-resolution image is not particularly limited, and a simple thinning process or a resizing method by interpolation such as bilinear or bicubic may be used. Note that the resizing process may cause wrapping distortion in the image as a result of resizing, so it is advisable to perform pre-processing such as applying a low-pass filter before the resizing process. As an example of generating a multi-resolution image, a case where the input image is reduced is described as an example, but the same applies to the case where the enlargement processing is performed.

The multi-resolution image generation unit 105 stores the generated multi-resolution image in the memory 104. The multi-resolution image generation unit 105 sequentially updates the image stored in the memory 104 each time a new frame process is performed.

In step S203, the motion vector detection unit 106 sets the image of the layer to be processed among the multi-resolution images for the current frame and the image of the same layer of the multi-resolution images of different frames in response to the instruction of the control unit 111. Detect motion vectors between. In this embodiment, as a process of detecting a motion vector, for example, a method using template matching will be described as an example.

FIG. 3 shows an outline of the template matching process according to the present embodiment. FIG. 3A represents an original image, FIG. 3B represents a reference image, and these images are multi-resolution images stored in the memory 104. Here, the resolutions of the original image and the reference image are the same. The motion vector detection unit 106 arranges the template block 301 at an arbitrary position in the original image, and calculates the correlation value between the template block 301 and each region of the reference image. At this time, since the amount of calculation is enormous if the correlation value is calculated for the entire area of the reference image, the search range 302, which is a rectangular area for calculating the correlation value on the reference image, is actually set. do. At this time, there is no particular limitation on the position and size of the search range 302, but if the area corresponding to the movement destination of the template block 301 is not included in the search range 302, the correct motion vector is detected. I can't.

In the present embodiment, the sum of absolute differences (Su of Absolute Difference, hereinafter abbreviated as SAD) is used as an example of the method of calculating the correlation value. The motion vector detection unit 106 calculates SAD according to, for example, Equation 1.

In Equation 1, f (i, j) represents the luminance value at the coordinates (i, j) in the template block 301, and g (i, j) is the target of correlation value calculation in the search range 302. Represents each luminance value in the block 303. Then, in SAD, the correlation value S_SAD can be obtained by calculating the absolute value of the difference for each luminance value f (i, j) and g (i, j) in both blocks and calculating the sum. The smaller the value of the correlation value S_SAD, the smaller the difference in the luminance value between the two blocks, that is, the template block 301 and the texture in the correlation value target block 303 are similar.

In the present embodiment, the case where SAD is used as an example of the correlation value will be described as an example, but other correlation values such as the sum of squared differences (SSD) and the normalized cross-correlation (NCC) may be used. When using a correlation value other than SAD, there are two cases, depending on the characteristics, the smaller the correlation value, the higher the similarity, and the larger the correlation value, the higher the similarity. Also needs to be changed.

The motion vector detection unit 106 further moves the correlation value target block 303 for the entire area of the search range 302 to calculate the correlation value. Then, a correlation value is calculated between the template block 301 and the search range 302, and the position where the value is the smallest is determined. This makes it possible to detect where the template block on the original image has moved in the reference image, that is, the motion vector between the images.

The motion vector detection unit 106 performs such a motion vector detection process in a plurality of regions between input images of different frames. In this embodiment, an example using template matching has been described as an example of vector detection, but a method using a gradient method or a method such as a corresponding point search by feature point extraction may also be used.

In step S204, the reliability calculation unit 107 calculates the reliability of the motion vector for the image of the layer to be processed among the multi-resolution images in response to the instruction of the control unit 111. In the specific period, the reliability calculation unit 107 calculates the reliability for each motion vector using the input image obtained from the memory 104 and the motion vector detection unit 106 and the motion vector detection result.

In general, the captured scene may include a weak area where vector detection is likely to fail, such as a low contrast area or an occlusion area. In the low contrast area, it is difficult to find a highly correlated area because there is no meaningful texture in the template block or search range. Further, the occlusion area is an area in which what was visible in one image is hidden in the next image due to the influence of parallax or a moving object, or what is hidden appears. Since the texture in the template block in such an occlusion area becomes a different texture depending on the appearance in the reference image, it becomes difficult to find a highly correlated area. For such a weak area, the type and degree of weakness are calculated by analyzing the input image and the detected motion vector, and output as a reliability map for each motion vector.

In the present embodiment, as an example of the reliability calculation method, a method of calculating the reliability based on the low contrast region will be described as an example. The contrast in the image can be obtained according to Equation 2 as the dispersion value V of the luminance value.

Here, M and N are the window size of the area for which the dispersion value is calculated, and x (i, j) is the brightness value of the pixel at the coordinates (i, j) in the window. xave indicates the average brightness value in the window. The dispersion value V becomes a small value when the brightness value of each pixel in the window is close to the average value. Therefore, the reliability calculation unit 107 calculates the reliability as the dispersion value V is smaller and the luminance fluctuation in the region is smaller (that is, the contrast is lower). Further, the size of the window size N × M is not particularly limited, but for example, in 3 × 3, the contrast in a narrow range is determined, and the larger the window size is, the wider the contrast is. It will be judged. The reliability calculation unit 107 calculates the variance value V in all the pixels for which the motion vector is detected and associates it with the motion vector to generate a contrast reliability map for each detected motion vector.

In the present embodiment, the method of calculating the contrast reliability based on the dispersion value has been described as an example of the reliability, but the type of reliability is not limited to this. As an example of other reliability, any region that is not good at vector detection in the input image, such as the occlusion region, edge strength, and corner feature amount described above, can be used as the reliability. Is possible. In addition, a method of analyzing the detected motion vector itself, such as separately exchanging the original image and the reference image to detect the motion vector and using the consistency with the detection result before the exchange as the reliability, is used. Also, the type of reliability to be output does not have to be one.

The processes of steps S204 and S203 are executed for all layers of the multi-resolution image generated by the multi-resolution image generation unit 105. At this time, the motion vector and the reliability map may be generated individually for the image of each resolution, or the motion vector detection result in the low resolution image is used as the initial value of the detection in the high resolution image. Detection may be performed using such a hierarchical structure.

In step S205, the control unit 111 determines whether the processes of steps S203 and S204 have been performed for all layers of the multi-resolution image. When the control unit 111 determines that these processes are performed for all layers of the multi-resolution image, the control unit 111 proceeds to the process in step S206. On the other hand, when the processing for all layers of the multi-resolution image is not completed, the processing is returned to S203 in order to perform the processing of steps S203 and S204 for all layers of the multi-resolution image.

In step S206, the reliability determination unit 109 determines the reliability of the motion vector of the image of the output resolution in response to the instruction of the control unit 111. Specifically, the reliability determination unit 109 uses a reliability map of each resolution obtained from the reliability calculation unit 107 and a shooting parameter obtained from the shooting parameter acquisition unit 108 to obtain a motion vector of an image having an output resolution. Generate a confidence map for. In this embodiment, the reliability map for the motion vector of the image of the output resolution is also referred to as reliability information. The output resolution represents a resolution that outputs a motion vector as a processing result among the multi-resolution images generated by the multi-resolution image generation unit 105. This output resolution does not necessarily have to be the same as the resolution of the input image. The resolution at which the motion vector detection result is output may be automatically changed, for example, according to the intended use of the motion vector, or may be arbitrarily set by the photographer. For example, if a low-resolution image is used as the output resolution, the processing time is fast but the accuracy of the motion vector is low, and conversely, if the high-resolution image is used, the processing takes time but the motion vector can be output with high accuracy.

As described above, the reliability map of the same resolution is output together with the motion vector information. Therefore, by using the reliability map in the subsequent processing, it is possible to select the necessary motion vector from the motion vectors in the image. That is, it is possible to output a better reliability map than when only a reliability map for the same resolution as the output resolution is generated or when the reliability map of a specific layer is resized and output.

A case where the reliability based on the contrast is used for each resolution will be described with reference to FIG. FIG. 4A shows an original image, and it is assumed that the background region shown in 401 is a low contrast region such as a blue sky. Then, FIG. 4B schematically shows the motion vector detection result when the same size 4 (a) is used as the original image and the same size resolution is used as the output resolution. The arrow 410 indicates the motion vector detection result, and in the example of FIG. 4B, it indicates that the motion to the left occurs in the area 402 of the person. In FIG. 4B, the display of the motion vector is omitted in the region where there is no motion, but the motion vector having an extremely small length is calculated in such an region.

Further, FIGS. 4 (c) and 4 (d) represent reliability maps for the motion vector shown in FIG. 4 (b). FIG. 4 (c) represents a reliability map at the same resolution, and FIG. 4 (d) represents a reliability map at a lower resolution than FIG. 4 (c). Here, the white region shown in 403 of FIG. 4C represents a region with high reliability, that is, a region with high contrast. On the other hand, low contrast areas such as background areas are shown in black as having low reliability.

When the reliability determination unit 109 determines the reliability of the motion vector using the reliability map, the reliability determination unit 109 refers to the reliability at the same position as the coordinate position where the motion vector is detected in the reliability map. If the output resolution of the motion vector and the resolution of the reliability map are different, the resolution of the reliability map is resized according to the output resolution in advance.

Here, the advantages of using the reliability map for each resolution will be described. In the high-resolution reliability map shown in FIG. 4C, the contour shapes of objects and textures in the scene are clear, and the reliability is determined even for fine textures. When the resolution is high, it becomes easy to react to minute textures, so the reliability judgment tends to be strict, and even in the area where the motion vector is correctly detected, it is judged that the reliability is low. There is.

In the low-resolution reliability map shown in FIG. 4 (d), as shown in 404, the high-reliability region has a spread with respect to the high-resolution one, and the reliability for fine textures is calculated. It disappears. In other words, in a low-resolution reliability map, the reliability judgment tends to be loose, and even a motion vector detected in a weak area such as a uniform background may be judged to have high reliability. be. In the case of a resolution intermediate between FIGS. 4 (c) and 4 (d), the property is also intermediate between (c) and (d). Although the properties of the reliability map for each resolution have been described based on the contrast reliability shown in FIG. 4, these properties are not limited to the contrast reliability, but are related to other types of reliability. The same can be said for.

As described above, although the properties of the reliability map differ depending on the generated resolution, simply outputting the reliability map having the same resolution as the output resolution only reflects the properties of the reliability map at high resolution. That is, it is difficult to improve the accuracy of the reliability determination and obtain a motion vector of higher quality than before. Therefore, in the present embodiment, in the final generation of the reliability at the output resolution of the motion vector, the reliability maps calculated at a plurality of different resolutions are reflected according to the shooting conditions. Specifically, the reliability determination unit 109 performs weighted synthesis of reliability maps calculated at a plurality of different resolutions based on the acquired shooting parameters. A specific example of this reliability determination process will be described later. When the reliability determination process by the reliability determination unit 109 is completed, the control unit 111 ends a series of operations related to the reliability information generation process.

(A series of operations related to reliability determination processing)
Next, the reliability determination process in step 206 will be further described with reference to FIG. In step S501, the reliability determination unit 109 acquires a reliability map for each resolution generated by the reliability calculation unit 107. In step S502, the reliability determination unit 109 acquires the shooting parameters at the time of input image acquisition via the shooting parameter acquisition unit 108.

In step S503, the reliability determination unit 109 weights the reliability map of each layer according to the acquired shooting parameters. At this time, the reliability determination unit 109 generates weighting information representing weights for each resolution image. The weighting according to the shooting parameter can be different in the mode of weighting depending on the type of the shooting parameter and the effect obtained by changing the shooting parameter.

For example, when the shooting parameter is the exposure time, the movement blur in the image becomes large due to camera shake and the movement of the subject in a long exposure time. At this time, in the low-resolution reliability map, there is a high possibility that even the region where the motion blur occurs is judged to be highly reliable. However, the region where blurring occurs, that is, the region with low contrast is a region where the texture is poor and the detection of the motion vector is likely to fail, and it must be determined that the reliability of the motion vector is low. Therefore, when the shooting parameter is the exposure time, the reliability determination unit 109 makes the weight for the reliability map in the high-resolution image larger than the weight for the reliability map in the low resolution. At this time, the longer the exposure time, the greater the weight on the reliability map in the high-resolution image.

The same applies when the shooting parameter is an F value. If the F value is small, the depth of field becomes shallow, so that the area other than the subject of interest, such as the background area, becomes blurred. Therefore, the reliability determination unit 109 makes the weight of the reliability map in the high-resolution image larger than the weight of the reliability map in the low-resolution image as the F value becomes smaller, as in the case of the above-mentioned exposure time. That is, when the shooting parameter is a shooting parameter related to the degree of blurring of the shot image, the reliability determination unit 109 weights the reliability map in the high-resolution image as the value of the shooting parameter increases the blurring. Greater than the weight of the confidence map for low resolution images.

Further, when shooting is performed with the sensitivity of the image sensor set to high in a night scene or the like, the amount of noise in the image increases. Therefore, when the reliability is calculated for a high-resolution image, the noise may be judged as a significant texture and the reliability may be high. Therefore, when the shooting parameter is the sensitivity, the reliability determination unit 109 makes the weight for the reliability map in the low-resolution image in which fine textures can be ignored larger than the weight for the reliability map in the high resolution. .. That is, when the shooting parameter is a shooting parameter related to an increase or decrease in noise, the reliability determination unit 109 weights the reliability map at a low resolution at a high resolution as the value of the shooting parameter increases the noise. Greater than the weight for the confidence map. By doing so, it is possible to generate reliability that is not easily affected by noise.

As an example of further shooting parameters, it is also possible to weight the reliability map for each resolution by using the motion information of the digital camera 100 obtained from the angular velocity sensor or the acceleration sensor mounted on the digital camera 100. As a weighting method, in the motion information obtained from the sensor, it is determined that the larger the rotation, tilt, and scaling motion components in the input image, the greater the change in the shape of the texture between the original image and the reference image. do. Motion vectors can be detected well when the textures only translate between images, such as translational movements, but when the above deformation occurs, the same textures disappear between the images, so the movements occur. The vector detection accuracy is reduced. Even when the movement of rotation, tilting, and scaling is large, the amount of movement itself is small in the low-resolution image, so it can be approximated to translational movement. Therefore, in a low-resolution image, it is possible to stably detect a motion vector in a state where the influence of texture deformation between images due to rotation, tilting, and variable magnification movement is small. Therefore, the more the motion information indicates that the amount of deformation of the subject in the input image (due to rotation, tilting, scaling) increases, the more the weight of the reliability map in the low resolution image is weighted in the reliability map in the high resolution image. It should be larger than the weight of.

As other information used for determining the weighting, for example, a shooting mode specified by the user may be used. An example of a shooting mode specified by the user is blur correction processing. As a general method of image correction for blur correction, a feature point in an image is extracted, and the amount of geometric deformation of the image for blur correction is calculated from the locus information obtained by tracking the points over a plurality of frames. There is a way. In the process of estimating the amount of geometric deformation between frame images, it is sufficient to have several tens of high-precision locus information, so by increasing the weighting of reliability in high-resolution images, highly reliable motion vectors are carefully selected. It is necessary to do.

On the other hand, in the case of a mode in which image processing is performed on all pixels in an image, for example, motion blur is applied, as many motions as possible can be performed even if the accuracy of each motion vector is allowed to decrease. I need to get a vector. Therefore, in such a case, the weight of reliability in the low-resolution image is increased. That is, when the shooting mode specified by the user is a mode in which image processing is performed based on the information of some pixels in the image, the reliability determination unit 109 weights the reliability map in the high-resolution image to a low resolution. Greater than the weight for the confidence map in the image. On the other hand, when the shooting mode specified by the user is a mode in which image processing is performed based on the information of all pixels in the image, the weight for the reliability map in the low resolution image and the weight for the reliability map in the high resolution image are applied. Make it bigger. In the subsequent steps, a reliability map according to the output resolution is generated based on the weighted information for each generated resolution.

In step S504, the reliability synthesis unit 110 performs resizing processing on the reliability map of each resolution to unify the reliability map of each resolution to the size of the output resolution. For the resizing process, a general method such as Nearest Neighbor or Bilinear may be used. At this time, the resolution whose weight is determined to be zero in step S503 is not used in the generation of the reliability map at the final output resolution, so it is discarded in this step without performing the resizing process. May be good.

In step S505, the reliability synthesis unit 110 synthesizes the reliability map of each resolution subjected to the resizing process into one reliability map at the output resolution, and generates reliability information. FIG. 6 shows the structure of the weighting information at an arbitrary coordinate position on the reliability map. When the type of resolution (the number of multi-resolution images) is N, the reliability at each coordinate position of the output resolution is stored in 601 which is data for N bits. That is, the weighting information is obtained by storing the reliability value for each resolution in each bit of the N-bit data 601.

Specifically, in order to generate the weighting information, first, the reliability for each resolution is binarized. In order to binarize, it is necessary to set a threshold value for the reliability in advance, and as a setting method thereof, for example, there is a method of experimentally determining the motion vector according to the purpose of use. Regarding the method of determining the threshold value for each resolution, for example, if the threshold value for a high-resolution image is determined and the value is used for other resolutions, the reliability can be determined by a unified scale for different resolutions. It can be carried out. As a result, it can be determined that the area where similar textures continue to exist at any resolution, for example, where very good detection can be performed when detecting a hierarchical motion vector, is highly reliable. I can. In addition, an adaptive reliability determination may be performed for each resolution, such as changing the threshold value according to the ratio of the resolutions based on the threshold value in the output resolution.

When the binarization process of the reliability of each resolution is completed, the result is stored in the N-bit data 601 shown in FIG. For example, by storing the reliability of a resolution having a larger weight in a higher bit, it is possible to generate reliability information in consideration of the reliability weight of each resolution.

Specifically, for example, when N = 8 bits, the reliability of the resolution having the largest weight among the eight resolutions is stored in the most significant bit 602. Then, the reliability of the resolution having the smallest weight is stored in the least significant bit 603. Similarly, for other resolutions, the reliability of the resolution is stored at the bit position according to the weight of the reliability. At this time, for which resolution corresponds to which bit, for example, a predetermined table may be used according to the setting of the shooting parameter. For example, when shooting with predetermined shooting parameters, if the weighting decreases from the lowest resolution to the highest resolution, the lowest resolution corresponds to the most significant bit 602 and the highest resolution corresponds to the least significant bit 603. Will be done.

In the reliability data finally created in this way, a value of 127 is assigned to the reliability with the highest weight, and a value of 1 is assigned to the reliability with the lowest weight. Therefore, the larger the value, the higher the value. It shows that the reliability is high at the resolution with a large weight. Further, even when it is determined that the reliability is high at more resolutions, the value of the reliability data shown in FIG. 6 becomes large. Therefore, even if the weight is small, if the reliability is high at many resolutions, the overall high reliability is obtained. It is also possible to judge that it is reliable. In addition, in the data structure in the figure, the reliability judgment result of each resolution is stored completely independently for each bit, so only the reliability at a specific resolution is extracted during the subsequent processing. It can also be used easily.

In the above-described embodiment, the method of integrating the reliability maps weighted for each resolution to generate one reliability map (that is, reliability information) according to the output resolution of the motion vector has been described. It is not limited to this. If there are no particular restrictions on the data capacity or processing time, the size of each resolution may be output as it is, and the weight and positional relationship for each layer may be calculated during processing in the subsequent stage. It is also possible to use the motion vector output resolution reliability by multiplying the multi-valued reliability by a weighting coefficient and then adding the reliability for each layer without binarizing the reliability. It is possible.

When the reliability synthesizing unit 110 generates the motion vector at each pixel position of the output resolution and the reliability for the motion vector, the reliability synthesizing process ends the resized reliability map, and then the series of operations ends. The motion vector at the output resolution and the reliability information for the motion vector are recorded in the memory 104 or the recording unit 112, or transmitted to the control unit 111, for example. Further, the motion vector at the output resolution and the reliability information for the motion vector may be recorded as additional information in the moving image file generated by the imaging.

In addition to the above-mentioned reliability information generation process, for example, the control unit 111 selects a motion vector (corresponding to a reliability equal to or higher than a predetermined threshold value) in an image having an output resolution by using the generated reliability information. You may try to do it. Then, predetermined image processing such as blur correction processing and image compression processing may be performed using a highly reliable motion vector selected using the reliability information.

As described above, in the present embodiment, when calculating the reliability of the motion vector, the reliability calculated for each resolution is weighted based on the shooting parameters and the like, and one reliability map for the output resolution (that is, that is). Changed to generate reliability information). By doing so, the reliability for each motion vector can be obtained with high accuracy. That is, it becomes possible to generate good reliability information with reduced erroneous determination, and it becomes possible to accurately select motion vectors that are detected correctly with high reliability.

(Embodiment 2)
Next, the second embodiment will be described. In the present embodiment, it is determined what kind of weight is applied to the reliability for each resolution before the motion vector is detected. The digital camera 700 of the present embodiment has a different configuration for weighting the reliability for each resolution, but other configurations are the same as those of the first embodiment. For this reason, the same components are designated by the same reference numerals, duplicate explanations are omitted, and differences are mainly described.

FIG. 7 shows the configuration of the digital camera 700 according to the second embodiment. In FIG. 7, the reliability determination unit 109 is arranged immediately after the multi-resolution image generation unit 105 with respect to the configuration shown in FIG. A series of operations related to the reliability information generation process in such a configuration will be described with reference to FIG. Also in the present embodiment, in the reliability information generation process, the control unit 111 expands and executes the program recorded in the recording unit 112 in the working area of the memory 104, and the multi-resolution image generation unit 105 and the like. It is realized by controlling each part.

First, the development processing unit 103 and the multi-resolution image generation unit 105 execute steps S201 and S202 in the same manner as in the first embodiment.

In step S801, the reliability determination unit 109 performs the reliability determination process according to the present embodiment by using the imaging parameter information acquired via the imaging parameter acquisition unit 108. Here, the reliability determination process of this step corresponds to steps S502 and S503 in the first embodiment, as shown in FIG. In this way, by determining the reliability weighting for each resolution before generating the reliability for each resolution, for example, the reliability calculation itself can be prevented at the resolution where the weight is zero. As a result, it is possible to save processing resources. The same is true for resolutions with very low weights (ie, weights below a given weight). Alternatively, for resolutions with extremely small weights, the contribution of the final composite of the reliability maps of each resolution is low, and if it is judged that they can be ignored, the reliability generation process may not be performed. good. As an example of the degree of contribution, it may be determined that the ratio of the weights of the resolution of interest is extremely small with respect to the largest weight, for example, 1/10 or less.

The process of generating the reliability information according to the present embodiment will be described with reference to FIG. 8 again. In steps S203 to S205, the motion vector detection unit 106, the reliability calculation unit 107, and the control unit 111 execute the same processing as in the first embodiment to detect the motion vector and calculate the reliability in all layers. .. However, in the reliability determination in step S801, the reliability calculation process in step S204 may not be performed for the resolution for which the weight is determined to be zero or extremely small.

In step S802, the reliability weighted for each resolution is combined into one reliability map at the output resolution of the motion vector. The reliability synthesis process in step S802 will be described with reference to FIG.

Each step shown in FIG. 10 is substantially the same as steps S501, S504 and S505 in the first embodiment. However, the reliability synthesis unit 110 resizes and synthesizes the reliability map without considering the reliability of the resolution for which the reliability was not calculated in S204. When the reliability map synthesis in step S505 is completed, the reliability synthesis unit 110 returns the process to S802.

As described above, in the present embodiment, the weighting of the reliability for each resolution is determined before the reliability calculation processing is performed. By doing so, it is possible to suppress the calculation of the reliability of the resolution unnecessary for the generation processing of the reliability of the motion vector at the output resolution, and it is possible to reduce the memory and the processing time.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

107 ... reliability calculation unit, 109 ... reliability determination unit, 110 ... reliability synthesis unit

Claims (13)

An image acquisition means for acquiring a plurality of hierarchical images having different resolutions for an input image,
Determining means for determining the weight for each layer of the plurality of layered images according to the photographing parameters for photographing the input image, and
A reliability calculation means for calculating the reliability of each of the plurality of motion vectors for a plurality of motion vectors obtained between the images of the same layer of the plurality of layered images acquired from input images having different frames.
It has a reliability synthesis means for synthesizing the reliability for each of the plurality of motion vectors for each layer, which is calculated by the reliability calculation means using the weight for each layer .
The image processing apparatus is characterized in that the reliability calculation means does not calculate the reliability for each of the plurality of motion vectors for a layer having a weight lower than a predetermined threshold among the weights for each layer . The image processing apparatus according to claim 1 , wherein the determination means has a different weighting mode depending on the type of the photographing parameter. When the shooting parameter is a shooting parameter related to the degree of blurring of the input image, the determination means is such that the value of the shooting parameter increases the blurring, the higher the resolution image among the plurality of layered images. The image processing apparatus according to claim 1 or 2 , wherein the weight of the image is made larger than the weight of the low-resolution image. The image processing apparatus according to claim 3 , wherein the photographing parameter relating to the magnitude of the blur of the input image includes at least one of an exposure time and an F value. When the shooting parameter is a shooting parameter relating to an increase or decrease in noise of the input image, the determination means means that the higher the value of the shooting parameter is the value that increases the noise, the lower the resolution of the plurality of layered images. The image processing apparatus according to claim 1 or 2 , wherein the weight is made larger than the weight of the high-resolution image. The image processing apparatus according to claim 5 , wherein the photographing parameter relating to the increase / decrease in noise of the input image includes the sensitivity of the image pickup device. When the shooting parameter is a shooting parameter relating to the amount of deformation of the subject in the input image, the determination means is so low among the plurality of hierarchical images that the shooting parameter indicates that the amount of deformation of the subject is large. The image processing apparatus according to claim 1 or 2 , wherein the weight of the resolution image is made larger than the weight of the high-resolution image. The image processing apparatus according to claim 7 , wherein the photographing parameter relating to the amount of deformation of the subject in the input image is a photographing parameter relating to the movement of the image processing apparatus. 1 . The image processing apparatus described in the section. An output means for outputting the plurality of motion vectors for a hierarchical image having a predetermined resolution and the reliability for each of the plurality of motion vectors synthesized by the reliability synthesizing means corresponding to the hierarchical image having the predetermined resolution. The image processing apparatus according to any one of claims 1 to 9 , further comprising. A selection means for selecting a part of the plurality of motion vectors for a hierarchical image having a predetermined resolution by using the reliability for each of the plurality of motion vectors synthesized by the reliability synthesizing means.
The image processing according to any one of claims 1 to 10 , further comprising an image processing means for performing a predetermined image processing using a part of the selected motion vector. Device. An image acquisition process in which the image acquisition means acquires a plurality of hierarchical images having different resolutions with respect to the input image.
A determination step in which the determination means determines the weight for each layer of the plurality of layered images according to the photographing parameters for photographing the input image.
The reliability calculation means calculates the reliability of each of the plurality of motion vectors for the plurality of motion vectors obtained between the images of the same layer of the plurality of layered images acquired from the input images having different frames. Reliability calculation process and
The reliability synthesis means has a reliability synthesis step of synthesizing the reliability for each of the plurality of motion vectors for each layer, which is calculated in the reliability calculation step by using the weight for each layer. ,
In the reliability calculation step, the control method of the image processing apparatus is characterized in that the reliability for each of the plurality of motion vectors is not calculated for the layers having weights lower than a predetermined threshold among the weights for each layer. .. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 11 .

Images